Assessment

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Are all measures equal?
PISA – The Programme for International Student Assessment
PIRLS - The Progress in International Reading Literacy Study
TIMSS – The Trends in International Mathematics and Science Study
What are the differences between PISA, PIRLS and TIMSS?
How comparable are the results from PISA, PIRLS and TIMSS?
What can we conclude about PISA, TIMSS from all of this?
Read More:
Date20/02/24
AuthorAQI team

Are all measures equal?

The PISA (Programme for International Student Assessment) rankings for maths, reading and science are some of the most eagerly anticipated data in the education world. Their publication prompts headlines, applause, criticism and sometimes soul searching.

The same goes for two other well-respected International Large-Scale Assessments - TIMSS (Trends in International Mathematics and Science Study) and PIRLS (Progress in International Reading Literacy Study).


You might think that these three assessments would pick up similar trends in each country where they are used, but that is not always the case.

For example, PISA’s last report showed that English students’ maths had declined, and that a downward trend in science performances, which started in 2012, was continuing.

But analyse TIMSS’ data and you’ll see that maths performances have been on an upward trend for both Year 5 and Year 9 students since 2011.

Not only that, when it comes to science, TIMSS recorded eight years of continual improvement by Year 5 students.

PISA’s findings on reading skills also seemingly conflict with those of PIRLS.

In its 2023 report, PISA recorded a 10point drop for England on its previous study (roughly equivalent to six month’s schooling).    

Compare that with PIRLS’ 2023 report which showed England’s literacy score had fallen a statistically insignificant one point, even though the country had been through a challenging lockdown. 

So, why do separate studies of the same subjects come up with different findings? Can they be compared?

To answer these questions let’s examine PISA, TIMSS and PIRLS, see how they work and what they measure. 

PISA – The Programme for International Student Assessment

PISA has been conducted by the OECD since 2000 and is the most high-profile International Large-Scale Assessment.
It relies on a minimum of 150 schools with 4,500 to 10,000 students in each country/economy. The last survey involved 81 countries and almost 700,000 students. 
It evaluates 15-year-olds’ ability to use their reading, maths and science skills and knowledge to solve ‘real life’ problems and is produced every three years.
Students sit computer-based assessments over two hours with a mix of multiple-choice and open-ended questions. They also complete questionnaires about themselves and their school, including behaviour and discipline.
Teachers, headteachers and parents also complete questionnaires about lessons and the wider school/home environment.

PISA's Selling point

Pisa is the biggest global measure of education performance levels.
The breadth of its reach and depth of its testing goes beyond the other international studies. 
It measures attainment in a wider range of subjects than PIRLS and TIMSS.
The statistics it produces allow policymakers, educationalists and others to draw comparisons between education systems, examine elements within those systems and track changes over time.

PIRLS - The Progress in International Reading Literacy Study

PIRLS was first carried out in 2001 and is run by the International Association for the Evaluation of Educational Achievement – a co-operative of national research institutions, government agencies, educational researchers and analysts in more than 60 countries.  
The IEA (CORR) conducts its study every five years on pupils aged 9-10. 
PIRLS 2021 results were based on data from around 400,000 students, 20,000 teachers, 13,000 schools and 57 education systems. 
The study, which students can sit either digitally or using pen and paper, uses multiple-choice and open-ended questions to assess the reading attainment of pupils focusing on reading for literary experience and reading to acquire and use information. 
PIRLS assesses pupils in two 40-minute sessions. They also complete a questionnaire about their attitudes to learning, while teachers, headteachers and parents submit one about lessons and the wider school/home environment. 

PIRLS' Selling point

The assessment is designed in collaboration between PIRLS and national bodies to fit each country’s curriculum. Each assessment is linked to the preceding one which means policymakers can compare their own results over time, make global comparisons and set benchmarks.
Texts used are constantly being reviewed to ensure they reproduce the student’ typical reading experience, are clear and coherent, and culturally relevant to each country.
By assessing literacy skills in younger students, PIRLS identifies potential problems early which allows for targeted interventions and support. 
Policymakers can use PIRLS findings to share best practice, shape policy and allocate resources.

What are the differences between PISA, PIRLS and TIMSS?

They test different things

PIRLS/TIMSS are designed to evaluate students’ command of the curriculum and ability to apply their learning and skills.  
PISA on the other hand aims to evaluate the country’s education system and emphasises socio-economic and equity factors. For this reason it is important that a representative selection of students sit the assessment which is not always the case. 

These Large-Scale Assessments assess the abilities of different aged pupils

Reading 
PIRLS looks at reading for enjoyment and information retrieval and use in Primary pupils.  
PISA measures a wider range of literacy skills in secondary students, from basic decoding of text to “knowledge about the world.” 

Maths/Science 
TIMSS assesses students’ understanding of concepts in maths and science and their application mainly using multiple-choice and open-response questions.  
PISA uses interactive computer-based assessment and performance-based tasks as well as traditional multiple-choice and open-response questions. The framework is based on the Real Mathematics Education (RME) philosophy which means that education systems using standards that match RME are more likely to score highly on PISA but not TIMSS.  

How comparable are the results from PISA, PIRLS and TIMSS?

Research has shown that TIMSS and PISA provide similar pictures of student achievement at the country level. A recent assessment found close alignment between country mean scores from both studies. That study of the 2015 TIMSS and PISA results found the higher achieving East Asian countries did better in TIMSS maths than their PISA results predicted, while some Nordic and English-speaking countries did a little better in PISA.
We should note here that the TIMSS and PISA scales are different even though both are centred around 500 with a standard deviation of 100. More information on assessment scales can be found on the TIMSS and PISA websites.

What can we conclude about PISA, TIMSS from all of this?

What this shows is that PISA, TIMSS and PIRLS, taken together, can be good indicators of country achievement - on the proviso that the data is robust, and the trend is clear.
Most of the time, however, country-level data is messy and fine details get lost. 
These surveys are not granular enough to be of use for policy setting.
Sure, sometimes their findings point in the same direction, but other times they don’t. So how are we supposed to know whether the similarities or differences are real or due to each survey’s own idiosyncrasies?  
International assessments set the policy discourse and often pave the way in terms of carrying out surveys at scale, but lack in nuance when there are specific policy issues requiring an answer.
The best way to get those answers is through national assessments tailored to the country and the questions that need to be tackled.  
All that said, the vast range of data PISA, TIMSS, PIRLS and others generate can be a boon to researchers in the field.

Education

TV subtitles as an aid to literacy: What does the research say?

Jack Black is probably best known in educational circles for playing a renegade substitute teacher in School of Rock. But the Hollywood star has made a more conventional foray into education by backing the use of TV subtitles to improve child literacy. Stephen Fry and the World Literacy Foundation also want parents to use their TV remotes to get children reading. So, could this simple click of a button be a solution to boost pupils’ reading skills? AQA’s resident expert on language teaching, Dr Katy Finch, casts her eye over the research to see whether it stacks up.

Read Moreicon / report
Data Analysis

What is left behind now education’s Data Wave has receded?

Is data the solution to all education’s issues? About a decade ago the prevailing wisdom said it was. Advocates of this Data Wave argued that harvesting internal statistics would help schools solve issues such as teacher accountability and attainment gaps. As with all waves, after crashing onto the beach they recede, leaving space for another to roll in. In this blog, teacher, author and data analyst Richard Selfridge looks at the legacy of the Data Wave to see what schools can take from it.

International Approaches

Finland & PISA – A fall from grace but still a high performer?

Finland was once recognised as one of the most successful educational systems in the world. At the turn of the millennium, it topped the PISA rankings in reading, maths and science. But by 2012, decline set in. The last set of results showed performances in maths, reading and science were at an all-time low. In this blog Dr Jonathan Doherty of Leeds Trinity University outlines some reasons that may account for the slide.

Briefing

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Read Moreicon / chart
Adaptive Assessment

Adaptive Assessment: A missing ingredient in the resit recipe?

The number of students resitting their maths GCSE is growing, but the proportion getting a grade 4 or higher is falling. This situation is not only dispiriting for the young people striving to get the qualifications they need, but also for the teachers working hard to help them. How can outcomes for this cohort be improved? Bart Crisp, associate director at the Centre for the Use of Research and Evidence in Education, thinks adaptive assessment may be part of the solution.

SEND

Student success: Every milestone matters

Baroness Morgan is calling for students to be given ‘self-belief’ lessons as a way of developing their characters and preparing them for the future. She is not the first to notice that a student’s sense of their own ability and their level of success are part of a virtuous circle. But how can teachers get the snowball rolling for students with SEND or in alternative provision? In this blog, former headteacher, John Tomsett, pulls out a swimming certificate he earned more than half a century ago to use as an inspiration for others.

Assessment

Progress 8 – How much can it flex?

Progress 8 has come in for criticism in recent months for the way it combines with the Ebacc to skew schools’ focus towards academic subjects at the expense of creative and technical options. Both Labour and a Lords committee have called for reform to boost uptake of declining subjects such as D&T, dance and drama. In the second of two blogs looking at Progress 8, AQi examines whether the accountability measure could be realigned to address these concerns and if so, how could that be done?

Assessment

Progress 8 – Schools’ flexible friend?

Eight years after Progress 8 came into existence it is facing calls to be updated. Some say reforming the schools accountability measure can halt the decline in take up of the arts, while others want it to boost numbers taking vocational and technical subjects. Are its shoulders broad enough to take on such a load? In the first of two blogs on Progress 8, AQi looks at how it came into being, what problems it was designed to solve and whether it is flexible enough to be reformed.

Assessment

2023: A year in review

As we wrap up a year full of exciting developments in education, it is the perfect time to take stock of what has gone on. In this blog, editor of AQi, Anil Dawar, reflects on some of his highlights for 2023.

Briefing

Levers of change: Ways that policymakers can shape the education system

With a general election looming there is much debate in the world of education about the next government’s decisions on what our children learn. But deciding this is only part of the issue for any new government. Just as important is understanding how they can actually implement those decisions. Knowing the advantages and drawbacks of all the different levers at government’s disposal is vital. In this blog, AQA’s head of external affairs Reza Schwitzer discusses what these levers are and their pros and cons.

Join the conversation on Twitter

Follow us for discussion and new post alerts

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

Sign up to our newsletter

Sign up for news and alerts

Pages
BlogsPublicationsData StoriesEventsAbout
Hosted and Curated by AQA with AQA logo
© 2024 AQA