Assessment

Adaptive testing – tailoring the future of assessments?

Bespoke tests could pave the way to fairer and more flexible assessments, but what are the hurdles and limitations?

Date20/12/22

AuthorDr Yaw Bimpeh, Lead Researcher, AQA and Dr David West, Lead Researcher AQA

Earlier this year Ofqual announced its intention to work with exam boards to “support the use of innovative practice and technology”. Part of this involves exploring the potential role of adaptive testing – computer-based tests that select question difficulty depending on how the student answered previous questions.

In particular, Ofqual is considering whether adaptive testing could offer an alternative approach to the current system of “tiering” – whereby for certain GCSE subjects such as maths, sciences and languages, pupils can sit a “foundation” paper or a more demanding “higher” paper. Under this current system, pupils sit different exams and are limited in the range of grades they can achieve.

An extensive item bank, covering a wide range of content for different ability levels, is an essential pre-requisite. One approach to adaptive testing, item response theory, uses a statistical model to estimate a numerical value for each individual’s level of proficiency in a subject. These can help compare scores for students who have taken very different assessments.

How does it work in practice?

An adaptive assessment begins with a random selection of a few mid-difficulty items (questions). The student’s responses to these allow an initial estimate of his or her proficiency Subsequent items can be more or less difficult based on this estimate. If they are answered correctly, the next question is more difficult. If they are answered incorrectly, the next one is less challenging. The computer continuously updates its estimate of the student’s proficiency until the process is stopped.

The first generation of computerised adaptive tests starting in the 1980s only used multiple-choice questions, however, advances in psychometric techniques and more powerful computers have enabled the use of other types of questions. Some modern foreign language assessments, such as listening and reading, can also use adaptive assessment.

Adaptive tests are already in existence in the UK. The Scottish National Standardised Assessments uses adaptive testing to provide teachers with diagnostic information and policymakers with national data on student progression. In Wales, statutory personalised assessments in reading and numeracy use adaptive tests throughout Years 2 to 9.

While benefits of computerised adaptive tests include a more personalised experience for students and flexible assessment, they are not suitable for all subjects and can’t assess all skill types. Setting essays or seeking longer narrative responses within adaptive tests presents designers with more of a technical challenge. More research is required into how best to extract assessment information from more complex responses containing diagrams, data plots, tables or performance tasks, which usually need human intervention to provide a score.

The benefits of adaptive testing

students can be presented with a shorter, bespoke sequence of exam questions.
Individuals can work at their own pace
the exam schedule can be more flexible because there is no single fixed paper, and the requirement for secrecy of question papers is relaxed
feedback for students can be provided during the assessment and immediately afterwards.

The challenges to overcome

Any ambiguity about test difficulty could undermine public confidence and would require a clear explanation of how adaptive assessment works.
There are unanswered questions to resolve around how best to create large pretested item banks, select and sequence questions, and when to stop testing
High demand for new assessment items
How to ensure fairness among students when they are presented with different assessments
Student experience, i.e. conventional question papers allow a student to review the whole paper, to skip more difficult items initially and to return to these later. Computer-based adaptive testing may require exam questions to be answered in the order in which they are presented.

Questions for policymakers

Rolling out computerised adaptive testing in England, particularly in relation to high-stakes qualifications such as GCSEs and A-levels, requires careful thought and consideration.

Policymakers considering how adaptive assessment can be realised in England may wish to consider the following questions.

How would trust be maintained in any ‘black box’ algorithm to control assessments?
Will all centres have the technological infrastructure to support computer adaptive assessment?
How will the cost of developing computer adaptive assessments compare with current paper-based fixed exams?
How will computer adaptive assessment data be used to maintain an academic standard and inform policymaking?
How will teachers, head teachers, and administrators be trained to use and understand computer adaptive assessment?

Assessment

Assessing oracy: Is Comparative Judgement the answer?

Oracy skills are vital to success in school and life. And yet, for many children, opportunities to develop them are missed. Educationalists are engaging in a growing debate about where oracy fits into the school system. Labour has put it at the heart of its plans to improve social mobility and an independent commission is looking at how it is taught in the classroom. This renewed focus on oracy means it is more important than ever that teachers have a way to reliably assess and understand their students’ attainment and progression. Amanda Moorghen of oracy education charity Voice 21 explains how Comparative Judgement can help with that and why it may be a game changer.

Education

TV subtitles as an aid to literacy: What does the research say?

Jack Black is probably best known in educational circles for playing a renegade substitute teacher in School of Rock. But the Hollywood star has made a more conventional foray into education by backing the use of TV subtitles to improve child literacy. Stephen Fry and the World Literacy Foundation also want parents to use their TV remotes to get children reading. So, could this simple click of a button be a solution to boost pupils’ reading skills? AQA’s resident expert on language teaching, Dr Katy Finch, casts her eye over the research to see whether it stacks up.

Data Analysis

What is left behind now education’s Data Wave has receded?

Is data the solution to all education’s issues? About a decade ago the prevailing wisdom said it was. Advocates of this Data Wave argued that harvesting internal statistics would help schools solve issues such as teacher accountability and attainment gaps. As with all waves, after crashing onto the beach they recede, leaving space for another to roll in. In this blog, teacher, author and data analyst Richard Selfridge looks at the legacy of the Data Wave to see what schools can take from it.

International Approaches

Finland & PISA – A fall from grace but still a high performer?

Finland was once recognised as one of the most successful educational systems in the world. At the turn of the millennium, it topped the PISA rankings in reading, maths and science. But by 2012, decline set in. The last set of results showed performances in maths, reading and science were at an all-time low. In this blog Dr Jonathan Doherty of Leeds Trinity University outlines some reasons that may account for the slide.

Briefing

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Adaptive Assessment

Adaptive Assessment: A missing ingredient in the resit recipe?

The number of students resitting their maths GCSE is growing, but the proportion getting a grade 4 or higher is falling. This situation is not only dispiriting for the young people striving to get the qualifications they need, but also for the teachers working hard to help them. How can outcomes for this cohort be improved? Bart Crisp, associate director at the Centre for the Use of Research and Evidence in Education, thinks adaptive assessment may be part of the solution.

SEND

Student success: Every milestone matters

Baroness Morgan is calling for students to be given ‘self-belief’ lessons as a way of developing their characters and preparing them for the future. She is not the first to notice that a student’s sense of their own ability and their level of success are part of a virtuous circle. But how can teachers get the snowball rolling for students with SEND or in alternative provision? In this blog, former headteacher, John Tomsett, pulls out a swimming certificate he earned more than half a century ago to use as an inspiration for others.

Assessment

Progress 8 – How much can it flex?

Progress 8 has come in for criticism in recent months for the way it combines with the Ebacc to skew schools’ focus towards academic subjects at the expense of creative and technical options. Both Labour and a Lords committee have called for reform to boost uptake of declining subjects such as D&T, dance and drama. In the second of two blogs looking at Progress 8, AQi examines whether the accountability measure could be realigned to address these concerns and if so, how could that be done?

Assessment

Progress 8 – Schools’ flexible friend?

Eight years after Progress 8 came into existence it is facing calls to be updated. Some say reforming the schools accountability measure can halt the decline in take up of the arts, while others want it to boost numbers taking vocational and technical subjects. Are its shoulders broad enough to take on such a load? In the first of two blogs on Progress 8, AQi looks at how it came into being, what problems it was designed to solve and whether it is flexible enough to be reformed.

Assessment

2023: A year in review

As we wrap up a year full of exciting developments in education, it is the perfect time to take stock of what has gone on. In this blog, editor of AQi, Anil Dawar, reflects on some of his highlights for 2023.

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

Adaptive testing – tailoring the future of assessments?

How does it work in practice?

The benefits of adaptive testing

The challenges to overcome

Questions for policymakers

Assessing oracy: Is Comparative Judgement the answer?

TV subtitles as an aid to literacy: What does the research say?

What is left behind now education’s Data Wave has receded?

Finland & PISA – A fall from grace but still a high performer?

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

Adaptive Assessment: A missing ingredient in the resit recipe?

Student success: Every milestone matters

Progress 8 – How much can it flex?

Progress 8 – Schools’ flexible friend?

2023: A year in review

Join the conversation on Twitter

Download a PDF version.

Sign up to our newsletter