Assessment

Comparable Outcomes: Setting the standard?

What is the comparable outcomes framework, how does it underpin grade standards and are there alternatives?

Why is the role of comparable outcomes being debated?
What is the comparable outcomes framework?
Why is the comparable outcomes framework used?
What is the principle underlying the comparable outcomes framework?
How does the comparable outcomes approach work in practice?
What are the benefits of the comparable outcomes framework?
What are the challenges with applying the comparable outcomes framework in setting grade boundaries?
What are the alternatives to standardising GCSE, AS and A-level grades?
What is the future for the comparable outcomes framework?
Date18/11/21
AuthorCorina Balaban (Researcher), James Lloyd (Head of Policy and Public Affairs) & Phoebe Surridge (Researcher)- AQA

Why is the role of comparable outcomes being debated?

The comparable outcomes framework has been at the heart of standards and grade setting for GCSEs, AS and A-levels in England over the last decade. Politicians credit it for having “halted” grade inflation.[1]

However, over time, stakeholders have raised concerns around its use and perceived impact on schools in particular.

This briefing describes how comparable outcomes is used, its key benefits, challenges and alternative approaches to setting standards.

What is the comparable outcomes framework?

Comparable outcomes is a framework that exam boards in England, under the oversight of the exams regulator Ofqual, use to guide the setting of grade boundaries for GCSEs, AS and A-levels. By applying the comparable outcomes framework, exam boards and the regulator are better able to standardise the grades awarded each year at a national level. The aim of the comparable outcomes framework is to ensure that grades are comparable over time and across exam boards.[2]

Why is the comparable outcomes framework used?

For students, the value of GCSE, AS and A-level grades depends on their currency with employers and education institutions, such as universities.

Without any attempt at standardisation, qualifications would lose their value. If an employer or university cannot be confident as to what level of attainment is represented by different grades awarded to a student, the value of that student’s qualifications is undermined.

The comparable outcomes framework is used in England as the preferred method for enabling greater standardisation of grades.

Crucially, applying the comparable outcomes framework means a grade in one subject in one year can more meaningfully be compared:

  • with other years, e.g. a grade 6 in GCSE Maths in 2018 will reflect a similar level of attainment to a grade 6 in GCSE Maths in another year;
  • across exam boards, e.g. the outcome of a student taking an AQA exam should be comparable to the outcome of a student taking the same qualification from another exam board.[3]

What is the principle underlying the comparable outcomes framework?

The basic principle that underpins the comparable outcomes framework is that for a qualification where the entry cohort is similar to previous years, the overall proportion of students achieving each grade should also be similar to previous years, with some adjustment for differences in the difficulty of the papers.[4]

How does the comparable outcomes approach work in practice?

Each exam board uses the same statistical process to model grade outcomes for each of their qualifications. Ofqual oversees the application of comparable outcomes by exam boards, and describes the approach as follows:[5]

“Predictions are based on the relationship [emphases added] between prior attainment and national results in a reference year. Exam boards use prior attainment at Key Stage 2 when predicting GCSE outcomes, and prior attainment at GCSE when predicting AS and A level outcomes.”

These predictions are used to generate statistically recommended grade boundaries i.e. where the statistics suggest the position of grade boundaries should be. Senior examiners then look at student exam papers around each boundary to see whether they represent an appropriate level of performance. The examiners make adjustments to the boundaries as they see necessary. The weighting given to the statistical predictions in making decisions about grade boundaries also depends on the robustness of the statistics.

Final grade boundaries are therefore decided by a combination of the examiner perspective and the statistics.

What are the benefits of the comparable outcomes framework?

In addition to improving comparability over time and across exam boards, there are other key benefits to the application of the comparable outcomes framework.

First, the comparable outcomes approach enables new qualifications – and new versions of existing qualifications – to be introduced without penalising the earliest cohorts who take these qualifications.[6]

When new qualifications are introduced, there can be a sudden drop in performance followed by a slow recovery, often referred to as the ‘Sawtooth Effect’.[7] There are a number of reasons why the first students to take a new qualification might be expected to perform worse relative to later cohorts.[8] For example, teachers may be less familiar with teaching the content of the new specification, and there may be fewer supporting materials available, such as text books and sample exam papers.[9]

The comparable outcomes framework ensures there is a smooth transition between old and new qualifications (e.g. in the wake of a reform of the national curriculum).

Second, the comparable outcomes approach makes it easier to separate out changes in performances caused by differences in the difficulty of the paper from changes caused by differences in the students taking the paper. For example, if a GCSE Physics paper is harder than the paper from the previous year – despite the many quality assurance processes exam boards use to make the difficulty of papers as similar as possible – the comparable outcomes approach is used to help ensure students sitting the harder paper aren’t unfairly penalised for doing so.[10]

Third, the comparable outcomes approach has proved very effective in addressing concerns about grade inflation. In periods of stability, there has been a historical tendency for average grades to increase over time, damaging trust in the qualifications.[11] The comparable outcomes framework helps stabilise the value of the qualifications for students, universities and employers.[12]

What are the challenges with applying the comparable outcomes framework in setting grade boundaries?

A number of stakeholders have criticised the use of the comparable outcomes approach to inform grade boundaries for GCSE, AS and A-levels in England.

These criticisms typically do not stem from concerns with the use of the comparable outcomes framework itself, nor the policy decision to attempt to standardise grades.

Instead, these concerns typically relate to the interaction of these policy choices with other aspects of the school and accountability system.

In particular, there is concern that the comparable outcomes approach does not reflect any genuine rise (or fall) in attainment in the grade outcomes awarded.[13] [14] By maintaining stability of outcomes at a national level, it is sometimes perceived that individual schools cannot demonstrate improvements, or that they cannot do so without a corresponding drop in another school’s results, although there has been no decline in the standards achieved by that school.[15] [16] It is therefore felt that comparable outcomes cannot fully reflect improvements in teaching and professional development.

Another concern arises from the way in which applying the comparable outcomes framework can mean a broadly comparable proportion of students is awarded a GCSE grade 3 to 1 each year. Although exam boards do not award ‘fail’ grades, the government’s policy of describing grade 4 as a ‘standard pass’ means grades 3 to 1 are often viewed implicitly as fail grades. Some stakeholders argue that in effect, some students will ‘inevitably’ fail, and that this is unacceptable in the context of student trust and experience, as well as damaging to student motivation. 

Several points can be made in response to these concerns.

First, in practice, large cohort-level changes in performance in a single year are rare.[17] [18] Small changes at a national level can and do occur even within comparable outcomes – the process aims for stability over time, it does not fix outcomes. Over several years, trends in performance could be observed.

Second, as an additional safeguard, Ofqual has introduced the National Reference Test (NRT) to detect any improvements in teaching standard over time in GCSE English Language and Maths, at a national level.[19] The test is administered to a sample of Year 11 students in the March before they sit their GCSE examinations. The results could be used to amend the GCSE grade boundaries if any large changes in performance were observed.

Historically, prior to the adoption of the comparable outcomes approach, two other processes were used to ensure comparability of grades within the exam system in England. Both processes are, to some extent, still applied alongside the comparable outcomes framework, but in a more nuanced way.

Norm-referencing sees each grade consistently awarded to the same percentage of each subject cohort consistently year on year and across exam boards without adjusting results based on previous years. This approach was popular in the 1950s but fell out of favour towards the 1980’s.[20]

However, norm-referencing confronts a number of challenges associated with the comparable outcomes approach – e.g. the inability to demonstrate systemic improvements in learning outcomes – without the additional flexibility provided for comparable outcomes by the use of data from the National Reference Test.

The second approach, criterion-referencing, was popularised in the 1980s as replacement for norm-referenced tests.[21] It aims to measure a candidate’s performance against a set of pre-defined assessment criteria; each grade is awarded to all those who satisfy the performance-related criteria stated for that grade.[22] In contrast to norm-reference tests, in criterion-referenced tests, the performance of other students does not affect a student’s score.[23]

Crucially, however, unlike the comparable outcomes framework, when used alone, criterion referencing does not protect against the Sawtooth Effect described above.

What are the alternatives to standardising GCSE, AS and A-level grades?

If policymakers decided to end the use of the comparable outcomes framework and any attempt to standardise grades, and instead rely entirely on subjective examiner judgement, this would result in the loss of the benefits described above in relation to the currency of grades, grade inflation and the smooth transition between cohorts.

In addition to the loss of such benefits, policymakers would likely need to accept accompanying consequences. For example, the development of bespoke entrance tests to enable differentiation between applicants by employers and education institutions who no longer felt were able to rely on GCSE, AS and A-level grades.

However, the use of bespoke grades would undoubtedly lead to concerns around fairness. For example, variations in the ability of individual schools (or families) to prepare students for the entrance tests of specific institutions would be likely to lead to challenges for social mobility.

What is the future for the comparable outcomes framework?

The principal beneficiaries of comparable outcomes are students, given the critical role the framework has had in underpinning the currency of GCSE, AS and A-level qualifications.

As noted, many criticisms of the use of the comparable outcomes approach to standardise grades do not reflect objections to the approach itself - or the policy decision to standardise GCSE, AS and A-level grades – but rather, they relate to the interaction of these decisions with policy decisions around school accountability and the GCSE ‘pass’ framework.

There is no doubt that the comparable outcomes approach represents a complex process that is prone to misunderstanding by stakeholders,[24] with accompanying risks to public confidence.

Ultimately, policymakers may conclude public confidence in comparable outcomes cannot be maintained alongside the existing school accountability framework. In this situation, policymakers may need to choose between the relative importance of grading standards versus other policy objectives, or consider other options to meet those objectives.

[1] Gibb, N. (2016). Government Response to the Consultation on Ofqual’s National Reference Test. Department for Education. https://questions-statements.parliament.uk/written-statements/detail/2016-03-24/HCWS650

[2] Newton P., E. (2020) What is the Sawtooth Effect? The nature and management of impacts from syllabus, assessment, and curriculum transitions in England. Ofqual.

[3] Ofqual (2017). Inter-board comparability of grade standards in GCSEs AS and A levels 2017. Inter-board comparability of grade standards in GCSEs, AS and A levels 2017 (publishing.service.gov.uk)

[4] Ofqual (2017). Inter-board comparability of grade standards in GCSEs AS and A levels 2017. Inter-board comparability of grade standards in GCSEs, AS and A levels 2017 (publishing.service.gov.uk)

[5] Ofqual (2017) Inter-board comparability of grade standards in GCSEs, AS and A levels 2017. Inter-board comparability of grade standards in GCSEs, AS and A levels 2017 (publishing.service.gov.uk)

[6] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[7] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[8] Jadhav C (2016) Setting standards for new AS qualifications. The Ofqual Blog. https://ofqual.blog.gov.uk/2016/08/09/setting-standards-for-new-as-qualifications/

[9] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[10] Newton P., E. (2020) What is the Sawtooth Effect? The nature and management of impacts from syllabus, assessment, and curriculum transitions in England. Ofqual.

[11] Benton, T. & Sutch, T. (2014). Analysis of use of Key Stage 2 data in GCSE predictions, ARD Research Division. Ofqual

[12] Benton, T. (2016). Comparable Outcomes: Scourge or Scapegoat? Cambridge Assessment Research Report. Cambridge, UK: Cambridge Assessment

[13] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[14] Benton, T. (2016). Comparable Outcomes: Scourge or Scapegoat? Cambridge Assessment Research Report. Cambridge, UK: Cambridge Assessment

[15] Bousted, M. (2015) ‘England’s secondary heads and teachers are stuck in a zero-sum game from which it’s impossible to escape’. TES News. ‘England’s secondary heads and teachers are stuck in a zero-sum game from which it’s impossible to escape’ | Tes News

[16] Benton, T. (2016). Comparable Outcomes: Scourge or Scapegoat?. Cambridge Assessment Research Report. Cambridge, UK: Cambridge Assessment

[17] Sizmur, J., Ager, R., Bradshaw, J., Classick, R., Galvis, M., Packer, J., ... & Wheater, R. (2019). Achievement of 15-year-olds in England: PISA 2018 results: Research report, December 2019. Research and analysis overview: PISA 2018: national report for England - GOV.UK (www.gov.uk)

[18] Coe, R. (2007). Changes in standards at GCSE and A-level: Evidence from ALIS and YELLIS: A report for the ONS.

[19] Stacey, G. (2015). The national reference test. The Ofqual Blog. The national reference test - The Ofqual blog

[20] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[21] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[22] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

[23] Prince, R. (2016). Predicting Success in Higher Education: The Value of Criterion and Norm-Referenced Assessments. Practitioner Research in Higher Education10(1), 22-38.

[24] Newton, P. E. (2021). Demythologising A level Exam Standards. Research Papers in Education, 1-32

Adaptive Assessment

Adaptive Assessment: A missing ingredient in the resit recipe?

The number of students resitting their maths GCSE is growing, but the proportion getting a grade 4 or higher is falling. This situation is not only dispiriting for the young people striving to get the qualifications they need, but also for the teachers working hard to help them. How can outcomes for this cohort be improved? Bart Crisp, associate director at the Centre for the Use of Research and Evidence in Education, thinks adaptive assessment may be part of the solution.

Read Moreicon / report
SEND

Student success: Every little achievement counts

Baroness Morgan is calling for students to be given ‘self-belief’ lessons as a way of developing their characters and preparing them for the future. She is not the first to notice that a student’s sense of their own ability and their level of success are part of a virtuous circle. But how can teachers get the snowball rolling for students with SEND or in alternative provision? In this blog, former headteacher, John Tomsett, pulls out a swimming certificate he earned more than half a century ago to use as an inspiration for others.

Art and Design

Progress 8 – How much can it flex?

Progress 8 has come in for criticism in recent months for the way it combines with the Ebacc to skew schools’ focus towards academic subjects at the expense of creative and technical options. Both Labour and a Lords committee have called for reform to boost uptake of declining subjects such as D&T, dance and drama. In the second of two blogs looking at Progress 8, AQi examines whether the accountability measure could be realigned to address these concerns and if so, how could that be done?

Education Policy

Progress 8 – Schools’ flexible friend?

Eight years after Progress 8 came into existence it is facing calls to be updated. Some say reforming the schools accountability measure can halt the decline in take up of the arts, while others want it to boost numbers taking vocational and technical subjects. Are its shoulders broad enough to take on such a load? In the first of two blogs on Progress 8, AQi looks at how it came into being, what problems it was designed to solve and whether it is flexible enough to be reformed.

Assessment

2023: A year in review

As we wrap up a year full of exciting developments in education, it is the perfect time to take stock of what has gone on. In this blog, editor of AQi, Anil Dawar, reflects on some of his highlights for 2023.

Briefing

Levers of change: Ways that policymakers can shape the education system

With a general election looming there is much debate in the world of education about the next government’s decisions on what our children learn. But deciding this is only part of the issue for any new government. Just as important is understanding how they can actually implement those decisions. Knowing the advantages and drawbacks of all the different levers at government’s disposal is vital. In this blog, AQA’s head of external affairs Reza Schwitzer discusses what these levers are and their pros and cons.

Read Moreicon / chart
Assessment

Extended Project Qualifications: What do the statistics say about student growth?

A former student from Nottingham College won top prize in AQA’s Project Excellence Awards 2023/24 for her Extended Project Qualification on Queerness and Vampirism. Jessica Burton is now embarking on the next stage of her education journey on Cambridge University’s Foundation Year in Arts, Humanities and Social Sciences, thanks in part to the UCAS points earned through her EPQ. With that in mind, it seems the ideal moment for AQi to explore what the statistics say about how completing an EPQ can improve progression to Higher Education, degree course outcomes and social mobility.

Assessment

Digital exams: A chance to make assessment more accessible for all

A recent symposium on inclusive educational assessment at Hughes Hall, Cambridge, attracted experts from across the country. AQA lead researcher Dr Katy Finch was in the audience listening to leading academics from the fields of social exclusion, autism, dyslexia and neurodiversity. In this blog, Dr Finch draws on ideas discussed that day to deliberate on the opportunities digitising assessment offers those facing barriers in mainstream education and asks: Can we use Digital Exams as a chance to make assessment more inclusive for everyone?

Education

The Baker Reforms: What is their legacy 35 years on?

In 1988, Lord Baker created arguably the most important piece of education legislation since the war. The impact of the Education Reform Act is almost undeniably greater than anything that has come since. Its significance was immediately apparent but, if anything, it may be even more significant now. AQi investigates the legacy of the Baker education reforms 35 years on.

Assessment

Post exam analysis: How to improve student experience

Every year, once exam season ends, AQA experts look at student scripts to assess how well the exam papers were constructed. AQA’s 2023 Maths GCSE performed particularly well, even though grading returned to 2019 standards and the Advance Information used in 2022 was not there. Andrew Taylor, head of maths curriculum and Shaun Procter-Green, maths assessment lead, tell how their work redesigning the papers paid off, allowing students to better show off their skills.

Join the conversation on Twitter

Follow us for discussion and new post alerts

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

Sign up to our newsletter

Sign up for news and alerts

Pages
BlogsPublicationsData StoriesEventsAbout
Hosted and Curated by AQA with AQA logo
© 2024 AQA