Assessment

What comes first – the question or the mark scheme?

With exam season on the horizon, AQi pulls back the curtain to reveal one of the unsung heroes of assessment - the Mark Scheme

Date09/03/23

AuthorSuzanne Oates

Focused mark schemes make question writing clearer and exams fairer
Photo: Scott Graham/Unsplash

From the outside, people probably think marking an exam is easy. Subject experts write questions and similarly qualified exam markers then award points according to how well students answer them. The harder the question the more marks they get, right?

The truth is far more complicated than that. AQA has to ensure hundreds of examiners mark thousands of student answers to the same standard. Here AQA’s Deputy Head of Assessment Design, Suzanne Oates, reveals the complexities she and her colleagues face and how her department’s work improves student outcomes.

Mark scheme first?

In training, assessment writers are encouraged to write mark schemes before they write the question.
‘How can you write a mark scheme before knowing what the question is?’ I hear you cry.

Well, it's very easy to think you've written a question which is clear and focused on what it is testing, but then inadvertently create a mark scheme crediting a broader range of knowledge than has actually been targeted.

If a question targeting one aspect of student knowledge is accompanied by a mark scheme crediting a wider understanding, then examiners will be confused.

They could query how to mark this response, leading to delays and uncertainty.
They may mark by the letter of the scheme and not award marks for a clearly correct answer (to the question that has been asked) or determine that the response given is good enough and award a mark.

All of these are unfair outcomes.

Why the mark scheme should come first

Writing the mark scheme first, clearly outlining how we want students to respond, helps define the question they need to ask.

It's not as simple as 'yes' or 'no'

Most exams are more complicated than just having right or wrong answers.
English, Drama, Psychology, even Computer Science assessments can be complex and prompt a vast array of answers for the same question.
Examiners have to know what they are meant to be looking for in a response.

Has the student identified the right information? Do they show a detailed understanding of the subject matter?

The myriad ways students could do this needs to be included in the mark scheme.
It is necessarily a complex guide that must be clearly constructed to ensure it can be applied fairly.
We must carefully create a ‘levels of response’ mark scheme that identifies the qualities expected in a response.

Do we want to see analysis of language used or application of theories? How do we differentiate between good and excellent analysis?

There also needs to be some indicative content. We know the kinds of things students are likely to refer to in responses so we make it clear to examiners what kinds of content are worthy of credit.
And there has to be room for students to make points we might not have even thought of!

Not just a tick box exercise

What’s important with mark schemes is that we reward the quality of the responses rather than just crediting use of concepts or key phrases students have learned.

Correct application of these concepts and phrases is what must be rewarded to ensure fair and accurate outcomes.

Getting this right has huge implications for the fairness of our exams.

How reviewing mark schemes helps improve exam fairness

We always review our mark schemes looking for improvements.

One things we actively seek to eliminate are redundant marks - marks students can’t get because of the way schemes operate.

A good example is a question such as ‘State the colours of the rainbow in the correct order.’ [8 marks] accompanied by the mark scheme - 1 mark per correct colour in its correct place and 1 mark for completely correct answer

In this example the student cannot score 7 marks.

If any of the colours are missing or in the wrong order, then they would be able to achieve a maximum of 6 marks. If they have all of the correct colours and they are in the correct order then they would be awarded 8 marks.

So, one mark is, essentially, redundant. It represents a wasted opportunity to differentiate between students and their abilities.

A far better approach in this case would be for it to be worth seven marks instead of eight - with the seventh mark being awarded for getting the correct order - and applying a cap of six to any student who makes an error.

So next time you hear about exam marking, don’t assume it’s just putting a tick next to an answer, the thinking around it goes a lot deeper than that.

Through the looking glass: How polling the public can help policymakers learn about themselves

Public attitude data is key to effective policymaking. Proper polling can reveal what people think about existing policies and what they want for the future. But, if looked at from a different angle, it can also help policymakers question themselves and their assumptions about the public. In this blog, AQA’s Policy and Evidence Manager Adam Steedman-Thake, reveals the lessons he learned about himself while reading a recent public attitude survey.

Assessment

Assessing oracy: Is Comparative Judgement the answer?

Oracy skills are vital to success in school and life. And yet, for many children, opportunities to develop them are missed. Educationalists are engaging in a growing debate about where oracy fits into the school system. Labour has put it at the heart of its plans to improve social mobility and an independent commission is looking at how it is taught in the classroom. This renewed focus on oracy means it is more important than ever that teachers have a way to reliably assess and understand their students’ attainment and progression. Amanda Moorghen of oracy education charity Voice 21 explains how Comparative Judgement can help with that and why it may be a game changer.

Education

TV subtitles as an aid to literacy: What does the research say?

Jack Black is probably best known in educational circles for playing a renegade substitute teacher in School of Rock. But the Hollywood star has made a more conventional foray into education by backing the use of TV subtitles to improve child literacy. Stephen Fry and the World Literacy Foundation also want parents to use their TV remotes to get children reading. So, could this simple click of a button be a solution to boost pupils’ reading skills? AQA’s resident expert on language teaching, Dr Katy Finch, casts her eye over the research to see whether it stacks up.

Data Analysis

What is left behind now education’s Data Wave has receded?

Is data the solution to all education’s issues? About a decade ago the prevailing wisdom said it was. Advocates of this Data Wave argued that harvesting internal statistics would help schools solve issues such as teacher accountability and attainment gaps. As with all waves, after crashing onto the beach they recede, leaving space for another to roll in. In this blog, teacher, author and data analyst Richard Selfridge looks at the legacy of the Data Wave to see what schools can take from it.

International Approaches

Finland & PISA – A fall from grace but still a high performer?

Finland was once recognised as one of the most successful educational systems in the world. At the turn of the millennium, it topped the PISA rankings in reading, maths and science. But by 2012, decline set in. The last set of results showed performances in maths, reading and science were at an all-time low. In this blog Dr Jonathan Doherty of Leeds Trinity University outlines some reasons that may account for the slide.

Briefing

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Adaptive Assessment

Adaptive Assessment: A missing ingredient in the resit recipe?

The number of students resitting their maths GCSE is growing, but the proportion getting a grade 4 or higher is falling. This situation is not only dispiriting for the young people striving to get the qualifications they need, but also for the teachers working hard to help them. How can outcomes for this cohort be improved? Bart Crisp, associate director at the Centre for the Use of Research and Evidence in Education, thinks adaptive assessment may be part of the solution.

SEND

Student success: Every milestone matters

Baroness Morgan is calling for students to be given ‘self-belief’ lessons as a way of developing their characters and preparing them for the future. She is not the first to notice that a student’s sense of their own ability and their level of success are part of a virtuous circle. But how can teachers get the snowball rolling for students with SEND or in alternative provision? In this blog, former headteacher, John Tomsett, pulls out a swimming certificate he earned more than half a century ago to use as an inspiration for others.

Assessment

Progress 8 – How much can it flex?

Progress 8 has come in for criticism in recent months for the way it combines with the Ebacc to skew schools’ focus towards academic subjects at the expense of creative and technical options. Both Labour and a Lords committee have called for reform to boost uptake of declining subjects such as D&T, dance and drama. In the second of two blogs looking at Progress 8, AQi examines whether the accountability measure could be realigned to address these concerns and if so, how could that be done?

Assessment

Progress 8 – Schools’ flexible friend?

Eight years after Progress 8 came into existence it is facing calls to be updated. Some say reforming the schools accountability measure can halt the decline in take up of the arts, while others want it to boost numbers taking vocational and technical subjects. Are its shoulders broad enough to take on such a load? In the first of two blogs on Progress 8, AQi looks at how it came into being, what problems it was designed to solve and whether it is flexible enough to be reformed.

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

What comes first – the question or the mark scheme?

Mark scheme first?

Why the mark scheme should come first

It's not as simple as 'yes' or 'no'

Not just a tick box exercise

How reviewing mark schemes helps improve exam fairness

Through the looking glass: How polling the public can help policymakers learn about themselves

Assessing oracy: Is Comparative Judgement the answer?

TV subtitles as an aid to literacy: What does the research say?

What is left behind now education’s Data Wave has receded?

Finland & PISA – A fall from grace but still a high performer?

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

Adaptive Assessment: A missing ingredient in the resit recipe?

Student success: Every milestone matters

Progress 8 – How much can it flex?

Progress 8 – Schools’ flexible friend?

Join the conversation on Twitter

Download a PDF version.

Sign up to our newsletter