Assessment

Automated Marking: What do driverless cars and auto-markers have in common?

Rapid developments in artificial intelligence are fuelling huge leaps forward in Large Language Models that underpin chatbots. Popular natural language processing tools such as ChatGPT4 are now so sophisticated they are beginning to be used in the exam marking world. Just like driverless cars they’re not infallible so are not used in England and will not replace humans any time soon. So where are we with Automated Marking? AQA researcher and AI expert Dr David West gives an overview of the technology, its pros and cons, how it is currently used and where it might go next.

Date27/06/23

AuthorDr David West

Most modern cars do not use all the driverless technology available much like Automated Marking and Natural Language Processing tools

What is Automated Marking?

Let’s start by trying to clarify what is a complicated area of computing and assessment.

Automated Marking systems are built using Large Language Models (LLMs) like the ones underpinning ’generative’ AI such as ChatGPT.

However, because these conversational models are built with such a broad range of information – some that will be out of date or plain wrong – they generate human-like responses that seem plausible but may be incorrect or irrelevant.

Automated Marking (AM), on the other hand, uses a ‘predictive’ AI aimed at far greater accuracy.

Researchers building AM use thousands of relevant student responses to specific assessment items to fill an AI framework for marking further work from across the grade spectrum. Markers notes and mark schemes could also be added.

The AM identifies language patterns and features associated with different levels of subject proficiency.

From this it could analyse and score unmarked writing.

The strengths of Automated Marking

It would be as if a human has memorised the work of a panel of the subject’s top examiners.

In addition, they have instant recall of all the scripts and mark schemes and, to top it off, can use that information to process a paper in seconds.

No human could match AM for speed.

And Automated Marking's weaknesses?

Conversely, there are some things humans can do which machines can’t.

Auto-markers are not great for nuanced marking above functional level subject assessment.

Experienced markers, unlike AM, can distinguish between a long jumble of relevant jargon and a coherent, logical and accurate open form answer.

This relative unsophistication means we are not looking to replace humans with AM but rather to assist them.

So what should we do with Automated Marking?

A good analogy for Automated Marking are driverless cars.

Technology exists to replace drivers' feet, hands and eyes, but currently it struggles with split-second ethical decisions humans face while driving.

The accountability chain in accidents involving AI also becomes complicated.

This is why the adoption of driverless technology more often than not focuses on smaller features like Lane Assist or Automated Parking aids, rather than replacing the driver wholesale.

Similarly, AM has a raft of abilities that can support human markers with specific tasks.

How can Automated Marking benefit students?

If used strategically, it should be able to improve marking efficiency, reliability, and transparency, all of which ultimately benefits students.

Automated Marking is efficient

Let's examine these three goals starting with efficiency.

AM excels in grading short factual answers and Selected or Closed Response items such as multiple choice questions.

Because there is no level of sophistication in these, it can read and compare responses with mark schemes to see easily which requirements have been met.

Where AM would struggle to mark as consistently as a human is with more complex and nuanced open form responses that can come in many guises.

Relieving humans of the simpler tasks could free them up to focus on assessing open response items.

How reliable is Automated Marking?

Next, there's reliability.

Consistent marking and scoring are essential for any assessment.

Current systems effectively support this goal.

Senior examiners monitor markers and correct anomalies such as keying errors or deviations from the mark scheme.

AM could be used to target key areas for greater scrutiny.

Its ability to cluster thousands of marked responses for any specified item could make it easier to identify aberrations such as drifting off the mark scheme and human error.

Automated Marking as a tool for transparency

Finally, transparency. This is crucial.

Students want to know that their marking and grades are accurate and, in the case of an appeal, exam boards want to show that they’ve got it right first time.

Visually highlighting key features of the mark scheme and linking it to corresponding sections in a response could be relatively simple for auto-markers.

This would provide insight into how scores were assigned.

This transparency could be shared with students which hopefully will boost confidence in marking and reduce the demand for appeals.

The future of Automated Marking

So, what does all this mean for the future of AM?

Well, we've identified some key trust and accountability concerns to the point where we know AM’s benefits and limitations.

Now, we’re exploring and improving these three ways in which it might be used to the benefit of examiners and the existing system.

Having determined where we want to go with Automated Marking places us in an excellent position to progress and achieve that.

Assessment

Assessing oracy: Is Comparative Judgement the answer?

Oracy skills are vital to success in school and life. And yet, for many children, opportunities to develop them are missed. Educationalists are engaging in a growing debate about where oracy fits into the school system. Labour has put it at the heart of its plans to improve social mobility and an independent commission is looking at how it is taught in the classroom. This renewed focus on oracy means it is more important than ever that teachers have a way to reliably assess and understand their students’ attainment and progression. Amanda Moorghen of oracy education charity Voice 21 explains how Comparative Judgement can help with that and why it may be a game changer.

Education

TV subtitles as an aid to literacy: What does the research say?

Jack Black is probably best known in educational circles for playing a renegade substitute teacher in School of Rock. But the Hollywood star has made a more conventional foray into education by backing the use of TV subtitles to improve child literacy. Stephen Fry and the World Literacy Foundation also want parents to use their TV remotes to get children reading. So, could this simple click of a button be a solution to boost pupils’ reading skills? AQA’s resident expert on language teaching, Dr Katy Finch, casts her eye over the research to see whether it stacks up.

Data Analysis

What is left behind now education’s Data Wave has receded?

Is data the solution to all education’s issues? About a decade ago the prevailing wisdom said it was. Advocates of this Data Wave argued that harvesting internal statistics would help schools solve issues such as teacher accountability and attainment gaps. As with all waves, after crashing onto the beach they recede, leaving space for another to roll in. In this blog, teacher, author and data analyst Richard Selfridge looks at the legacy of the Data Wave to see what schools can take from it.

International Approaches

Finland & PISA – A fall from grace but still a high performer?

Finland was once recognised as one of the most successful educational systems in the world. At the turn of the millennium, it topped the PISA rankings in reading, maths and science. But by 2012, decline set in. The last set of results showed performances in maths, reading and science were at an all-time low. In this blog Dr Jonathan Doherty of Leeds Trinity University outlines some reasons that may account for the slide.

Briefing

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Adaptive Assessment

Adaptive Assessment: A missing ingredient in the resit recipe?

The number of students resitting their maths GCSE is growing, but the proportion getting a grade 4 or higher is falling. This situation is not only dispiriting for the young people striving to get the qualifications they need, but also for the teachers working hard to help them. How can outcomes for this cohort be improved? Bart Crisp, associate director at the Centre for the Use of Research and Evidence in Education, thinks adaptive assessment may be part of the solution.

SEND

Student success: Every milestone matters

Baroness Morgan is calling for students to be given ‘self-belief’ lessons as a way of developing their characters and preparing them for the future. She is not the first to notice that a student’s sense of their own ability and their level of success are part of a virtuous circle. But how can teachers get the snowball rolling for students with SEND or in alternative provision? In this blog, former headteacher, John Tomsett, pulls out a swimming certificate he earned more than half a century ago to use as an inspiration for others.

Assessment

Progress 8 – How much can it flex?

Progress 8 has come in for criticism in recent months for the way it combines with the Ebacc to skew schools’ focus towards academic subjects at the expense of creative and technical options. Both Labour and a Lords committee have called for reform to boost uptake of declining subjects such as D&T, dance and drama. In the second of two blogs looking at Progress 8, AQi examines whether the accountability measure could be realigned to address these concerns and if so, how could that be done?

Assessment

Progress 8 – Schools’ flexible friend?

Eight years after Progress 8 came into existence it is facing calls to be updated. Some say reforming the schools accountability measure can halt the decline in take up of the arts, while others want it to boost numbers taking vocational and technical subjects. Are its shoulders broad enough to take on such a load? In the first of two blogs on Progress 8, AQi looks at how it came into being, what problems it was designed to solve and whether it is flexible enough to be reformed.

Assessment

2023: A year in review

As we wrap up a year full of exciting developments in education, it is the perfect time to take stock of what has gone on. In this blog, editor of AQi, Anil Dawar, reflects on some of his highlights for 2023.

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

Automated Marking: What do driverless cars and auto-markers have in common?

What is Automated Marking?

The strengths of Automated Marking

And Automated Marking's weaknesses?

So what should we do with Automated Marking?

How can Automated Marking benefit students?

Automated Marking is efficient

How reliable is Automated Marking?

Automated Marking as a tool for transparency

The future of Automated Marking

Assessing oracy: Is Comparative Judgement the answer?

TV subtitles as an aid to literacy: What does the research say?

What is left behind now education’s Data Wave has receded?

Finland & PISA – A fall from grace but still a high performer?

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

Adaptive Assessment: A missing ingredient in the resit recipe?

Student success: Every milestone matters

Progress 8 – How much can it flex?

Progress 8 – Schools’ flexible friend?

2023: A year in review

Join the conversation on Twitter

Download a PDF version.

Sign up to our newsletter