In a GRE Trial, AI Tutors Matched the Human Experts

A new study puts a number on something the tutoring business has worried about for a while. Given an hour and a willing student, the better AI tutors produced the same learning gains as expert human ones, and did it for a fraction of a cent on the dollar.

The work, called StudentBench and run by researchers at Handshake AI, recruited 2,383 people and split them at random into three groups. One group got coaching from an AI tutor, drawn from thirteen different large language models. One got an expert human. The third got nothing, as a control. Everyone sat a 27-question GRE pre-test, took a single one-hour session, then sat a matched post-test. The team logged more than 175,000 messages between students and the AI tutors.

What the numbers said

Overall, AI tutoring came out statistically equivalent to expert human tutoring on GRE learning gains. In five of the seven GRE skill areas, the best AI tutor actually edged ahead of the human on average. The human tutors were no amateurs either. They were former ETS or Kaplan staff who had written GRE questions themselves, or coaches with at least five years on the job.

Then there is the cost. One AI tutor matched human learning gains at what the authors calculate as 918 times less expense: about half a cent per percentage point of improvement, against $4.81 for the human. For anyone running a tutoring service, that gap is the whole story.

Read the fine print

It pays to be careful with a result like this. The test was one hour of GRE preparation, a narrow and well-defined task, measured by gains on a test taken straight afterwards. It says nothing yet about whether the knowledge sticks a month later, or whether the same pattern holds for a struggling teenager rather than a motivated adult sitting a graduate exam. Standardised test prep is close to the ideal case for a machine tutor: clear right answers and an endless supply of practice questions, with no need to read the room.

One finding does hint at why the AI did well. In the quantitative sessions, faster replies from the tutor led students to send more messages, more messages led to more correct practice, and more practice led to bigger gains. Patience and speed, in other words, rather than some spark of teaching genius. That is a modest claim, and a believable one. Whether it survives contact with a real classroom is the question the next study will have to answer.