What The Grade Did Not Register

On July 28, The Hechinger Report covered the first randomized field trial of a generative AI teaching assistant in real classrooms.¹ The working paper had gone up on SSRN a month earlier, under the title Generative AI Can Harm Teaching.² The authors were Alp Sungu of the Wharton School, Benjamin Lira, and Angela Duckworth, best known outside the academy for the word grit. The trial ran ten weeks across fourteen middle and high schools in Turkey. One hundred ninety three teachers. Two thousand eight hundred sixteen students. Half the teachers were randomly assigned a customized GPT-4o assistant, with Turkish Ministry of Education materials baked in. Half were left with what they had.²

When the ten weeks ended, average grades in the two groups were essentially the same. What was different was harder to grade.

What The Students Could Feel

Students of the AI-using teachers rated their classes as less enjoyable, less interesting, and less important, by about a tenth of a standard deviation.² That is a small number. It is not, at first glance, a headline. It is the number a superintendent would set aside on a Friday and forget by Monday.

Sit with it. A tenth of a standard deviation, systematically, across a semester, across almost three thousand students, in the direction of less. The students could not have named what they were noticing. They only knew, in the way students always know, that something in the room felt lighter. The teacher was still there. The lesson still landed. But something had been done somewhere else that in another September would have been done inside her.

Where The Grade Did Move

Average achievement did not change. That is the sentence the tool's makers will quote. Inside the average, one place the grade did move was among the students of teachers who had been weaker before the study. Exam scores dropped there, and student confidence dropped with them.¹² The teachers who most needed the tool to compensate were the ones whose classrooms it seems to have quietly hollowed out.

The lead author, Alp Sungu, put it this way. Teachers, just like students or coders, might be using AI as a crutch.¹ The metaphor is worth pausing over. A crutch is not the injury. It is the thing that lets you avoid rebuilding the leg.

What The Preparation Held

The teachers in the trial used the tool for what any teacher would use it for. Lesson planning. Assessment. Feedback. Differentiated instruction. Administrative communication.³ The list reads like the pitch deck of every ed tech company launching this month.

What the pitch deck does not describe is what happens to the teacher during those hours. The lesson plan is not only preparation for what she will say. It is what happens to her while she prepares. She rereads the passage. She notices, again, what tripped her the first time she taught it. She invents the analogy. She discards it. She picks a different one. By the time she stands at the board, she is carrying something the plan itself does not print. When the tool writes the plan, the plan appears. The rest, quietly, does not.

The students in the Turkish trial did not read the lesson plans. They stood in the room the teacher had, or had not, prepared herself inside. That is what a tenth of a standard deviation measured.

The Layer Under The Lesson Plan

The temptation with a number like 0.11 is to shrug and keep the tool. Faster planning. Same grades. Everyone can go home earlier. The temptation with the weaker teacher finding is to say, quietly, that the tool needs better onboarding. Both readings miss what the paper is actually about. The trial did not conclude that AI is bad for teaching. It found that the thing being lost was not measured by the same instruments that measured the grade.

At Koan we work on the layer under student work. The pause. The revision. The paragraph deleted and rewritten in different words. The moment a student pasted something in and then translated it back into her own language, and the moment she did not. What the Turkish trial makes clear is that the same layer exists under the teacher's work. The reread. The discarded analogy. The Tuesday afternoon spent sitting with what the class got wrong last week. Nothing about the finished plan will tell an administrator whether that layer was present. The class will tell them, in a tenth of a standard deviation nobody set up to hear.

If your students felt the difference between a lesson their teacher wrote and a lesson their teacher approved, would you have any way, other than the grade, to know?