What Sally Was Built To Refuse

Later this month, in a rural school district on the Seneca Nation reservation in western New York, a humanoid robot named Sally will be unpacked and installed in a classroom.¹ The Salamanca City Central School District paid a Las Vegas company called Realbotix $57,590 for her, a discount from her ordinary sticker price of about ninety-five thousand.² When students return in September, she will help teach roughly five hundred of them in the district's AI and robotics courses, built on a curriculum designed by Apple co-founder Steve Wozniak.¹

Sally sits. She does not walk. She has silicone skin and a range of facial expressions, and she will speak in more than a hundred languages with students asking for homework help, a lesson on a topic they are curious about, or a translation of what a teacher just said.³ She will not collect names, record video or audio, or transmit data back to Realbotix. She has no facial recognition, no open web access, no ability to search anything beyond the closed curriculum the district has loaded onto her.³

Most usefully, she has been programmed to say, when asked a question outside that curriculum, that she does not know.³

The Refusal Is The Story

Almost every AI tool that has entered an American classroom in the last two years has been built to answer. If a child asks a chatbot to explain a math problem, the tool answers. If she asks it to write her essay, the tool answers. If she asks a question the tool is not qualified to answer, the tool answers anyway, often with the same fluent confidence it uses for the ones it is. The economics favor an answer over a silence. The pedagogy is a byproduct.

What Salamanca did, quietly, when it bought Sally, was refuse that default. They took a system with the general-purpose glibness of a modern language model and walled it in. They gave her permission to say, out loud, that she does not know. That is a philosophical position. It says the tool in the room should have visible edges, and a child should be able to see, on her fifth question of the morning, where the tool's competence stops.

The Superintendent's Small, True Sentence

Salamanca's superintendent, Mark Beehler, said something to the press that reads more useful the longer you sit with it. "Many schools are taking the easy solution of simply banning it," he told reporters, "but I have found that students will find a way around most rules that schools put in place. I also believe it is critical for schools to teach proper use of technology, not simply exclude it."⁴

Beehler is speaking from a district that does not have the option of a moratorium. Roughly a third of Salamanca's students are American Indian or Alaska Native, and the school sits on the Allegany Territory of the Seneca Nation.¹ His families are choosing between a school that helps their child understand these tools now and one that pretends the tools are not already, at eleven at night, in every phone in every bedroom in the county. The interesting move is not the decision to bring in an AI. It is the decision to bring in a bounded one.

The Half We Still Cannot See

What Sally will demonstrate for her students is only half of what a modern classroom actually needs. She is a tool that knows its own limits. The student sitting across from her does not, at least not yet. And the school around her has no way, on an ordinary Tuesday, to see when she knew and when she did not.

Say Sally is asked to help a tenth grader with a paragraph on the Iroquois Confederacy. She responds within the corpus. The girl thanks her, edits the sentence, and turns the paper in on Friday. Somewhere in that arc she either wrote the paragraph or accepted the help. The tool has been transparent about its own reasoning. The girl's reasoning is still invisible.

That is the layer we spend our days on at Koan. The quiet substrate underneath the screen where a student's thinking leaves a legible trace. The pauses. The half-drafts. The moment a hint was asked for, and the moment it was not. The revisions kept, and the ones thrown away. What a well-designed tool can now honestly say about itself, the school still cannot honestly say about the student.

Salamanca In September

Five hundred students will meet Sally this fall in one of the country's earliest experiments in what a bounded classroom AI looks like. Some parents will be uneasy. Some students will love it. Most of the coverage will be about the fact of the robot. The more useful conversation is about the design principle. A tool in a classroom should be able to tell you what it knows and, more importantly, what it does not. If we hold classroom AI to that small, radical standard, the country's argument about whether to allow the tools will slowly reshape into an argument about how to see them, and how to see the children beside them.

If the machine has learned to say "I don't know," what would it take for the school to say the same about its students?