The Sentence That Knew Who It Was Talking To

The coding club had emptied out, and the radiator was making its end-of-day clicking noise. Soren had not gone home because the laptop someone left behind was still warm, still running the little language model from the afternoon demo.

"Type something," Maya said.

Soren typed: When Anna and Ben went to the store, Anna gave a drink to.

The model finished it. Ben.

"Okay," Maya said. "It knows grammar."

"It's not grammar though." Soren was already frowning at the screen the way he frowned at a bike chain that skipped. "Grammar doesn't tell you it's Ben. Anna gave a drink to Anna is a perfectly fine sentence. It just isn't true here."

Maya leaned in. "Do it again with different names."

When Maya and Soren went to the lab, Maya handed a notebook to. Soren.

They both went quiet.

"It picked the other one," Maya said. "Every time. The one that wasn't repeated."

"That's a rule," Soren said. "Whoever already showed up gets ruled out. The leftover name wins."

The demo guy that afternoon had said the model was three years of math poured into a file, billions of little numbers, and that nobody had told it any rules at all. It had read a mountain of text and figured out whatever it figured out. Which was supposed to make it a black box. A thing that worked and nobody knew why.

"So how does it know to rule out the repeated one," Maya said, "if nobody put a rule in."

"It's a black box," Soren said, but he said it like he didn't like the taste of the words.

Maya pulled the laptop toward her. The afternoon demo had left a panel open, the boring one nobody clicked, that let you peek at what the layers inside were doing. Numbers stacked on numbers. She scrolled.

"What are you looking for," Soren asked.

"Don't know yet. The place where it decides."

Soren watched her scroll and started thinking out loud, which was how he thought. "If there's a rule, the rule has to happen somewhere. It has to look at Anna, notice Anna already happened, and then push Anna down. Push, like, lower the chance of saying Anna."

"Push it down," Maya repeated. "So somewhere in there is a thing whose whole job is to say not that one."

They ran the sentence again with the panel watching. Anna and Ben. Anna gives. Most of the layers were a fog. But near the middle, a small cluster of numbers lit up hard, and one of them in particular went strongly negative right when the name Anna came around the second time.

"There," Maya said. She put her finger on the screen, on the one number. "That one hates Anna."

"It doesn't hate Anna." Soren leaned in. "Run it with Ben repeated. If I'm right, the same spot should turn on Ben instead."

When Ben and Anna went out, Ben gave the keys to.

The model said Anna. And the same small cluster lit up, the same number went negative, except this time it went negative on Ben.

"It's not about Anna," Soren said slowly. "It's not about any name. It's a thing that finds whichever name already showed up, and pushes it down. That exact job. A little machine for one job."

Maya sat back. "Somebody built that."

"Nobody built it. That's the part. It built it. It read a billion sentences and somewhere in all that reading it grew a piece that does the leftover trick, and it's the same piece every time, in the same place."

"So it's not a black box." Maya was talking fast now. "It's a box full of little machines and we can open them. There's a machine for the leftover-name thing. There has to be a machine for other things too. A machine for plus. A machine for is this a city."

"You can't know that," Soren said.

"You just watched one," Maya said. "You made it light up on purpose. You guessed Ben would flip it and Ben flipped it."

Soren stopped. That was true. He had made a guess about a thing inside a mind nobody understood, and the thing had done exactly what he guessed.

"Try a number one," he said.

Maya typed: Seven plus six equals.

Thirteen.

"Now watch where," she said, and reset the panel.

This time a different cluster woke up, off to the side, nowhere near the name spot. It pulsed in a tidy little pattern, the same pattern, over and over, like a wheel coming around.

"That's not the name place," Soren said. "That's a whole different neighborhood."

"Because it's a different machine." Maya's voice had gone almost careful. "The adding one lives over here. The name one lives over there. They have addresses."

Soren had his notebook open and his pen was moving, copying the shape of the little wheel-pattern, the way it came around and came around.

"You know what's weird," he said, not looking up. "It taught itself to add the way a person adds. Not a calculator way. It made a wheel. People who study these found the wheel. It's a real thing people found."

"So when it does seven plus six," Maya said, "it's not faking. There's an actual adding happening in there, in a spot, that we could point to."

"We did point to it."

The radiator clicked. Down the hall a custodian's cart squeaked past and faded.

Maya looked at the screen, at the two neighborhoods, the name one and the number one, sitting in the same warm laptop with all the fog between them.

"It's full of rooms," she said quietly. "Nobody drew the rooms. It grew the rooms. And the same rooms grow in the same places every time, so they must be the rooms you'd have to grow. Anything that learns to read might have to grow a leftover-name room. Maybe we have one."

Soren's pen stopped.

"Type a person," he said. "Type a sentence and let's find the room where it decides what we are to it."

Maya put her hands on the keys. Then she lifted them off, and turned the laptop a little, so the open panel faced them both, and started to type.