The Weight of the Next Word

Soren's cousin Dara had not slept. This was obvious from the four empty coffee cups arranged in a diagonal across her desk, from the way she kept pushing her glasses up even though they hadn't slid down, and from the fact that she answered the door to the lab seventeen seconds after Soren knocked, which meant she'd been sitting right there all night.

"I broke it," Dara said, by way of greeting.

"The model?"

"The model." She dropped into her chair. "I'm supposed to present Monday. I changed one thing and now the whole thing is giving me garbage. Come in, don't touch anything."

Soren came in. He did not touch anything, but he looked at everything. Dara's monitor showed columns of text, some highlighted green, some red. The green ones looked like normal sentences. The red ones looked like a person having a dream and trying to describe it.

Dara's lab studied large language models. Soren knew what these were. They were neural networks trained to do one thing: given a sequence of words, predict the next one. That was it. That was the entire task. You fed them enormous amounts of text and they learned, word by word by word, what was likely to come next.

"What did you change?" Soren asked.

"I pruned a cluster of neurons in layer forty-one. They weren't supposed to matter. They had low activation on most prompts. I figured they were dead weight." She rubbed her eyes. "They were not dead weight."

Soren pulled up a stool and looked at the screen. Dara had a comparison running. On the left, the original model completing sentences. On the right, her pruned version.

The original model, given the prompt "The capital of France is," completed it with "Paris." The pruned model completed it with "a city of remarkable beauty, situated along the banks of the."

"It forgot Paris?" Soren said.

"It didn't forget Paris exactly. It still knows Paris is a city. Watch." Dara typed a new prompt: "Paris is located in." The pruned model responded: "the northern part of the country, where the river flows through its."

"It won't say France," Soren said.

"It won't say France."

Soren opened his notebook. He wrote: Layer 41 cluster. Low activation. Remove it and the model loses geography. Not the words. The relationships.

Dara was already onto her next theory, muttering about attention heads and residual streams. Soren stayed quiet and read more of the comparison outputs.

The original model could do arithmetic. Not always, but often. Given "Forty-seven plus sixteen equals" it would produce "sixty-three." The pruned model produced "a number that is greater than both." True, technically. But hollow.

The original model, given the beginning of a sentence in French, would continue in French. The pruned model would drift into English by the fourth word, as if it had lost track of which language it was in.

"Dara," Soren said. "What were those neurons actually doing?"

"Predicting the next word. That's all any of them do. That's the only thing the model was ever trained to do."

"But they were doing geography. And math. And tracking grammar across languages."

"No," Dara said, and then stopped. "Well. Yes. But not because anyone told them to. There's no geography module. There's no math module. There's no grammar module. There's just prediction. Next word, next word, next word. And somehow, in order to get better at that one job, the network built all of those things on its own. Inside itself. Without being asked."

Soren stared at the screen. He had known about language models before today. He had not known this.

"Nobody programmed the geography in," he said slowly.

"Nobody programmed anything in. The network was given text. Just text. And it turns out that if you want to be really good at predicting what word comes next, you have to build a model of the world. Not because anyone rewards you for having one. But because the world is what generated the text in the first place."

Soren's pen was not moving. He was holding it above the page.

The words had come from the world. The world had left its shape in the words. And the network, trying to predict nothing but the next word, had been forced to discover that shape. Geography. Arithmetic. Grammar. The structure of reality, pressed into language like a fossil pressed into stone, and the network had found it just by learning what came next.

No one had taught it what a country was. No one had taught it addition. No one had drawn it a map. It had rebuilt these things from scratch because you could not predict the next word well enough without them.

"You look weird," Dara said.

"I'm thinking."

"About?"

"You said the neurons had low activation on most prompts. What prompts did they activate on?"

Dara blinked. She turned to her keyboard. "I didn't check. I just saw low average activation and assumed." She typed rapidly, pulling up logs. "I'm filtering for high activation events in that cluster."

The list populated. Soren read over her shoulder.

The cluster activated strongly when the model was given prompts that required connecting two facts that never appeared together in the training data. The capital of the country where the Eiffel Tower is located. The sum of the number of continents and the number of days in a week. A sentence that began in Spanish and referenced a city in Japan.

"They're the bridge neurons," Soren said.

"The what?"

"They don't store geography or math or grammar. They connect them. When the model needs to use two different things it figured out, those neurons are how it reaches across itself."

Dara was already typing, testing. She gave the pruned model a simple geography question, one fact, no bridge needed. It got it right. She gave it a question requiring two facts linked together. It collapsed into vagueness.

"Oh," Dara said quietly.

Soren wrote in his notebook: It built a map of the world just by predicting words. And when it needed to cross from one part of the map to another, it built roads.

Then he stopped writing. He was thinking about himself. About how he learned things. About how sometimes he would read something about history and something about biology and suddenly they would connect in his head and he would understand both of them differently. Nobody had told him to do that. He just needed the world to make sense, and making sense meant building bridges.

Dara was on the phone now, talking fast to someone in her lab group about the cluster, about bridges, about Monday's presentation.

Soren looked at the blinking cursor on the screen. The model was waiting for its next prompt, ready to predict, ready to reach for the next word in a sequence that no one had finished yet.

He leaned forward. He typed the kind of question he wrote in his notebook late at night, when two ideas had connected in his head and he did not yet have a name for the connection.

He typed: When you learn something from history and something from biology and they suddenly explain each other without warning, what is that called?

The cursor blinked. The model predicted the next word, and then the next.

Sometimes this is called transfer, or analogical reasoning. A pattern recognized in one context is applied to another. The information in both domains becomes more compressible. This is also how generalization works in neural networks.

Soren read it twice. The model had not meant anything by it. It had just predicted words.

He wrote one phrase in his notebook: both domains become more compressible.

Then he opened to a new page.