After the Flat Line

Maya and Soren built a little universe out of ninety-seven numbers.

It lived on the lab screen as a circle, not because circles were pretty, but because after ninety-six came zero again. Add forty to seventy, walk around the circle, and you landed on thirteen. Add five to five, and you landed on ten. Every answer was just a step on the ring.

The neural network did not get a ring.

It got examples.

Soren fed it rows of numbers from the training set, two numbers in, one number out. Maya named the network Pigeon because it pecked at mistakes one at a time. The server fan made a soft, feathery sound under the table.

At step eight hundred, Pigeon answered every training example correctly.

The lab wall lit up green.

Maya jumped once. Soren did not jump, but his pencil made a sharp black dot in his notebook.

Ms. Lio looked over from the soldering bench, where she was repairing a greenhouse drone with her glasses pushed onto her forehead.

“Done?” she asked.

“Training is done,” Soren said.

“Testing is red,” Maya said.

On the second wall, the hidden examples glowed like warning berries. Pigeon had memorized the sums it had seen. Give it a sum it had not seen, and it guessed like someone throwing socks into the dark.

Ms. Lio wiped solder from her thumb. “That is overfitting. It happens. Make it smaller or give it more data.”

“We are not allowed more data,” Soren said. “That is the point of the challenge.”

“The point of the challenge is to make a model that generalizes,” Ms. Lio said. She glanced at the queue board. “Also the point of my evening is to get this drone flying before the basil freezes.”

Maya was still looking at the green wall.

“It stopped changing,” she said.

“Yes,” Ms. Lio said. “Because it learned the training set.”

“No,” Maya said. “It stopped changing where we are measuring.”

Soren turned his pencil sideways between his fingers. “That is not the same sentence.”

Ms. Lio gave them the look adults gave when they had decided to be patient for exactly twelve more seconds.

“The server shuts jobs off when training is perfect,” she said. “Perfect training plus bad testing is usually a dead end.”

“Usually,” Maya said.

Ms. Lio sighed and pointed her soldering iron at the little server under the table. “Midnight to morning. One run. Low power. If the basil dies, I am naming a wilted plant after you.”

At midnight, the big lights clicked off. The lab became blue with screen glow and aquarium light from the algae tank near the window.

Soren changed the stop rule.

Do not stop at perfect training.

Keep going.

Maya read it and nodded. “Good.”

“That is not a reason,” Soren said.

“It is a doorstop,” Maya said. “It keeps the door from shutting.”

The first hour was boring in a way that made the room feel rude.

Training stayed perfect. Testing stayed terrible. The loss number, which Ms. Lio cared about, slid lower by tiny amounts that did not change a single red square. Soren wrote the steps down until the numbers began to look like rain.

Maya left her chair and came back with two cups of water and a banana from the lab snack box.

At step three thousand, nothing happened.

At step six thousand, nothing happened.

At step nine thousand, nothing happened so completely that Soren began to suspect nothing was a thing with weight.

He opened the map tool to keep himself awake. It showed the network’s hidden layer as dots, squashed down into two dimensions so humans could look at it. The dots floated in loose clumps, not quite random, not quite anything. Like a crowd waiting for a bus that had not been announced.

Maya leaned in.

“That one moved,” she said.

“They all move a little.”

“No. That clump stretched.”

Soren saved the map image, then waited five hundred steps and saved another.

The red test wall did not change.

The training wall did not change.

But the dots in the hidden map were tugging themselves into a curve.

Soren’s mouth went dry.

“It is still wrong,” he said.

“I know,” Maya said.

“The only thing changing is the part we do not grade.”

“I know.”

He looked at her. “You like that too much.”

Maya smiled without looking away from the screen.

At step twelve thousand, the curve was almost a loop.

At step fifteen thousand, the test wall was still red.

Soren had a feeling he did not like, because it did not fit any graph he knew. Learning was supposed to show itself. First a little less wrong, then less, then less, like walking down stairs. This was a hallway with no doors.

He checked the code again. The training examples were separate from the test examples. The answers were calculated correctly. The model was small. The optimizer was running. The regularization was on, gently pressing the weights smaller at every step.

Nothing was broken in the ordinary way.

Maya tapped one red square on the test wall.

“Ask that one again,” she said.

“It will be wrong.”

“Ask it.”

Soren typed the hidden sum. Forty-one plus eighty-nine.

Pigeon answered thirty-three.

The true answer was thirty-three.

The square turned green.

Soren sat up so fast his chair squeaked.

“Again,” Maya said.

He typed another hidden sum.

Green.

Another.

Green.

Then the wall changed before he could type. Not one square. Not a row. A scatter first, like sparks catching. Then columns. Then the whole hidden test set flushed green so quickly the lab seemed to blink.

The server fan kept making its small feather sound.

Maya did not cheer. Soren did not write. For several seconds, both of them only watched the wall that had been red for hours.

Ms. Lio came running from the greenhouse bay with a drone propeller in one hand.

“What caught fire?” she asked.

Soren pointed.

Ms. Lio looked at the green testing wall. Then at the step counter. Then at the training wall, which had been perfect since eight hundred. Her face changed in small pieces.

“Oh,” she said.

Maya asked, “Is there a bug that makes everything green?”

Ms. Lio put the propeller down very carefully. “There are many bugs. Check.”

They checked.

Soren made a fresh test set the network had never seen. Maya chose sums by walking her fingers around the number circle with her eyes closed. Ms. Lio was not allowed to say any answers. The lab calculator checked them.

Green. Green. Green.

Pigeon had not learned the list. It had found the circle.

Ms. Lio sank into the chair beside them. “People call this grokking,” she said. “When a network looks like it is only memorizing, sometimes for a very long time, and then it suddenly generalizes.”

“Sometimes?” Soren asked.

“Sometimes,” Ms. Lio said.

“Do we know why it waits?” Maya asked.

Ms. Lio looked at the loop of dots on the map tool. “Not completely.”

it for thousands of steps with almost no sign on top.

Soren thought of all the times a teacher had looked at his blank answer box and said he was not trying yet. Maya thought of adults saying, “Slow down and explain,” when the answer had arrived in her before the path to it. Neither of them said these things. The server said enough, softly, under the table.

Maya opened the hidden-layer map again.

The dots had changed while no one was looking at the score. They were not perfect. They were not a drawing. But the numbers with the same wrapped-around rhythm sat near one another, and the whole cloud bent around an empty center.

Soren reached toward the screen, stopped before touching it, and traced the air above the curve.

“Ninety-seven places,” he said.

Maya dragged the map wider. Soren leaned close until his breath clouded the glass. On the dark screen, ninety-seven pale dots held their circle.