The Valley Between

The program was getting worse, and Maya could not stop smiling.

"It's doing it again," Soren said, hunched over the laptop on the gymnasium floor. Around them, other students' science fair displays stood covered in sheets for the night, pale shapes like sleeping ghosts. They had special permission to stay late. Their teacher, Mr. Huang, sat in the bleachers grading papers and occasionally yawning.

"Play it," Maya said.

Soren hit enter. The laptop speakers produced a melody that sounded like a cat walking across a piano while another cat knocked over a glass of water.

"That's worse than the last one," Soren said. He wrote something in his notebook. "Way worse. Yesterday's version was actually decent."

"I know," Maya said, still smiling.

"Why are you happy about this?"

"Because it shouldn't be worse. We made it bigger. We gave it more capacity to learn. More should be more."

Soren looked at the screen. Their project was a small neural network that learned to compose short melodies by studying patterns in a dataset of two hundred folk songs. The version from two days ago, the simple one with fewer parameters, had produced tunes that were basic but pleasant. So they'd made the model larger. More parameters. More room to learn.

And the music had collapsed into garbage.

"We overfitted," Soren said. "That's the textbook answer. The model memorized the training songs instead of learning how music works. It's just regurgitating fragments."

"Play one of the training songs," Maya said.

Soren queued one up. A familiar folk melody came through the speakers, perfect, note for note.

"Now play one it generated."

The cat-piano sound again. Soren winced.

"See?" he said. "It can reproduce exactly what it saw. It can't make anything new. Classic overfitting. We need to make the model smaller, or get more training data, or add regularization. That's what the textbook says."

"What if the textbook is wrong?" Maya said.

Soren looked at her. He had known Maya long enough to know that smile. "Wrong how?"

"Wrong about what happens next. What if we make it bigger?"

"We just made it bigger. It got worse."

"So we make it much bigger."

Soren opened his mouth, then closed it. He wrote something in his notebook. Then he said, "That's the opposite of every principle we learned. The bias-variance tradeoff. You find the sweet spot. Too simple, it underfits. Too complex, it overfits. You don't keep going past overfitting. There's nothing past overfitting."

"Maybe there is," Maya said. She sat down cross-legged next to him. "Think about it. What does the model do when it has just barely enough parameters to memorize everything?"

"It memorizes everything."

"Right. It uses all its capacity just to hold the data. Every parameter is busy remembering. There's no room left for anything else. But what if there are way more parameters than data points? Like, way more?"

Soren was quiet for a moment. He tapped his pen against the notebook. "Then the model has extra capacity. It doesn't need to use all of it just to memorize."

"So what does it do with the extra?"

Mr. Huang called from the bleachers. "You two need a ride home soon? My wife's patience has a half-life of about forty minutes."

"Twenty more minutes," Maya called back.

"Make it fifteen," Mr. Huang said, and went back to his papers.

Soren was already typing. "I'm tripling the parameters. No. I'm going ten times."

"Go fifty," Maya said.

"Fifty times. That's absurd. That's millions of parameters learning from two hundred songs."

"Do it."

He did it. The training took seven minutes while they sat on the cold gymnasium floor and listened to the ventilation system hum. Around them, the sheet-covered displays waited. Someone had built a volcano. Someone else had grown crystals.

Maya and Soren were asking what happens on the other side of a valley that wasn't supposed to exist.

The training finished. Soren's hand hovered over the key.

"It's going to be worse," he said. "Every part of my brain that understands statistics says it's going to be worse. You can't keep adding complexity past the point of memorization and expect it to get better. That's not how the tradeoff works."

"Play it," Maya said.

He pressed enter.

The melody that came out was not a folk song. It was not a copy of any folk song. It was something else entirely. It had the shape of the folk songs, the way they breathed and paused and resolved, but the notes were new. The phrasing was new. It sounded, somehow, like something a person might actually hum.

Soren played it again.

Neither of them spoke.

He generated another one. Different melody, same quality. Then another. Each one was distinct, and each one sounded like music.

"That's not possible," Soren said, but he was writing furiously.

"It went down," Maya said quietly. "And then it came back up."

Soren drew it. A curve in his notebook. Performance on one axis, model size on the other. Up, as the simple model learned. Then down, as the model hit the memorization zone. Then, past a valley that classical statistics said was the end of the story, up again. Better than before. Better than anything before.

"It's like there are two different ways to fit data," he said. "The first way is finding a simple pattern. The second way, past memorization, is finding something deeper. Something that only shows up when you have so much capacity that memorization is trivial and the model has to find structure just to organize itself."

"Like it has so much room that brute memorization is the hard way," Maya said. "And the easy way is actually understanding the music."

Soren stared at the curve he'd drawn. "Everything in the textbook says this valley is the end. It says you stop here, or you go back. Nobody tells you what happens if you keep going."

"Somebody did, though," Maya said. "Researchers found this. They called it double descent."

"You knew?"

"I read something. I didn't understand it until right now. I didn't understand it until I heard the music get worse and thought, what if worse isn't the end?"

Mr. Huang stood up in the bleachers, gathering his papers. "Five minutes, scientists."

Soren looked at the curve in his notebook. The valley in the middle. The unexpected rise on the far side. He thought about every time he had stopped when something got harder, assuming harder meant wrong. He thought about the place past the worst part, where the worst part turned out to be a valley between two peaks, and the second peak was higher than the first.

Maya hit the key one more time.

A new melody filled the gymnasium, finding its way up into the steel rafters, past the sleeping volcanoes and careful crystals, and neither of them moved until the last note finished echoing.