Worse Before Better

The club room smelled like solder and old carpet. Everyone else had gone home. Mr. Okafor had left the laptop with them and one instruction: get the little handwriting reader working, lock the door when you're done.

"It's getting worse," Maya said. She was staring at the number on the screen. Test accuracy. It had been eighty-one percent. Now it was seventy-four.

"We made it bigger," Soren said. "More neurons. It should be smarter."

"It's dumber."

They had a folder of five hundred handwritten fours and nines, scanned from the club's own notebooks. The model was supposed to look at a digit it had never seen and say four or nine. Small model, small brain. They kept adding to the brain, and the little brain kept doing better, right up until it didn't.

"Run it again," Soren said.

Maya ran it again. Same thing. The bigger they made the model, the better it got, then the graph bent and slid downhill.

"There," Soren said, pointing. "Right there. That's the worst it gets. That's the exact size where it goes bad."

Maya leaned in. At that size the model had almost exactly as many knobs to turn as there were digits to learn. Five hundred-ish of each.

"It memorized them," she said slowly. "All five hundred. Perfectly. Zero mistakes on the ones it studied."

"So why is it terrible on the new ones?"

"Because it only has enough room to memorize. Not enough room to notice anything." Maya tapped the trackpad. "It's like it wrote down every answer to the practice test and nothing else. First new question, it panics."

Soren wrote the size down. He drew a small dip on the page, a valley, and marked the bottom of it. "Okay. That makes sense. So this is the wall. This is as big as we can go."

"We stop here?"

"We stop before here. We back off, keep it small." He was already reaching to shrink the model back down.

"Wait." Maya put her hand flat on the desk, not the laptop. "Don't go smaller. Go bigger."

Soren looked at her. "It's at the bottom of the hole, Maya. Bigger is deeper."

"Is it, though."

"That's what the graph says."

"The graph says up to here." She dragged her finger past the valley, off the edge of Soren's drawing, into blank paper. "It doesn't say anything about there. Nobody made it go there."

Soren didn't argue. He never argued when she pointed at the place nobody had checked. "What made you think that?"

"Because it's already memorized everything at the bottom of the hole. It can't get more perfect on the practice test. So if we keep adding room, the extra room can't go into memorizing." She shrugged, a little helpless with it. "It has to go somewhere else. I don't know where. Somewhere."

"Somewhere," Soren repeated. He liked the word better than she expected him to. "One way to find out."

He made the model bigger. Way past the wall. Twice the knobs, then four times, then ten. Each run took a minute. The fan on the laptop started to whine.

First run past the valley: seventy-one percent. Worse.

"See," Soren said, but quietly, without any triumph in it.

"Keep going."

Second run: seventy-two.

Third: seventy-six. Then she said, "Do the next one."

Fourth run, the model now enormous, far more knobs than there were digits in the whole folder: eighty-eight percent.

Soren sat back. "That's better than eighty-one. That's better than it ever was."

"Do a huger one."

Ninety-one percent.

The two of them stared at the screen. The graph on Soren's paper was wrong now. He had drawn one valley and stopped. The real shape kept going: down into the valley, up the far side, higher than where it started. Two hills with a ditch between them.

"That's not supposed to happen," Soren said. "Everybody knows the rule. Too small, it's dumb. Just right, it's smart. Too big, it memorizes and gets dumb again. Up, then down. One hill." He tapped his own drawing. "That's the rule. My uncle does statistics. That is the actual rule."

"It went down and then up again."

"I know."

"It got worse and then it got better."

"I know." He was writing fast now, the size of every model, the accuracy of every run, the exact spot where the ditch was. His hand couldn't keep up with the runs. "Why. Why would more room make it smarter after it already knew everything."

Maya was quiet, chewing on it. "When it's just barely big enough, there's only one way to memorize the practice test. One ugly way. It has to bend itself into a horrible shape to hit every single answer, and that horrible shape is useless on anything new."

"And when it's huge?"

"When it's huge there's a million ways to get the practice test perfect." Her eyes were wide. "A million. And out of the million, it doesn't pick an ugly one. It picks a smooth one. A simple one. Like it would rather be simple if it's allowed to be."

"Why would it rather be simple?"

Maya opened her mouth. Closed it. "I don't know," she said. "That's the part I don't know."

Soren looked at her, and he was almost laughing, but not because it was funny. "That's the best part, then."

"That's the best part," she agreed.

Because nobody in the room knew. Not Mr. Okafor, who had gone home. Not Soren's uncle who did statistics for a living and would have said one hill, and would have been wrong, and wouldn't have believed them until he ran it himself. The laptop had done a thing that the actual rule said it could not do, and the reason it did the thing was sitting right there in the blank part of the paper where nobody had drawn anything yet.

"Run the biggest one it can do," Maya said. "The absolute biggest. I want to see how high it goes."

Soren set the model as large as the whining little laptop would hold and pressed the key. The fan roared. On the screen a single bar began to climb, digit by digit, past every number they had ever gotten, into a place neither of them had drawn a line for yet.