What the Network Forgot

The graduate student's name was Priya, and she had explained the neural network to Soren in the elevator, speaking fast, not quite looking at him.

"It learned to recognize handwritten digits," she said. "Trained on sixty thousand examples. Seven hundred thousand weights. Each weight is just a number, a strength of a connection, and together they encode everything the network knows about what a three looks like versus a five."

She had a coffee in each hand and a bag on one shoulder. The elevator doors opened. She walked fast.

"Don't touch the compression slider," she said, and then she was gone around a corner.

Soren sat down in front of the terminal. He had his notebook open to a blank page. He wrote: seven hundred thousand weights. He drew a quick sketch of what a neural network looked like from the diagrams Priya had shown him in the hallway: layers of circles connected by lines, the lines being the weights, each one a small number that either boosted a signal or weakened it or ignored it entirely.

Seven hundred thousand of those.

He wrote: what does it mean for a number to know something?

He did not have an answer. He moved on.

The terminal had a simple interface. There was a canvas where he could draw a digit with his mouse. There was a bar showing the network's confidence. He drew a three. The bar snapped to 99.2% for THREE. He drew it sloppy, with a tail at the bottom. 97.8%. He drew it backward, like a capital E. 61.3% for THREE, 34.1% for EIGHT.

He drew it as badly as he could, just two bumps and a line. 89.4% for THREE.

He wrote: it knows what a three is even when I barely do.

Then he looked at the compression slider.

Priya had said don't touch it. She had not said why. There was a small label above it: ACTIVE WEIGHTS. It showed a number: 700,412. Below it, a percentage: 100%.

Soren looked at the door. He looked at the slider.

He moved it to 90%.

The number dropped. 70,041 active weights. The other six hundred and thirty thousand had been switched off, zeroed out, removed from the calculation entirely.

He drew a three.

98.8% for THREE.

He put his pencil down.

He drew it with the tail. 97.1%. He drew it sloppy. 96.5%. He drew it as two bumps and a line. 88.9%.

Almost identical.

He moved the slider to 50%.

Three hundred and fifty thousand weights. He drew a three. 97.4%. He drew the bad one. 93.1%.

He moved it to 20%.

One hundred and forty thousand weights. He drew a three. The bar said 95.9%.

Soren stopped and stared at that number for a long time.

He moved the slider to 10%.

Seventy thousand weights. He drew a three.

94.2% for THREE.

He moved it to 5%.

Thirty-five thousand weights out of seven hundred thousand.

91.7% for THREE.

He moved it to 2%.

Fourteen thousand weights.

He drew a three very carefully, trying to be fair.

87.3% for THREE.

He drew the bad E-backward three.

59.1% for THREE, 30.8% for EIGHT.

He moved the slider to 1%.

Seven thousand weights out of seven hundred thousand.

He drew a clean three.

78.4% for THREE.

He wrote in his notebook: 1% of the weights. 78% accurate.

Then he sat back and looked at that sentence and felt the floor do something.

Not fall. Just shift.

He had thought about memory before, the way everyone thinks about it: as a place. Things go in, things stay, things come out. You need a certain amount of space. That was the model. If you have a book with seven hundred thousand sentences and you cross out six hundred and ninety-three thousand of them, you no longer have the book.

But the network did not work that way.

Somewhere in those seven hundred thousand weights, the knowledge of what a three looked like was not spread out evenly. It was bunched. Clumped. Concentrated in maybe thirty thousand connections out of seven hundred thousand, and the rest were doing something else, or doing a little of the same thing redundantly, or doing almost nothing at all.

He wrote: the knowledge lives in one corner of the room.

He wrote: so what is the rest of the room for?

He did not write an answer because he did not have one. Priya would probably know. Or she would know who to ask. Or she would say it was an open question and give him that look researchers give when something is actually unsettled, not the look teachers give when they mean you should already know this.

He moved the slider back to 100% because that seemed like the right thing to do.

He drew one more three, just to check.

99.1%.

He looked at the network diagram on the side of the screen, seven hundred thousand connections drawn as a dense blur because there was no way to show each one. Most of them had spent the last ten minutes switched off while the network still recognized everything he drew.

He thought about his own notebook. The parts he reread, the parts he never opened again. He thought about the way he sometimes forgot words but not the shape of what the words meant. He thought about the sixty thousand handwritten threes this network had learned from and how something in that learning had decided, without being told, that most of what it built wasn't strictly necessary.

He had no idea what to do with that.

He drew a nine, quickly, not thinking, and the bar said 97.6% for NINE and he found himself looking at the empty canvas for a long time after the confidence bars had settled, his pencil not moving, not touching anything.