The Edges of Everything

The model was supposed to find cracks in bridges.

Soren had spent three Saturdays in a row feeding it photographs. Close-ups of concrete, rebar shadows, hairline fractures running through overpasses. He had two hundred and twelve images, which Dr. Kapur said was not enough, which Soren already knew, which was the whole problem.

The workshop gave them access to a university GPU cluster and a set of pre-trained neural networks. Most of the other kids were building fun things. Lila had a model that could tell the difference between fourteen types of candy bar. Marcus was training one to identify dog breeds from their silhouettes. Soren was trying to find structural damage in aging infrastructure, which was, he admitted to his notebook, maybe too much.

His model was terrible. It guessed randomly. Fifty-three percent accuracy on a two-class problem, cracks versus no-cracks, which was barely better than flipping a coin.

Dr. Kapur walked the rows of the lab on Saturday mornings, coffee in one hand, reading glasses pushed up on her forehead where they did nothing. She paused behind Soren's chair, glanced at his screen, and said, "Still coin-flipping?"

"I need more data."

"You do," she agreed. "You could also try not starting from scratch."

She moved on to help Marcus, whose dog silhouettes were getting confused by cats.

Soren wrote in his notebook: not starting from scratch. He underlined it twice, which meant he didn't understand it yet.

The pre-trained networks they'd been given were big ones. ImageNet models, trained on millions of ordinary photographs. Cars, teacups, fire hydrants, golden retrievers, spatulas. Nothing about bridges. Nothing about cracks. Soren had looked through the list of the thousand categories the biggest model could recognize, and the closest thing to structural damage was "chain-link fence."

He almost closed the list. Then he stopped.

He opened the visualization tool Dr. Kapur had shown them the first week, the one that let you see what individual layers of a network had learned. He loaded the big ImageNet model and fed it one of his bridge photos.

The first layer was simple. It had learned to detect edges. Not edges of anything in particular. Just edges. Bright against dark. The shift where one thing stopped and another thing started.

Soren leaned closer.

The second layer combined those edges into textures. Repeating patterns. Bumps. Grooves. The third layer built shapes from the textures. Curves, corners, grids.

None of these layers knew what a bridge was. None of them knew what a crack was. But when Soren looked at the activation maps, the bright spots showing where the network was paying attention, the early layers were already lighting up along the fracture lines in his concrete photographs.

The network had learned edges from pictures of dogs and coffee mugs and running shoes. And edges were edges. A crack in a bridge was an edge too.

He sat very still for a moment.

Then he stripped off the top layers of the ImageNet model, the ones that knew about golden retrievers and spatulas and chain-link fences. He kept the bottom layers, the ones that only knew about edges, textures, and shapes. He attached new top layers, blank ones, and started training only those new layers on his two hundred and twelve bridge photographs.

It took eleven minutes.

Eighty-nine percent accuracy.

Soren pushed back from the desk and stared. He pulled his notebook toward him but didn't write anything. He just stared at the number.

Eighty-nine percent. From the same data that had given him fifty-three.

The knowledge was already in there. Not knowledge about bridges. Not knowledge about cracks. Knowledge about what an edge looks like. What a texture feels like. Where one thing ends and another begins. The network had learned that from fourteen million photographs of ordinary life, and it turned out that knowing where a dog's ear ends and the sky begins was the same skill as knowing where solid concrete ends and a fracture starts.

He ran it again. Eighty-seven percent. Again. Ninety-one. The numbers were real.

Lila looked over from her candy bar project. "Did yours just work?"

"It knew about edges," Soren said, which he realized explained nothing.

He went back to the visualization tool. He pulled up a medical imaging dataset from the workshop's example files, chest X-rays labeled with and without nodules. He had no reason to do this. He was supposed to be finding bridge cracks. But the question was pulling at him now.

He took the same ImageNet base, the same layers trained on dogs and teacups and fire hydrants, and attached new top layers for the X-ray task. He had less than a hundred X-ray images.

Seventy-eight percent accuracy. On lungs. From a model that had never seen lungs. From a model that had learned its foundations by looking at pictures of everyday objects.

Soren opened the activation maps for the X-ray model's early layers. The same edge detectors were firing. The boundary between a rib and the tissue behind it lit up the same way the boundary between a dog's nose and its fur had. The same way a crack in concrete had.

Dr. Kapur was back, standing behind his chair again. She looked at his screen for a long time.

"You went off-task," she said.

"The features transfer," Soren said. "The low-level ones. They're the same no matter what you're looking at."

"Yes," she said. And then, because she was Dr. Kapur and could never stop at one thing: "Why do you think that is?"

She walked away before he could answer, which was fine, because he didn't have one. He had something better. He had the question.

Why would a universe full of completely different things, dogs and lungs and bridges and candy bars, share the same edges? Why would the boundary between any two things look, at the lowest level, like the boundary between any two other things?

He thought about all the things he'd ever noticed by paying attention to what was wrong. The crack in the sidewalk on Elm Street. The way his grandmother's breathing sounded different last winter. The wobble in the second step of the porch. He'd always thought those were separate skills. Noticing cracks. Noticing sounds. Noticing wobbles.

Maybe they weren't.

Maybe noticing was one thing, all the way down.

He picked up his pencil, held it over the notebook, and then set it down again. He turned back to the screen and loaded a dataset he hadn't tried yet, satellite photos of coastlines, and began building new layers on top of the old edges.