The Machine That Couldn't Say Why

The screening van had parked outside the clinic since morning, and by afternoon the waiting room had emptied out. Maya and Soren were supposed to be helping their aunt, who ran the front desk, but there were no more patients, so they had drifted into the back office where a technician named Devon was clicking through photographs on a laptop.

"Are those eyes?" Maya asked.

"Backs of eyes," Devon said, not looking up. "Retinas. The camera shoots through the pupil. Then this program looks at the picture and flags anything that needs a doctor."

Soren leaned in. On the screen was an orange circle threaded with red rivers, a bright spot off to one side. Below it, the program had written: signs consistent with diabetic retinopathy. Refer.

"How does it know?" Soren asked.

Devon shrugged. "It's usually right. More right than I am, honestly."

"But how does it know," Soren said again. "Which part of the picture told it?"

"That," said Devon, "is the part nobody at the training even answered. It just does."

Maya pulled a chair over. "Click the next one."

Another orange circle. This one looked, to her, exactly like the first. Same red rivers. Same faint spots. The program wrote: no referral needed.

"Wait," Maya said. "Go back. Now forward. Those two are the same."

"They're not," Devon said. "One's sick, one's fine. But I couldn't tell you the difference either. I've looked at maybe two hundred of these in my life. The program looked at a hundred and thirty thousand before it ever saw a real patient."

Soren had gotten very interested. "So nobody told it the rules. Like, nobody said, a spot this big means sick."

"Nope. They just showed it a hundred thousand pictures with the answers attached. Sick, fine, sick, sick, fine. It figured out the difference on its own."

"That's cheating," Maya said, delighted. "That's completely cheating."

"How is that cheating?" Soren asked.

"Because that's how I learned to spot my brother lying," she said. "Nobody gave me a rule. I just saw him do it a thousand times. Now I know before I know why I know."

Devon laughed. "That's actually not a bad description."

Soren was pulling his notebook out of his back pocket. He drew two circles, side by side, and stared at the screen, and drew a small mark inside one of them. "So if you asked it," he said slowly, "which pixel made you say sick, it couldn't tell you."

"It couldn't tell you," Devon agreed. "There's no sentence in there. There's just, like, layers of numbers that got nudged a tiny bit every time it got one wrong, over and over, a hundred thousand times, until it stopped getting them wrong."

Maya went quiet. Then she said, "So it's not looking at the spot."

"What do you mean," said Soren.

"We're looking at the spot because the spot is the part we can point to. But it's not us. It didn't get taught the spot matters. Maybe it's looking at something we don't even have a word for."

Devon stopped clicking.

"There was a thing," he said carefully, "at the training. They said one of these programs learned to guess whether a person was a man or a woman just from the retina photo. Really accurately. And no eye doctor in the world can do that. Nobody knows what it's seeing. There's no rule for it. It's just there in the picture, and we've been looking straight at it for a hundred years and never noticed."

The room got quiet in a new way.

"Say that again," Maya said.

"There's information in the eye," Devon said. "Real information. Sitting right there in a photo we've all been taking forever. And we walked past it because we didn't know to look. The machine only knew because we didn't tell it what to ignore."

Soren put his pen down. "That's the whole thing," he said. "That's why it beats you. Not because it's smarter. Because you already decided what matters before you look. You look for the spot. It doesn't know there's supposed to be a spot."

"I look for what I was trained to look for," Devon said. "Yeah."

Maya was staring at the two orange circles like she could force the difference to surface. "So there are patterns," she said. "In everything. In eyes, in skin, in the lung scans, whatever. Real patterns that mean something. And they're just sitting there. And we can't see them because we already have a list of what's important."

"Some of them," Devon said. "Nobody knows how many."

Soren said, very quietly, "How many things are we looking straight at."

Nobody answered that.

Maya reached over and clicked the two photos herself, back, forward, back, forward. Sick. Fine. Sick. Fine. The rivers ran the same way in both. The spots sat in nearly the same places. Somewhere in there, in a place she had no name for and no eye for, one of these eyes was carrying a message its own owner couldn't read.

"It's like the picture is talking," she said. "And we've been in the room the whole time not hearing it."

"And the machine can hear it but can't tell us what it said," Soren added.

"So somebody has to figure out what it's hearing," Maya said. "Somebody has to sit here and go through them and go, okay, it says sick, why, what's different, until they find the thing nobody has a word for yet."

"People are trying," Devon said. "It's really hard."

"Good," Maya said, which made Devon laugh again, though she hadn't meant it as a joke.

Soren looked back at his notebook, at the two circles he'd drawn, one with a mark and one without, both of them wrong, because he'd guessed the way a person guesses and the difference wasn't a mark at all. He turned to a clean page.

A patient came in then, an old man with a cane, and Devon straightened up and swung the camera arm around and said, "Chin here, sir, look at the green dot." There was a soft flash. The old man's eye bloomed onto the screen, orange and red and threaded, holding whatever it was holding.

Maya leaned toward the glowing photograph until her nose was almost touching it, reading an eye the way you'd read a page in a language you had just found out existed.