What the Camera Thinks It Sees

The sign above the booth said: CAN YOU FOOL OUR AI? and the answer, according to everyone who had tried, was no.

Maya had been watching for twenty minutes. Not playing. Watching.

The demo worked like this: you held up an object or a printed photo in front of a camera, and the screen showed what the AI thought it was. A banana: banana, ninety-eight percent. A sneaker: sneaker, ninety-seven percent. A kid held up a drawing of a cat and the system said cat, eighty-one percent, and everyone laughed like it was a trick the AI had done on purpose.

The researcher running the booth was named Dr. Osei. He had a lanyard full of badge-clips and the particular exhaustion of someone who had answered the same questions forty times before noon. When he explained how the network worked, he used the word powerful six times in four sentences.

Maya had a printed photo in her pocket. She had printed it herself, at the library, that morning.

The photo was a school bus.

She already knew what the AI would say about the school bus. She had looked this up. But knowing a thing and seeing it were not the same, and Maya did not like operating on faith.

She stepped up and held the photo in front of the camera.

School bus, said the screen. Ninety-four percent.

She nodded, put it in her left hand, and pulled out the second photo from her right pocket.

The second photo looked exactly like the first one. Same yellow. Same windows. Same angle. She had printed both of them at the same library printer, from the same computer, in the same ten-minute window that morning.

She held it up.

Ostrich, said the screen. Ninety-nine percent.

Dr. Osei looked up from his phone.

A few kids nearby made sounds. One of them said she had used a different photo. Maya turned both photos outward so the small crowd could see them side by side. Same bus. Same windows. Same yellow.

Same, said Maya.

Dr. Osei took both photos. He held them at different angles under the gymnasium lights. He brought them within two inches of his face.

They look the same, he said. But they're not.

I know, said Maya.

He looked at her. His exhaustion had changed into something else, the way weather changes, not slowly but all at once.

The second image had been modified, Maya explained. Not in any way a human could see. Each pixel had been shifted in brightness by at most one step, one single unit on a scale of two hundred fifty-five. No human eye could detect a single brightness step. But those tiny shifts, placed across the image in a very specific pattern, had completely redirected the network's attention. Not to the windows. Not to the yellow rectangles. To something the network associated, through thousands of invisible layers of learned weights, with a long-necked bird in the savanna.

Nobody had cheated, she said. The photo had not changed in any way she could see, or that Dr. Osei could see, or that any person in this gym could see. But the AI had not seen what they saw. The AI had seen an ostrich, with more certainty than it had ever seen the bus.

Dr. Osei sat down on the edge of the table. The lanyard swung against his chest.

Where did you get the modified image, he asked.

I made it, said Maya.

He looked at her badge. It said sixth grade, regional fair, biology.

How, he said.

There are papers, she said. Published ones. The math is hard but the result is not secret. It has a name. Adversarial examples. People have been writing about it for years.

I know what adversarial examples are, said Dr. Osei. I study this. I'm asking why an eleven-year-old was up at whatever hour making them.

Maya thought about this. She had been up at six. Her parents had not known.

Because the sign said can you fool our AI, she said. And I wanted to know if it was actually possible or if it was just a thing people said.

The small crowd had mostly drifted away. One kid was still there, staring at the two photos as if he expected them to change.

Here is the part that sat wrong with Maya, and had been sitting wrong since she first read about this three weeks ago in a paper she had found by accident while looking up something else entirely.

The network was not malfunctioning. It was doing exactly what it had been trained to do. It was doing its job correctly. It had learned to see by finding patterns no human had told it to find, patterns compressed through millions of training images into something nobody could fully inspect or explain. And those patterns were real. They worked. The network recognized bananas and sneakers and drawings of cats with extraordinary accuracy.

But those same patterns, those same invisible learned preferences, could be played like an instrument. Someone who knew how could reach into the space between human vision and machine vision, the space where the two did not overlap, and play a note only the network could hear.

Maya did not know what was in that space. Nobody did, exactly. She had read four papers about it and the papers disagreed on what it meant.

Dr. Osei was writing something on the back of his conference badge, an email address.

You should send me your method, he said. I mean that. I want to see what you built.

He slid the badge across the table.

Maya picked it up. She looked at the two photos once more, side by side in her hands, one bus the world saw and one bus the network saw as a long-necked bird running across dry grass, both of them identical to every sensor in her body, the difference living somewhere she had no instrument to find.

She put both photos in her pocket and walked back out into the gymnasium.