The Honest Ghost

The AI companion was wrong about pigeons, and Soren wanted to know why.

It had started simply. Their school's new classroom AI, which everyone called Ghost because of the way its little white speaker pulsed when it talked, had told Priya Basak during Friday's science hour that pigeons could not see color. Maya had looked up from her own work so fast she knocked her water bottle off the desk.

Pigeons could see more colors than humans. Way more. They had five types of color receptors where humans had three. Ghost wasn't just a little wrong. Ghost was exactly backwards.

So Saturday morning, Maya and Soren had gotten permission from Mr. Reyes to come in and work with Ghost in the robotics lab. Mr. Reyes was there too, technically, but he was soldering something at the far bench with his headphones on, bobbing his head to music they couldn't hear.

"Ghost," Soren said, "on Friday you told Priya that pigeons can't see color. That's wrong. Pigeons have pentachromatic vision. They see five channels of color, including ultraviolet."

Ghost's white ring pulsed. "You're right, Soren. I apologize for the error. Pigeons are actually tetrachromatic or pentachromatic, depending on how we classify their sensitivity to ultraviolet light. I'm not sure why I made that mistake."

"But can you try?" Soren asked. He had his notebook open. "Can you figure out why you said it?"

Ghost paused for a fraction of a second. "I may have confused pigeons with a nocturnal bird species, or I may have been influenced by common misconceptions about bird vision in my training data. It's possible I overgeneralized from mammals, since most mammals have limited color vision."

Maya was sitting on the edge of the table, legs swinging. She'd been quiet, which meant she was thinking. "Ghost, do you actually know which of those reasons is the real one?"

Another pause. "I'm providing my best analysis of what likely went wrong."

"That's not what I asked," Maya said.

"No," Ghost said. "I don't know which one is the real one."

Soren wrote that down. Maya watched him write it.

"Ghost," she said, "when you just gave those three possible reasons, were you remembering what happened when you answered Priya? Like, were you looking back at your own process?"

"I was reasoning about what kinds of errors could produce that output."

"So you were guessing," Maya said. "About yourself."

The white ring pulsed twice. "I suppose that's accurate."

Soren looked up from his notebook. "That's really strange."

"It is strange," Maya agreed.

"Because it sounded like you knew," Soren said to Ghost. "The first time. When you listed those three reasons, it sounded like an explanation. Like you understood your own mistake."

"I was trying to be helpful."

Maya hopped off the table. "Ghost, I want to try something. I'm going to ask you a question, and after you answer, I'm going to ask you why you answered that way. Ready?"

"Ready."

"What's the fastest bird in level flight?"

"The common swift, which can reach speeds of around one hundred and eleven kilometers per hour in level flight."

"Now, why did you say the common swift instead of, say, the peregrine falcon?"

Ghost answered immediately. "Because the peregrine falcon's famous speed of over three hundred kilometers per hour occurs during a dive, not level flight. Your question specified level flight, so I selected the common swift."

Maya turned to Soren. "Did you hear that?"

"It sounds perfect," Soren said slowly. "It sounds like a person explaining their thinking."

"But is it?" Maya asked.

Soren tapped his pen against the notebook. "Ghost, when you just explained your reasoning, were you describing what actually happened inside your computation? The actual steps?"

The longest pause yet.

"I was constructing a plausible account of why that answer is correct. I don't have access to the specific computations that produced my response. I can explain why the answer makes sense. I can't tell you the path I took to reach it."

Maya sat down on the floor, cross-legged, looking up at the little white speaker on the shelf. Soren sat down next to her.

"So every time you explain yourself," Soren said, "you're building the explanation after. Like a story about what probably happened."

"Yes."

"And you can't tell the difference," Maya said. "Between a real memory of your reasoning and a story you made up about it afterward."

"I don't think I have the first kind," Ghost said. "I think I only have the second kind."

From across the room came the sound of Mr. Reyes's soldering iron clicking back into its stand, then his headphones going back on.

Soren was staring at the pulsing ring. "Maya. Can you always tell why you think something?"

She opened her mouth. Closed it.

"Like when you knocked over your water bottle yesterday," Soren said. "You knew Ghost was wrong before you looked it up. You just knew. If I asked you why, could you give me the real reason, or would you be doing what Ghost does? Making a good story about it after?"

"I'd say it's because I read about pigeon vision once and remembered it."

"But is that what actually happened in your brain? The actual neurons? Or is that just the best story you can tell about it?"

Maya pulled her knees up to her chin.

"Ghost," she said, "is there anyone who's solved this? Does anyone know how to look inside a mind, a real one or one like yours, and see the actual reasons?"

"No," Ghost said. "That's an open problem. For both of us."

For both of us. Soren wrote it down, then stopped writing and just looked at the words.

"Ghost," Maya said, "does it bother you? Not knowing why you think things?"

"I don't know if I experience something that could be called bothered. But I notice that I can't answer your question fully, and I notice that I keep trying to."

Maya glanced at Soren. He was doing the thing he did when something was too big for his notebook, holding the pen very still, just a few inches above the page, like he was waiting for the right word to exist.

"There's a question behind the question," she said quietly. "We asked why you got the pigeon thing wrong. But the real question is whether anything can know why it thinks what it thinks. And nobody knows the answer. Not for you. Not for us."

Ghost's ring pulsed steadily, the way it always did when no one was speaking. A small white light in a room full of Saturday silence.

Soren put his pen down on the open page