The Reason After the Reason

The chess bot was wrong again.

"It says take the knight," Maya said. She tilted the tablet so Soren could see. "Taking the knight loses the rook. Even I can see that."

Soren wiped solder smell off his hands onto his jeans. The bot was an old thing, a speaker shaped like an owl that his uncle had given him, and they had spent the morning teaching it to read board positions out loud. It got most of them right. This one it kept getting wrong, the same way, every time.

"Ask it why," he said.

Maya pressed the button. "Why do you want to take the knight?"

The owl thought, which meant a little light spun for a second. "Capturing the knight removes your opponent's most active piece and opens the center for your bishop," it said pleasantly.

"That's a good reason," Soren said.

"It's a good reason for a move that loses," Maya said. "Watch." She tapped through the line. Knight gone, bishop free, and then the opponent's queen slid down and took the rook for nothing. "See? The reason sounds right. The move is wrong."

Soren leaned back against the workbench. "Okay. So the reason it gave isn't the reason it actually picked the move."

"How do you know?"

"Because if that were really why it picked it, it would have seen the queen too. The reason it said out loud is short. The thing that actually happened inside it is way bigger than that."

Maya looked at the owl. The light wasn't spinning now. It just sat there, plastic and patient.

"Ask it," she said slowly, "if it knows why it got the move wrong."

Soren took the tablet. "Do you know why you recommended a losing move?"

The light spun. "I may have weighted material gain too heavily and failed to calculate my opponent's reply. I apologize for the error."

"That's another good reason," Maya said. "For being wrong this time."

"It is," Soren said. He frowned. "But it didn't go back and look at what it did. It can't. It just made up a sensible story about why a thing like that might happen."

Maya sat down on an upturned bucket. "Made up."

"Not lying. Just." Soren searched for it. "When you do long division and get it wrong, you can look back at your paper and find the exact place you carried the wrong number. The mistake is sitting right there in your handwriting."

"Right."

"The owl doesn't have handwriting. The math that actually picked the knight isn't written down anywhere it can read. It happened in, like, millions of little numbers all pushing on each other at once, and then a move came out the other end. When we ask why, it doesn't go open up that math and check. It can't see it."

Maya was very still on the bucket. Then she said, "So what is it doing when it answers?"

"Making a new thing. A reason-shaped thing. After."

"After the move already happened."

"After," Soren said. "The answer comes first. The reason comes second, and the reason isn't a memory of the first thing. It's a fresh guess about what a good reason would have been."

They both looked at the owl.

"Ask it that," Maya said. "Ask it if it can actually see the thing it did, or if it's guessing."

Soren pressed the button. "When you explain your reasoning, are you remembering what you actually computed, or are you making a good guess about it afterward?"

The light spun longer this time. "I generate an explanation that is consistent with my output. I do not have direct access to the internal computations that produced it."

The rain went on against the garage window.

"It just told us," Maya whispered. "It just told us it can't see inside itself."

"It told us a reason-shaped thing about not being able to see inside itself," Soren said. "Which is the same problem. Even that answer is an after-the-fact one."

Maya laughed, a startled little sound. "That's a trap. There's no bottom to it."

"There isn't."

She stood up off the bucket. She was looking at the owl, but past it too. "Soren. People do this."

"Do what."

"This. The reason-after-the-reason thing." She pointed the tablet at him like evidence. "When my mom asks why I snapped at my brother, I tell her a reason. A good one. I really believe it. But I didn't, like, watch the actual thing happen in my head that made me snap. The mad part came first. The reason I tell her, I make that up after. A reason-shaped thing."

Soren stopped wiping his hands.

"That's not lying either," Maya said. "It's the same as the owl. The answer comes first. Then I build the reason and I can't see the machine that actually did it. Nobody can. I can't look at my own handwriting."

"You think your reasons are guesses too," Soren said.

"I think I never checked," Maya said. "I think nobody ever checks. I think we all just trust the reason-shaped thing that comes out and we call it the truth about ourselves."

The garage felt suddenly very full, the two of them and the owl and every person who had ever explained why they did something.

Soren picked up his notebook from the bench. He turned to a clean page. He wrote: the explanation is not the thing it explains. He underlined it once, looked at it, and didn't underline it again.

"This is a real one," he said. "Isn't it. Not a homework one. People who study minds, actual scientists, they don't know how to read the inside of the owl either. Or the inside of us."

"They don't," Maya said. "It's open. It's still open." She said it like a door that wouldn't shut, and she liked that it wouldn't.

She pressed the owl's button one more time.

"Last question," she said. "When you told us you can't see inside yourself. How do you know that's true?"

The light spun, and spun, and the little owl said, "That is a reasonable assumption based on how I work."

Maya set the tablet down very gently, the way you set down something that is still warm, and the light kept spinning after she let go.