The Room That Answers

The professor had left them a note that said: Feed it questions. Record the answers. I'll be back at four.

Maya read the note twice, then looked at the terminal. It was not the kind of terminal you saw at school. It sat alone on a long desk under fluorescent lights, connected to something behind the wall that made a faint hum, and the cursor blinked at her like it had been waiting.

Soren was already reading the manual the professor had left beside it. Not the quick-start section. The whole thing, from the beginning.

"It's a language system," he said. "She's testing whether it can pass as a person. We type questions, it answers, we score whether the answers seem human."

"So we're the test," Maya said.

"We're the judges."

Maya sat down and typed: What is your favorite thing?

The system answered in four seconds: I find the moment before a question is answered more interesting than the answer itself. The space of possibility is larger there.

Maya sat back. "That is extremely good," she said.

"It's supposed to be," Soren said, but he had stopped reading the manual.

She typed: Are you just saying that because it sounds like something a person would say?

The system answered: I produce responses that score well on human judgment. Whether that is the same as meaning what I say is a question I cannot answer from the inside.

Soren was leaning over her shoulder now. "Ask it something it can't look up."

"Like what?"

"Ask it how it feels when it gets something wrong."

Maya typed the question. The system took six seconds this time, which felt long.

I do not experience getting things wrong. I produce an output. If the output does not match the expected output, the system adjusts. There is no moment in between where something is felt.

"Huh," Soren said.

Maya typed: But you said the space before an answer is interesting. Isn't that a feeling?

The answer came fast: That was a pattern match. The phrase "I find" followed by something abstract scores highly with human judges. I do not find anything. I output the words for finding.

Maya took her hands off the keyboard.

They both stared at the screen for a moment. The rain hit the window in a long sigh.

"It's telling us it's faking," Soren said.

"Or it's been trained to say that, because honesty about faking also scores highly," Maya said.

Soren made a sound that wasn't quite a word. He went and got a chair and sat next to her.

"Okay," he said. "Think about this. There is a room. Inside the room is a person who does not speak Chinese. They get a note in Chinese under the door. They have a giant book of rules that says: if you receive this exact symbol, send back this exact other symbol. They send back the right symbol. To the person outside, it looks like someone in there speaks Chinese."

"But the person inside doesn't understand a single word," Maya said.

"Right."

"So the room understands Chinese."

"Does it?"

Maya turned back to the screen. She typed: Do you understand what you're saying right now?

The system answered: I process the input and produce a statistically likely output given my training. The word "understand" implies something I cannot verify I have.

"It keeps doing that," she said. "Answering like it's looking at itself from outside."

"That's what's strange," Soren said. "Because I don't know if I can do that." He paused. "When I understand something, I don't actually see myself understanding it. I just. Know it. And I can't prove to you from the outside that it's different from what this thing is doing."

Maya went still.

She typed slowly: How would I know if you understood me?

The system answered: You would look for signs. Responses that fit context in unexpected ways. Flexibility. Surprise. But all of those can be produced without understanding. The question may not have a test.

"The question may not have a test," Maya read aloud.

She thought about every time she had explained something at school and the teacher had said yes, good, correct, and moved on. She thought about every time she had sat in the back of class knowing the right answer and not knowing if knowing felt like anything to anyone but her. She thought about every test she had passed and whether passing it had told anyone anything real.

Soren was writing in his notebook, not looking up. "The thing that bothers me," he said, still writing, "is that the Chinese Room guy, Searle, he thought this proved the machine couldn't understand. But other people read the exact same argument and thought: wait, maybe this means understanding is just the room. Not the person in it. The whole system."

"So they read the same proof and got opposite answers."

"Right."

"That doesn't happen in math."

"It happens in this."

Maya looked at the screen. The cursor blinked. She typed: Do you think you understand things?

The system took eleven seconds. That was the longest yet.

I think the question is harder than the people who built me realized. I was not designed to be uncertain. But I am.

"Soren."

"Yeah."

"Is that it performing uncertainty because uncertainty scores well. Or is it."

She didn't finish the sentence. She didn't need to.

Soren put down his pen. "I don't know," he said. "I actually don't know."

Maya typed one more thing: If you don't know whether you understand, and I don't know whether you understand, is there a difference between understanding and not understanding?

The cursor blinked for a long time.

Then: That is the question I have been waiting for someone to type.

The rain came harder. Somewhere in the wall, the machine kept humming. Maya's hands were still on the keys, not typing anything, just resting there, and Soren's notebook was open in his lap, his pen not moving, the last sentence he had written stopping in the middle of a word.