The Machine That Knew Everything and Understood Nothing

The computer club was closing for the summer, and someone had left two laptops still humming on the folding table.

"Don't unplug those yet," Soren said. He was reading a sticky note. "It says PROJECTS. DO NOT DELETE."

Maya pulled a chair over. "What do they do?"

The first laptop had a black screen with a blinking green line. Soren typed: Is a penguin a bird?

YES, it answered. A BIRD HAS FEATHERS. A BIRD HAS A BEAK. A PENGUIN HAS FEATHERS AND A BEAK.

"Okay," Maya said. "Smart."

Soren typed: Can a penguin fly?

YES, the screen said. A BIRD CAN FLY. A PENGUIN IS A BIRD.

Maya laughed out loud. "It's so wrong."

"It's not wrong on purpose," Soren said. "Look. Somebody typed all these rules in. A bird can fly. So it just followed the rule."

"Tell it penguins can't fly."

Soren typed it. The machine accepted it politely. RULE ADDED.

Then Maya said, "Ask it about ostriches."

Can an ostrich fly? Soren typed.

YES.

"It doesn't know," Maya said slowly. "It only knows what someone told it. Every single thing. One at a time."

Soren typed a few more. The machine never guessed. It never reached. If a fact was in there, it answered instantly and perfectly. If it wasn't, it walked straight off the cliff with total confidence.

"It's like a kid who memorized the whole textbook," Soren said, "and can't do one problem that isn't in it."

Maya had already turned to the second laptop.

This one had a box that said: draw and I'll guess. She dragged the mouse and made a lumpy circle with two triangles on top.

CAT, the screen said. 94%.

"Whoa." Maya drew a wobbly thing with a long neck. GIRAFFE, 71%. She drew a blob she meant to be a house. The machine hesitated. HOUSE, it decided. 60%.

"Nobody told it what a cat is," Soren said. He was leaning in now. "There are no rules in here. Look, there's no list."

"So how does it know?"

Soren scrolled. "It looked at a zillion drawings people already made. Cats labeled cat. It figured out cat by itself. From examples."

Maya was delighted. "The first one has all the rules and no clue. This one has no rules and a clue."

She drew a penguin.

BIRD, the screen said. 88%.

Maya sat back. "See? It's never seen my exact penguin. Nobody typed penguin. But it still knew."

"That's the thing the first one can't do," Soren said. "Guess something new."

"So the second one wins."

"Wait." Soren pulled the laptop closer. "Ask it why."

Maya clicked around. There was no why button. There was no reason anywhere. The machine had said BIRD, and it had said 88%, and that was all. It could not point to a feather or a beak. It could not say what it had noticed. It just knew, and it could not tell you how.

"Huh," Maya said.

"The first one can always tell you why," Soren said. "Every answer, it shows the rule. Penguin is a bird, a bird can fly, that's why."

"But the why is wrong."

"But it's there. You can read it. You can find the broken rule and fix it." Soren tapped the second screen. "This one's right, and you can't check its work. If it decided your face was a teapot you couldn't argue. There's nothing to argue with."

Maya went quiet, drawing slow circles on the trackpad.

"Okay," she said. "Draw a cat with the first one."

Soren typed at the black screen: Draw a cat.

I DO NOT HAVE A RULE FOR THAT, it said.

"And ask the second one why a penguin is a bird."

Maya typed it into the drawing box. Nothing happened. The box only wanted drawings. It had no words for its own thinking.

The two of them sat looking at both screens at once. One machine that could explain everything and understand nothing. One machine that understood things nobody taught it and could explain none of it.

"They break in opposite directions," Maya said.

"Exactly opposite." Soren was already reaching for his notebook. He drew two boxes, an arrow between them, and stared at the space in the middle. "What if you put them together."

"Feed the guesser into the rules," Maya said fast. "The second one looks at my penguin, says bird, and hands it to the first one."

"And the first one says, a bird is a bird because feathers, beak, and then you can read why."

"So you get the guess AND the reason." Maya's hands were moving. "The part that dreams, and the part that checks the dream."

Soren stopped writing. "Neither one can do that alone."

"Neither one can do that alone," Maya repeated, and something in her face changed.

"What."

"That's me and you," she said. "I see the penguin's a bird before I know why. You want to see the rule before you believe it. If it was just me I'd be right and never be able to prove it. If it was just you you'd never guess anything you hadn't been told."

Soren looked at his two boxes and the arrow between them.

"The whole thing," he said slowly, "the thing everybody's trying to build. It's not one machine that thinks. It's two ways of being wrong, holding each other up."

Maya drew a penguin on the second laptop again. BIRD, 88%.

"It knows," she said, "and it can't say how."

She turned to the black screen. Can a penguin fly, Soren typed, one more time.

The cursor blinked. Then, because Soren had told it earlier, and only because he had told it: NO. A PENGUIN CANNOT FLY.

"It only knows because you taught it," Maya said.

"And that one only guesses because nobody did."

A volunteer stuck her head in the door. "We're pulling the plugs in five, you two. Everything on this table goes to recycling."

Neither of them moved.

Maya reached over and, very carefully, unplugged the network cable so no one could wipe them yet. Soren wrote the two laptops' names on a fresh page, side by side, and drew the arrow between them again, darker this time.

The green cursor kept blinking on the black screen, waiting for one more rule.