7 min read
M06

Machines · No. 21

The Wrong Question

Turing called the question of whether machines can think too meaningless to discuss. We have argued about it ever since.

· · 7 min · 1,606 words

In short

  • Turing himself thought the question of whether machines can think was too vague to answer, so he replaced it with a test of imitation.
  • The imitation game rewards deception, and programs from ELIZA to Eugene Goostman showed that lowering expectations is the easiest way to pass it.
  • Asking whether a machine really thinks is mostly a question about the word think, like asking whether a submarine really swims.
  • Better questions are what a system can reliably do, for whom, and who is accountable when it fails.
  • In 2024 a Canadian tribunal held Air Canada responsible for what its chatbot told a customer, a reminder that accountability does not transfer to software.

Picture a party game. There are two rooms. In one sit a man and a woman. In the other sits an interrogator, who can only pass typewritten notes back and forth and has to work out which hidden player is the woman. The woman tries to help. The man tries to lie well enough to be picked instead.

In 1950, Alan Turing opened a paper in the philosophy journal Mind with this game. The paper was called "Computing Machinery and Intelligence," and its first sentence is one of the most famous in the history of computing: "I propose to consider the question, 'Can machines think?'"

Then, almost at once, he declined to answer it.

Words like machine and think were too slippery, he argued. If you tried to settle the question by looking at how people normally use those words, you would end up deciding it by something like an opinion poll, which he called absurd. So he swapped the question for the game. Take the man out of the room and put a machine in his place. If the interrogator cannot reliably tell the machine from the person, what more could you reasonably ask for?

It was a brilliant move. It also set the terms for seventy-five years of arguing about the wrong thing.

A test you pass by pretending

Look closely at what the imitation game rewards. Not truth. Not usefulness. Convincing performance. In the original party version, the winning strategy for the man is to be a good fake, and Turing kept that structure when the machine walked in.

He seems to have understood exactly what that implied. In the sample conversation he wrote for the paper, the interrogator asks the machine to add 34957 and 70764. The machine pauses for about thirty seconds and answers 105621. The correct sum is 105721.

Readers have argued for decades about whether Turing slipped or planted the mistake on purpose. Either way, it captures the game perfectly. To pass as human, a machine might have to be slower and worse than it really is.

The people who built chatbots learned this lesson fast.

In the mid-1960s, Joseph Weizenbaum at MIT wrote a program called ELIZA. Its most famous script played a certain kind of therapist, the kind who mostly turns your statements back into questions. You type that you are unhappy; it asks why you are unhappy. There was no understanding underneath, only pattern matching and some clever rephrasing.

People loved it anyway. Weizenbaum later wrote that his own secretary, who had watched him build the program and knew exactly what it was, asked him to leave the room so she could talk to it in private. The experience disturbed him so much that he spent much of the rest of his career warning about how readily people hand their trust to machines.

In 2014, a chatbot called Eugene Goostman took part in a Turing test event at the Royal Society in London and convinced about a third of the judges that it was human. Headlines announced that the test had finally been passed. The detail that mattered was the persona. Eugene was presented as a thirteen-year-old boy from Ukraine for whom English was a second language. Every odd answer, every dodge, every gap in knowledge had a ready excuse.

That is the quiet flaw in a test of imitation. The cheapest way to pass it is to lower what the judge expects, and getting smarter is optional.

The cheapest way to pass a test of imitation is to lower what the judge expects.

By the mid-2020s the game had stopped being an interesting bar at all. Large language models could hold long, fluent conversations on almost any subject, and researchers running careful versions of the test found that many people could no longer reliably tell. And yet the argument about whether these systems really think did not move an inch. If anything, it got louder.

That should tell you something about the question.

Submarines do not swim

The computer scientist Edsger Dijkstra once compared the question of whether machines can think to the question of whether submarines can swim.

It is a joke, but a precise one. A submarine crosses oceans. Whether that counts as swimming depends entirely on what you decide the word swim means. Nothing you learn about the submarine will settle it, because the disagreement was never about the submarine.

Planes do not flap their wings, and nobody refuses to fly because of it. A calculator does not "really" know arithmetic, and nobody checks its sums by hand to protect the dignity of the word know.

Turing saw this coming. He predicted that by the end of the twentieth century, the meaning of words and educated opinion would have shifted enough that people could speak of machines thinking without expecting to be contradicted. Look at how casually people now say a model "thinks for a while" before it answers, and you can see he was not far off.

The deep question about consciousness is real, and serious philosophers work on it. But for anyone building, buying or living with these systems, it is the wrong question to start with. It is unanswerable on any timescale that helps you, and it pulls attention away from three questions that are answerable, and urgent.

Better question one: what can it actually do?

Turing's real gift was not the game. It was the instinct behind it: replace a vague question with an observable one. Keep that instinct, drop the deception, and you get a much more useful habit.

Ask about tasks. Can it read this contract and find the clause that matters? Can it write code that passes these tests? Can it summarise this medical record without inventing a symptom? How often does it get it right, and when it fails, how does it fail?

That last part matters more than people expect. These systems are uneven in strange ways. A model can write a decent essay on Ottoman trade routes and then fumble a simple question about the order of letters in a word. Human intuition about difficulty does not transfer, so your sense of "if it can do the hard thing, surely it can do the easy one" will mislead you.

Testing is boring. You build a set of real examples, you run the system on them, you check the answers by hand. It has none of the drama of asking whether a machine has a mind. It is also the only thing that tells you whether to trust it with your work on Monday.

Better question two: for whom?

The same capability means different things to different people.

A translation tool is a convenience for a tourist ordering lunch. For someone trying to explain symptoms to a doctor in a language they do not speak, it is something much heavier, and an error is not a funny menu item. A voice that sounds perfectly human is a delight in an audiobook and a weapon in a phone scam aimed at someone's grandmother.

So when you hear that a system "can" do something, the follow-up is always: for whom, and at whose cost? Who gets the benefit, who absorbs the mistakes, and are they the same people? Most of the real ethical weight in AI sits here, in ordinary product decisions, not in debates about machine souls.

Better question three: who answers for it?

In 2022, a passenger in Canada asked Air Canada's website chatbot about bereavement fares after a death in his family. The chatbot told him he could book now and apply for the discount afterwards. That was not the airline's actual policy, and when he applied, the airline said no.

The case went to a civil resolution tribunal in British Columbia, and in 2024 the airline lost. Along the way, Air Canada had argued, in effect, that the chatbot was responsible for its own actions. The tribunal was not persuaded. The airline was responsible for all the information on its website, it said, whether it came from a static page or a chatbot.

Read that argument again, because you will hear versions of it for the rest of your life. The system said it, not us. It is the most tempting sentence in the age of AI, and it is the one that must never work.

A machine can produce an answer. It cannot be fined, fired, sued or ashamed. Somebody chose to deploy it, somebody decided what it was allowed to do, and somebody took the money it earned. That person, or that company, answers for it. If you build with these tools, decide before launch who that is. If the honest answer is "nobody," you are not ready to launch.

The question behind the question

It is worth remembering what kind of paper "Computing Machinery and Intelligence" really was. The game takes up only a small part of it. Most of the paper is Turing patiently working through objections, and then, near the end, sketching how you might build a "child machine" and teach it, instead of trying to program an adult mind directly. He wanted people to stop debating and start building and measuring.

He closed the paper with a line that has aged better than anything else in it: "We can only see a short distance ahead, but we can see plenty there that needs to be done."

That is still true, and the work in front of us is clearer than it was for him. Measure what these systems do. Ask who they serve. Put a human name next to every decision they make.

Whether the machine is thinking can wait. Whether we are is the question that cannot.

Further reading

  • Computing Machinery and Intelligence — Alan Turing
  • Computer Power and Human Reason — Joseph Weizenbaum
  • Alan Turing: The Enigma — Andrew Hodges

More from the shelf

M07 · Machines · 7 minThe Bitter LessonSeventy years of AI research keep teaching the same humbling thing, and builders keep having to learn it again. M01 · Machines · 7 minIn Defense of FrictionWe spent decades removing effort from everything, and accidentally removed some of the meaning with it. B06 · Builders · 6 minThe Ten-Person CompanyInstagram sold for a billion dollars with 13 people. AI is about to make that look crowded.

See the whole library →