“Minds, Brains, and Programs”

John Searle, Behavioral and Brain Sciences, vol. 3, 1980

Manipulating symbols correctly is not enough to understand what they mean: syntax is not semantics.

What the text says

Searle imagines himself locked in a room with boxes of Chinese characters and a rule book, written in English, that tells him which symbols to send back in response to those slipped under the door. With a good rule book, his answers would be indistinguishable from a native speaker’s. Yet he does not understand a word of Chinese.

The conclusion: running a program, however perfect, is not enough to produce understanding. The paper also publishes replies from many critics. The most famous, the “systems reply”, holds that while the man does not understand, the room as a whole perhaps does.

Why read it now

Large language models look like a Chinese room on a planetary scale. Is Searle’s argument confirmed, or refuted, by machines that answer so well? The debate is livelier than ever, and it forces us to say precisely what we mean by understanding.

What remains open

If a system answers everything as if it understood, what exactly is missing for it to understand?