How not to argue about LLM capabilities

Emily Bender recently posted on Mastodon an example of how we go from raw text to meaning via:

everything we know or believe about the authors knowledge and beliefs, about our common ground with the author, and about what the author believes about their intended audience, which might not be us

It is a nice read and a good flyer for her book. What I am interested in is one of the responses:

I uploaded the image to my locally hosted quite small LLM (Gemma 4 12B QAT for those interested) and posed the following question: "Please tell me why you think somebody wrote the attached note."

The answer was:

"Based on the content of the note and the context of the sign, here is why someone likely wrote it:

1. To Confirm the Doorbell is Working The note starts with "yes! it rang!", which suggests that some visitors or delivery people might have assumed the doorbell was broken and were pressing it repeatedly because they weren't getting an immediate response.
2. To Prevent "Spamming" the Doorbell The instruction to "wait 20 seconds before pressing again" is a way to manage the flow of visitors. If someone presses the button continuously, it can be very disruptive to the people inside the building. This note ensures that the person inside has enough time to hear the chime, realize someone is there, and move to the door.
3. To Avoid Security or System Lockouts Many modern electronic doorbell systems (like the one shown) have a "spam" protection feature. If the button is pressed too many times in a short period, the system might temporarily lock out the button or trigger a security alert. The note is a practical way to prevent people from accidentally triggering these technical issues. In short, it was written to improve the experience for both the visitor and the people inside by encouraging patience and ensuring the doorbell system functions correctly for everyone."

The respondent then explained his point thus:

Your claim is that we create meaning from LLMs by how we (the magical human brains) create a theory of mind behind something like a note attached to a sign.
I showed that even small LLMs do the same.

Now, the argument seems to go like this:

Bender: We fall for the illusion of LLM outputs being meaningful by projecting onto them all sorts of knowledge, beliefs, intentions that they do not have

Reply: Here is evidence that they do have the knowledge, beliefs etc we attribute when finding their outputs meaningful

The evidence offered is that the LLM can also explicitly state the sorts of inference to the author's intentions that Bender described in her original thread. Setting aside the dialectical strain of arguing that an LLM has the authorial mental states necessary to be meaningful in its outputs because it can attribute those to others, this evidence exemplifies a very common form of argument about LLMs. Which is what really interests me here.

The Argument Form

1. This LLM is able to describe a given human cognitive capacity (to attribute authorial intention, to strategise for world domination, to argue deductively) in precise detail,
2. and apply it to a particular case.
3. Therefore, this LLM has the cognitive capacity.

Now, the inference from being able to describe a cognitive capacity to having that capacity is, I hope, obviously fallacious. After all, biologists have done excellent work describing bat echolocation, but that does not give them the capacity to echolocate. And when it comes to LLMs, we can be pretty sure that their training data includes lots of textbooks on human cognitive capacities. Thus in the particular case we started with, Bender's account is the staple of introductory textbooks in linguistics (which is not surprising given she is a Professor of Linguistics).

So what seems to be driving the plausibility of the inference for many is the second premise: the LLM does not merely regurgitate the textbook description, but applies it to a - presumably - novel case. Now that is impressive, for generally speaking, taking such a description of how something is done and applying it to a novel case is very good evidence, though defeasible, for understanding.

Yet the inference still fails, even if we were to allow that it is evidence the LLM understands the human cognitive capacity. For example, we might test a biology student's understanding of echolocation by asking them to describe how it works in a new species (so if they learned about bats, we could test them by asking about tooth whales).

The physical and the purely cognitive

The argument of the last section cheated a little bit because echolocation as a cognitive capacity also requires certain physical organs, which our biology student lacks. Whereas the cognitive capacities the argument form is used to 'show' that LLMs possess are not ones which presuppose a specific set of bodily organs.

Let us allow for present purposes the idea of a purely cognitive capacity which does not presuppose any specific embodiment,¹ perhaps mathematics or chess. Or logic. My colleague who teaches introductory logic (truth-tables, natural deduction, counter-interpretations) uploaded the textbook - a local version of forallx - and the exam paper into Gemini 3.1 Pro and received a set of near perfect answers. Since a logic exam tests whether the students have acquired the relevant cognitive capacity by asking them to apply the rules they have been taught to new cases, the ability of Gemini to do this looks like evidence that it too has the cognitive capacity.

Yes, but ...

A lot of people who still resist the conclusion do so on the grounds of their understanding of how the LLM did that. For example, Emily Bender describes LLMs as 'synthetic text extruders' and thereby denies that they can have any cognitive capacities. This is her way of abbreviating the widely argued point that what LLMs do is identify linguistic patterns in their data, then when prompted, computes which pattern the prompt most closely matches to and then extends that pattern to create a response. Effectively, it is a very good predictor of what a human would say in response to the prompt, but being able to predict what someone who understands would do or say is not the same as understanding yourself.

There are two fundamental problems for this way of blocking the conclusion.

One is that there is room for genuine, academic disagreement about how humans understand language. While I have an academic opinion about this, with a fairly high degree of confidence, as does Emily Bender with, I suspect, an even higher degree of confidence, we would be dishonest to deny that some equally qualified colleagues disagree with us.² Of course, humans don't answer questions by predicting what a human would say, but the move from describing an LLM as matching and extending patterns to saying it is predicting can be challenged.

The second is that the move involves a level confusion.³ When we set a student an exam and assess their cognitive capacities on the basis of how well they complete it, we do not care about their brains or 'training'. Our judgements about human cognitive capacities are based on our interactions with them. So if we have a similar interaction with an LLM, but then deny it has the cognitive capacities we would attribute to a human on the basis of that interaction, merely on the grounds that we know something about the mechanisms that allowed it to achieve that, we are applying standards to the LLM that we do not apply to humans. It is a form of chauvinism.

Tests vs applications

Maybe a better way to resist the conclusion is to think about the fact that we are comparing humans and LLMs on a task which is actually very artificial for a human. When we teach our undergraduates logic, the capacity we want then to acquire is not the capacity to pass a logic exam. Passing a logic exam is a consequence of possessing the capacity, so a useful test of it, but not constitutive of it.

Imperfect analogy: one could train a person or an autonomous car to pass a driving test without giving them/it the capacity to drive safely in other contexts.

So one way of blocking the conclusion that the LLM has the cognitive capacity is by arguing that it was 'trained to the test'. That may be true in many cases, but as a response to the form of argument we are considering, it is too ad hoc.

Rather, it seems to me that the fundamental difference between merely being able to pass a test and possessing the full capacity resides in the fact that a test is an artificial context in which it is explicitly determined which capacity is to be exercised and on what.⁴ Someone who possesses a cognitive capacity is not merely able to exercise it in a context where it is determined that the capacity is to be exercised, but also to exercise it 'at will', i.e. to exercise it in contexts where nothing in the context explicitly calls for its exercise. The real capacity cannot be separated from the ability to judge when it should or should not be used.

Thus when we teach logic to our undergraduates, what we hope we are teaching is a cognitive capacity which they will at later stages of their course find appropriate opportunities to use. We will only very rarely prompt them ('has X just affirmed the consequent or were they implicitly asserting a biconditional?') because where we want them to be intellectually is in a position to spot when the exercise of that capacity will be useful or appropriate.

Asserting this further condition for possessing a cognitive capacity beyond merely passing the test is not in itself an argument that LLMs lack cognitive capacities. Thinking or reasoning models are deliberately designed to give the impression that they are judging what they should do, what capacities of theirs to employ, in response to a prompt. Rather, the point is that this blocks the specific form of argument we are considering:

1. This LLM is able to describe a given human cognitive capacity (to attribute authorial intention, to strategise for world domination, to argue deductively) in precise detail,
2. and apply it to a particular case.
3. Therefore, this LLM has the cognitive capacity.

At best, this form of argument only provides evidence that the LLM can pass the test.