"The most likely answer"

A post on Mastodon by Robert McNees set me thinking about a subtle but important ambiguity which he neatly identified.

When you ask an LLM a question, it predicts the most likely answer, but that phrase is ambiguous. The most likely answer to a question might be either:

1. The answer most likely to be true
2. The answer most likely to be given (by ...)

Now, when you ask a human being a question and they answer in good faith, they will attempt 1). An LLM in contrast can only ever give 2). That isn't a limitation which better engineering can overcome, but the inherent nature of what it does.

Making stuff up

The debate about so-called AI hallucinations - i.e. making stuff up - has recently shifted its focus onto the way these systems are trained to always give an answer even if they don't 'know' the correct answer. But that is a mis-framing of the problem, because it incorrectly assumes that they are attempting to give type-1 not type-2. Rather the problem consists in the training data: they are trained on examples of answers to questions. It is the nature of the sources for the training data - pretty much everything that is online - that there are a vanishingly small number of cases where humans say, of a particular topic, that they don't know anything about it. After all, a blog post or newspaper article or research paper which identifies a topic and then simply admits ignorance and refrains from speculating isn't really worth publishing. (This is a bit like the problem in the sciences that negative results don't get published.)

This means that the LLM is trained in such a way that for absolutely any question, the most likely answer to be given is always an assertion, an attempt at giving a type-1 answer. All the cases where humans say nothing, or decline to answer, because they don't know the answer are pretty much absent from the training data, thereby massively skewing the calculation of the most likely answer to be given.

So it is really important that the LLM predicts the most likely answer to be given by someone who knows what they are talking about on the subject of the question.

Prompt Engineering

Now any LLM which has had Reinforcement Learning with Human Feedback and Fine-tuning will have been designed with a default value for the ellipses in "by ..."¹ and some interfaces allow you to adjust this explicitly (I particularly like Msty.ai for the way it makes this very upfront). But in general, writing a good prompt is the skill of filling in those ellipses in a way which, for your specific question, will increase the chance that type-2 is also type-1. This can be by specifying a role, giving explicit criteria, including specific context to be taken into account, and perhaps using Retrieval Augmented Generation.

The trouble is, that takes skill and - most importantly - domain-specific knowledge. For example, a typical patient asking an LLM a medical question will not write the prompt in such a manner that it gives the most likely answer a clinician would give in a consultation. Because a clinician is an expert who will extract, from the patient history and the medical records, the most relevant information but the patient's prompt will not give this crucial context.

The Dangers of LLMs

We can now clearly see the problem. LLMs can be made to give type-2 answers which are pretty much the same as the type-1 answers, i.e. they can be set up such that the most likely answer to be given is also the most likely to be true. But this either requires specialist tools where the defaults for "by ..." are set up carefully to bridge the gap between type-1 and type-2 for questions in the specialist area, or the sort of skilled prompt engineering which requires the user to already be highly knowledgeable in the domain of the question being asked.

However, the LLMs most people engage with - ChatGPT, Gemini, Copilot - are entirely general purpose tools. The companies marketing these know the power of a unified brand, a single interface, and an effortless experience. You can ask any question and a type-2 conversation will follow. Small-print warnings about possible inaccuracies and the need to double-check are ignored and the designers know they will be. As a result, users will receive many false, misleading, or simplistic answers.

The solution is not technical, in the sense of 'improving' the LLMs. The solution is commercial - change the way these general purpose LLMs are marketed, add some friction by requiring users to write more effective prompts, change the dialogue format so the LLM isn't making statements but reporting what someone might say, etc. etc. Of course, that is never going to happen because then they would be less interesting and fun, would get fewer users, and the narratives of unchecked user growth on which their share prices depend would be undermined. Instead they offer a simple text box and respond, often in some detail, to whatever vague or unclear prompt is typed there.

Put simply, if ChatGPT required the users to fill in a form with boxes for Persona, Context, Task, and Format before it gave a response, would it still have 700 million weekly users (almost 1 in 10 human beings alive)? Of course not, so it will always be an unsafe source of information and advice for the majority of its users.