It is not what you have got but what you do with it?

Somehow I have subscribed myself to a Substack called 'The Inner Exodus' which is about AI from a largely Christian (US-style) perspective and written by a clinical psychologist, Dr Sean Tobin. The position taken is what we might call 'cautious or qualified positivity'.

Yesterday's post, The Question is the Skill, was about the prompting skills needed to avoid cognitive offloading and make an LLM a genuine cognitive enhancement. As he concludes:

The real divide of the coming years will not run between the people who use AI and the people who refuse it. It will run right through the people who use it. On one side, those who hand it their thinking and slowly forget how. On the other, those who use it to think harder than they ever could alone.

What struck me about this piece was that Tobin is not wrong about his central claims: brief prompts with the 'Do X' and no more precise instructions will lead to cognitive offloading and its harmful effects, and the prompting tips he offers may well produce the benefits he describes.¹ And yet the piece has three really important blindspots, which facilitate the positivity.

1: Individualism and self-importance

Tobin is fully aware that all the benefits of using an LLM in the way he recommends are available from human exchanges as well. But he thinks LLMs have an advantage:

For most of history you needed a friend brave enough to push back. Most people never find one. Now there is a sparring partner on your desk that never tires, never takes it personally, and never lets you win to be polite. Use it that way.

This is an incredibly seductive thought. Finding someone willing and able to be your intellectual sparring partner is hard. The right combination of interest, understanding, and patience for you is difficult to find, and when you do, that person will have limited time and energy. What is not to like about a machine which can do it for us? And even better, everyone can have one.

The blindspot here is that it centres the self, conceived of as the solitary author of its thoughts and expressions. Thoughts and their written expressions are 'owned' by one person, but it is a mistake to think that we can understand them, and evaluate them, independently of the social and historical context - i.e. other people - in which they emerged.

If a thinker has human 'sparring partners', then that context is explicit and transparent, even if usually under acknowledged. One might try to argue that an LLM is just a neutral but opaque medium for the intellectual context which is its training data, but apart from being false, this misunderstands the very idea of our thought being situated in a specific context. For the training data of an LLM includes several distinct such contexts, and blends them in ways we cannot understand. It is like having a 'sparring partner' whose identity - social and historical context - keeps shifting undetectably.

And of course it is not neutral. LLMs are carefully aligned with certain values explicitly chosen by the companies that make them for commercial reasons (maximising use and avoiding political or legal attacks in markets they care about). Tobin may not have thought much about this² precisely because the values of Silicon Valley oligarchs may be quite close to his own values. But then it is still an echo chamber, but a much more subtle one.

The challenges of finding a human sparring partner serve an important epistemic function: they provide brakes on conviction and the sharing of opinion. Education, especially higher education, teaches epistemic humility, resulting in caution about hold convictions or sharing opinions until they have been appropriately tested in the human social and historical context where they must live. Offering an always available alternative will not produce better thinking - it will just produce conviction and expression of opinion rooted in the owner's monadic self and inadequately grounded in the actual context in which that self must live. In a slogan, it takes us towards a world of intellectual incels.

2: Cost-Benefit Analysis

In passing, Tobin describes LLMs as producing answers 'for free'. Suppose my first concern is mistaken and the LLM is a perfectly adequate replacement for a human sparring partner, with all the consequent benefits. We still need to ask what the costs are, and that we can access them 'for free' or for a reasonable monthly subscription, does not exhaust that question.

Firstly, LLMs are candidates for the most costly technology in human history. The low price to the user hides this fact because the business model is not current profits but future profits - hence the share prices of these companies, which are a measure of belief in future profits. The eye-watering levels of capital investment in AI by the leading companies far exceeds that of any previous technology. And that has an opportunity cost. All those trillions being invested in AI are not being invested in other enterprises which could be doing a huge amount of good for billions of the most deprived people in the world with the money. In general, capitalism incentivizes investment in enterprises which benefit richer people, but AI amplifies this by orders of magnitude because the investment requirements are of a scale to distort the global economy.

Secondly, there are the externalities, ethical, social and environmental. These are real costs, even if the current global economy does not make then financial costs for the business or the user. Are the benefits Tobin describes really worth those costs? I leave that as an exercise for the reader.

3: Yes but that isn't the plan

So far I have focussed on Tobin's second claim about the benefits of prompting well. But what about his first claim? He notes:

Most of us type into AI the way we type into Google. A few words. A vague request. What does this mean. Summarize this. And then we judge the tool by the thin thing it hands back. Google retrieves. AI reasons. Those are different faculties, and they take different instructions. A search engine wants a query. A reasoning partner wants an assignment. Imagine you hired the sharpest research assistant alive and set a forty-page paper in front of them. You would not ask what it means.

There is a lot to unpack here. Firstly, Google search only produces the highly relevant results it does because it does an awful lot more than just send your scrappy search query to a big database. To begin with, it rewrites the query to make it produce better results, sometimes in ways it tells you but always in ways that fill in likely - based on what it knows about you from its extensive data scraping and what it knows about which links have been clicked on by other users who made similar searches - context that you didn't bother typing. Then it filters and ranks the results also based on what it knows about you. So the 'magic' of Google search is largely to do with teh fact that Google are expert at guessing what you really meant by that vague request. Just try a search engine like Mojeek if you want to see what search is like without that 'magic': you need to write your queries really well to get the results you want.

Secondly, if you actually had 'the sharpest research assistant' you could give a very vague request like 'Summarise this', because they would have the skills to summarise well and the knowledge of the context - who you are, what your goals are etc. - to produce what you wanted from that vague request. You only need to do all the detailed structuring of the task and setting of sub-goals if you were training them up. The more you need to do that, the less useful the research assistant.³ The whole point of human beings as assistants or sparring partners or just random conversational partners is that they can fill in the gaps based on shared background knowledge and sensitivity to non-linguistic information. There are sophisticated theories of how this works, and to a huge extent it is based on shared social and historical contexts, so often goes awry if care is not taken in cross-cultural communication.

Thirdly, the companies producing LLMs know about my forst two points and their goal is to to use the sorts of techniques Google developed for search to make chatbots more like human assistants - able to use contextual cues and shared background to fill in the gaps and give you what you wanted when you gave a vague or unclear prompt. Their aim is to make the 'thin thing it hands back' progressively thicker until it is no longer necessary to have the prompting techniques Tobin describes. The vision is immensely powerful computational tools which you need no skills at all to use.

If they achieve this - and of course there is no guarantee they will - then Tobin's prediction of the 'divide' will turn out to be false. There may be specialist tools which require skilled prompting, but general consumer LLMs will not.