LessWrong Rationalism as Classical Rationalism

Apparently I once made a remark that modern LessWrong rationalism, especially when it discusses AGI and superintelligence, is surprisingly similar to “naive unreconstructed 17th-18th century rationalist epistemology.” The context of this was discussions about AGI/ASI, whether it was possible for an AI to “FOOM” by exceeding the sum of human knowledge without the restrictions of having to do experiments.

Biology is the example that skeptics of imminent transformative AGI in STEM cite as something bottlenecked by laboratory time, while AGI maximalists like Eliezer Yudkowsky like to use AlphaFold as an example to show that a lack of empirical data is not so much of a constraint, and that a hypothetical AGI could become immensely transformative immensely quickly. I’ve found that people like to schematize this disagreement by reaching out into intellectual history to define the two groups as consisting of empiricists and rationalists. Sometimes in debates on AGI timelines, I’ve seen those who stick closely to existing data and extrapolations being called empiricists, and those who reason from first principles rationalists. (Usually this is invoked to criticize the latter.)

It’s interesting that the specter of classical rationalism is still so present among STEM people. Rationalism is thought of as the seductive idea that you can sit down at your armchair and deduce facts about the world. Sometimes this is attributed to Plato and Aristotle, too, and there’s a kind of history where it’s assumed that people thought until Hume or Newton that you could sit at an armchair, and only with the scientific method did people actually decide to check if their theories were correct by looking out into the world. Personally, while I have not so much interest in the AGI debate, I’m intrigued by the persistence of this fantasy as a symptomatic formation.

LessWrong rationalism has always been dogged by the accusation of being re-skinned classical rationalism. Rationalists of the LessWrong tendency like to emphasize that they are empiricists, not rationalists in the early modern sense, and that the unfortunate coincidence of name comes from the fact that rationalism comes from rationality, not rational. The most explicit statement of this can be found in Scott Alexander’s Slate Star Codex post Why I Am Not Rene Descartes. Scott, having been a philosophy major, is more familiar with the history of philosophy than your average LessWronger. It’s a clear account but it doesn’t go deep into what I think are the significant characteristics of classical rationalism. Neither does it look at the AGI/ASI problem.

First, though, I think it’s important to take a second, clearer look at the rationalism-empiricism debate. It should not be thought of as a debate over the status of scientific knowledge. It does not concern whether scientific or some other specific kinds of knowledge can be deduced through non-experimental means. Consider Spinoza, the arch-rationalist. His claim is that there are three kinds of knowledge. All he can say is that knowledge through signs (the first kind of knowledge) is unreliable. Higher forms of knowledge derive from common notions between things (the second kind of knowledge) and an intuitive-rational knowledge of God (the third kind). Neither of the latter two can be thought of as the kind of knowing that leads me to contemplate my way to any number of scientific facts, like a new theory of gravity for instance.¹

We can also take Leibniz, who thinks that “crossed the Rubicon” is a predicate that can be known through an analysis of the concept “Julius Caesar”. But he is explicit that this is only possible for an infinite intellect i.e., God, and not by us. His reply to Locke on the status of innate ideas in the New Essays on Human Understanding is also relevant. Locke is famous for holding a tabula rasa account of the mind. ² As a sensualist, he holds onto the axiom that there is nothing in the intellect that was not first in the senses. Leibniz, on the other hand, believes in innate ideas. What is the status that we have to accord these innate ideas?

Leibniz even seems to think that all the ideas that we have are innate. If taken naively, this is obviously wrong. I do not innately know the colour of the lamp shining above me; I have to take a look, not meditate on the ideas already given innately in my mind. But we can simply take a look at Leibniz’s response to Locke in the New Essays on Human Understanding.

For if the soul were like such a blank tablet then truths would be in us as the shape of Hercules is in a piece of marble when the marble is entirely neutral as to whether it assumes this shape or some other. However, if there were veins in the block which marked out the shape of Hercules rather than other shapes, then that block would be more determined to that shape and Hercules would be innate in it, in a way, even though labour would be required to expose the veins and to polish them into clarity, removing everything that prevents their being seen. This is how ideas and truths are innate in us – as inclinations, dispositions, tendencies, or natural potentialities, and not as actualities; although these potentialities are always accompanied by certain actualities, often insensible ones, which correspond to them. ³

Experience plays the role of the “labour … required to expose the veins and to polish them into clarity, removing everything that prevents their being seen.” Innate ideas exist as potentiality in the immaterial mind (this is Leibniz’s Platonistic heritage, and part of his attack on materialism). Why does Leibniz go all the way to defend a theory of innate ideas if he ends up agreeing with Locke on the importance of empiricism? ⁴ There is the polemic against materialism and the possibility of “thinking matter” that I mentioned, but also implicated is psychology. Broadly speaking the empiricists think of the mind as being passive, assembling its complex ideas out of the juxtaposition of simpler ideas (what Kant would call receptivity), while Leibniz wants to think of cognition as being active (what Kant would call spontaneity). Finally, there is the status of ideas that would necessarily exceed experience, and which might undergird it.

The status of the empiricism-rationalism debate, for me at least, hinges less on science and more on these problems. ⁵ 1) Philosophical psychology and the nature of cognition. 2) The status of ideas that transcend experience (to use the prototypical Kantian examples, God, the Soul and the World). 3) The foundation of empirical science, and the possibility of science having a rational structure (even this does not mean that we can “read off” scientific facts solely from this structure).

So the notion of the a priori should not be thought of in the sense of “in the absence of sensations” (this is a misunderstanding that I have come across in conversation). Rather, it has to be considered. This of course means that Eliezer Yudkowsky is being extremely silly when he says the following about the a priori.

If the brain has an amazing "a priori truth factory" that works to produce accurate beliefs, it makes you wonder why a thirsty hunter-gatherer can't use the "a priori truth factory" to locate drinkable water. It makes you wonder why eyes evolved in the first place, if there are ways to produce accurate beliefs without looking at things. ⁶

In fact, we shall see why it is silly for Eliezer Yudkowsky in particular to make claims about the a priori, given the important role that it plays in my opinion in his Sequences. Where I think that the rationalism-empiricism debate is relevant here is the classical-rationalist acceptance of the constitutive roles that apodictic principles hold for physical phenomena. I take the word “constitutive” from Kant, though for him such principles apply only to appearances, though for the classical rationalists they apply to the things themselves.

For example, Leibniz accepts the postulate of the principle of continuity, i.e. “Nature makes no leaps.” Leibniz, however, does not seem to think that it can be derived from the principle of sufficient reason like his other principles and adduces theological reasons for this. I believe that for Leibniz the principle of continuity is justified in that we would expect to God to ensure so in the best of all possible worlds. Leibniz draws on this principle in his metaphysical work (in considering the hierarchy of monads), but also in his work in physics and mathematics. That all change proceeds through degrees and that there are no sudden jumps is of obvious relevance to Leibniz’s invention of the infinitesimal calculus. It was also important in his physics, as Julian Barbour comments on Leibniz’s critique of Descartes' proposed laws of collision.

We may also note how unphysical was Descartes’ instinct: the a → 0 limit in (9.4) is not at all the same as (9.3), so that initial conditions differing infinitesimally lead to totally different outcomes. This observation was the basis of a devastating criticism of Descartes’ rules made by Leibniz. ⁷

There is an empirical argument here, but this also follows from Leibniz’s insistence that a systematic mechanics should follow the principle of continuity, we should not expect that there are mathematical discontinuities that exist in nature (unless we have a damned good reason). After Leibniz, we have Maupertuis, who uses a Leibnizian argument that trajectories must be optimal in some fashion (since we are in the best of all possible worlds) and hence there is some particular quantity that is minimized. While Maupertuis’ presentation was flawed, his work has obviously led to the principle of least (now stationary) action in physics, and thus the whole of analytical mechanics. ⁸

What is the status of these a priori principles? It’s especially important given that the principle of least action is an accepted part of mathematical physics today. ⁹ These principles are thought of by the rationalists to be apodictically justified through reason and structuring physical phenomena in some fashion. But they cannot be taken as determinate facts that allow one to directly come to scientific theories, and have to be conjoined with ad hoc hypotheses or those tested by experiment. They have to be justified like any other normal science.

What I am trying to get at is that Leibniz’s principle of continuity or even Maupertius’ belief in the economy exemplified by mechanics cannot be thought of as having the status of facts or something that allows you to deduce from your armchair various empirical facts, like the fall of a ball to the ground. The relationship between their principles and empirical phenomena I think can be thought of as the relationship between the principle of stationary action and actual mechanics. You can accept the former as true apodictically or by definition, and you can think that it underlies all mechanics, but at the same time you cannot deduce mechanics as a whole from it (you have to introduce masses, potentials, coordinate systems… Though you will never feel the need to drop a ball thirty times to calculate the amount of action to see if it is stationary or not).

I have gone on quite the detour into the history of philosophy and physics. It is now time to relate this to the original topic, rationalism. One of the core assumptions when it comes to the FOOM hypothesis is that AIs, at least in theory, are able to extract vast amounts of information from limited empirical data. Depending on how sparse this empirical data is, I think that this can only be justified through an implicit belief in such constitutive apodictic principles. The boldest and perhaps most bizarre assertion of this comes from Yudkowsky’s That Alien Message, where he makes this prediction.

Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

His platonic ideal of a reasoner is Solomonoff Induction, which he describes below with a small modification. Solomonoff induction is a hypothetical algorithm proven to be optimal at sequence-prediction, which works by selecting all the programs that are able to produce the first n terms of the sequence given to it, and predicting the next by weighting all the programs by length. Yudkowsky’s revision is that he elides “program” into “universe” (as a possible set of laws of physics) and experience as observation.

To see the theoretical limit on extractable information, imagine that you have infinite computing power, and you simulate all possible universes with simple physics, looking for universes that contain Earths embedded in them—perhaps inside a simulation—where some process makes the stars flicker in the order observed. Any bit in the message—or any order of selection of stars, for that matter—that contains the tiniest correlation (across all possible computable universes, weighted by simplicity) to any element of the environment, gives you information about the environment.

I did not think too much about this assertion of Yudkowsky when I first heard of it, but it bothered me enough to wonder about a while ago. Do three frames of an apple falling really contain enough information to reconstruct Newtonian physics and even promote General Relativity as a possible hypothesis? I assume that Yudkowsky still stands behind this claim, as do many rationalists involved in LessWrong and MIRI. ¹⁰ I found a critique of it by a computational physicist here that I personally find convincing.

The comments, though… Of course, since Eliezer Yudkowsky himself is not in this thread, I don’t know if he would endorse any of the arguments given here. But they are given in a Yudkowskyian tenor and they seem the most plausible arguments for Yudkowsky’s position. One commenter on another thread even claims that the apple-physics problem does not need a hypercomputer and that the “first few steps to be achievable by a smart-but-not-exceptionally-smart and mathematically inclined programmer with a decently powerful computer, given enough time.”

All the commenters who agree that this is possible seem to think that from a single frame an AI could deduce that here we have a three-dimensional object embedded into a two-dimensional plane. From there, we could proceed from minor chromatic aberration to deduce optics and quantum mechanics. Or that you could deduce the fall from the three frames, and from a Galilean-ish law, proceed to posit that this is a scene from an entire universe (since a simulation of an entire universe is simpler than a simulation of a portion of it), ending up in evidence for an inverse-square law of gravity, and so on.

I am summarizing the argument, and some of the commenters are obviously quite intelligent and have a good grasp of physics and mathematics. All the same I am not convinced; I think that if you gave a Solomonoff induction hypercomputer the three frames you might get a very compressed demoscene program and perhaps a few more frames of it falling. What I found perplexing is the assumption that you can apply Occam’s Razor recursively and that you could deduce through “simplicity” the successive higher levels, so that we can deduce the existence of an entire universe from these three frames. While Yudkowsky might claim that I do not have enough imagination about what could be extracted from limited data, I might point out that Yudkowsky does not have enough imagination about what we might find in the space of all possible programs on some Turing machine, and we can expect almost all of them to be incomprehensible and strange and have no relation to physics. It's unjustified metaphysics to insist that by definition the universe is actually identical to the shortest program that generates it, instead of taking program length as a useful heuristic.

To save this, Occam’s Razor has to be taken as a constitutive principle when it comes to the things themselves. Occam's Razor under Yudkowsky's re-interpretation, anyway. For Yudkowsky, all that matters when it comes to simplicity is that it can be represented by a shorter program, as compared to a longer program. The idea comes from Kolmogorov, Solomonoff, Chaitin. A theory of the laws of physics can be thought of as a computer program that produces a series of observations once started (perhaps given some starting conditions), and the simpler the better. ¹¹ Or rather it can even be thought of as a universe, and so what Solomonoff Induction does is to iterate over universes. ¹² It is simpler if it has fewer “types” i.e. a theory is simpler when it has lesser classes of entities. It does not matter how long this program takes to run. This of course makes this hard to work out in practice. Yudkowsky, for instance, thinks that the laws of physics contain about 500 bits of information, so we can expect that a program with 500 bits would be able to simulate our universe. Now of course it might take longer than the lifetime of the universe to run it even for a second, but that does not mean that it is wrong. And if we had 500 bits of information, we could select it out of a pool of two to the power of five hundred program-hypotheses.

I do not think that this computational version of Occam’s Razor, while definitely a useful heuristic, is able to deduce from a 2D grid that it is the projection of a 3D object. The only way in my opinion to square it is to decide that Occam’s Razor plays a role similar to that played by, say, Leibniz’s natura non facit saltus. I would go further and say that this applies to the LessWrong use of “anthropics”, too. It is the extension of the anthropic principle in cosmology, which might be used to argue that an ASI given only the three frames might be able to deduce that it has a creator and that it has been put in an artificial situation, and if unaligned could use this to harm said creators. Bayesian natural theology? I think you can use apply this to Yudkowsky’s claim that cognition by definition is Bayesian in structure, and that this structure is equivalent to thermodynamics.

I think that if we take this frame seriously enough, you can take Yudkowsky’s account of science as following this rationalist direction. His sequence of posts in Science and Rationality make this the most clear. Yudkowsky says that Science even as an ideal, as a social system based on empiricism, is necessarily less rigorous than Rationality. Science tells you that you can entertain any kind of hypothesis, no matter how ridiculous, as long as you promise to test it and change your mind accordingly. Bayes, on the other hand tells you which hypotheses to cull even before you do experiments. I do not mean that it’s not just in the sense that for example a Bayesian doing parapsychology experiments would rather reasonably give a lower prior for the existence of ESP. Rather, Yudkowsky seems to hold that the universe is such that we can reasonably expect that it minimizes minimum-message length, Kolmogorov complexity and so on. This is the main reason for his belief in the many-worlds theory of quantum mechanics. I shall quote from The Dilemma: Science or Bayes?

So now put on your Science Goggles—you've still got them around somewhere, right? Forget everything you know about Kolmogorov complexity, Solomonoff induction or Minimum Message Lengths. That's not part of the traditional training. You just eyeball something to see how "simple" it looks. The word "testable" doesn't conjure up a mental image of Bayes's Theorem governing probability flows; it conjures up a mental image of being in a lab, performing an experiment, and having the celebration (or public recantation) afterward.

Science-Goggles on: The current quantum theory has passed all experimental tests so far. Many-Worlds doesn't make any new testable predictions—the amazing new phenomena it predicts are all hidden away where we can't see them. You can get along fine without supposing the other worlds, and that's just what you should do. The whole thing smacks of science fiction. But it must be admitted that quantum physics is a very deep and very confusing issue, and who knows what discoveries might be in store? Call me when Many-Worlds makes a testable prediction.

Science-Goggles off, Bayes-Goggles back on:

Bayes-Goggles on: The simplest quantum equations that cover all known evidence don't have a special exception for human-sized masses. There isn't even any reason to ask that particular question. Next!

Now, I think that if Yudkowsky did come across irrefutable evidence that the “simplest quantum equations that cover all known evidence” have special exceptions for human-size maasses, he would of course rethink his support for many-world mechanics (though it does not refute Bayes itself, it would show in some way how much more perfectly it is simpler than MWI). But in the absence of that evidence, I think it is fair to say that Yudkowsky believes at least implicitly in constitutive principles like “phenomena are scale-invariant”.

These principles are able to determine the range of possible theories of physics, such that he can pull off his checkmate here drawing on his Bayes-goggles.

Okay, Bayes-Goggles back on. Are you really going to believe that large parts of the wavefunction disappear when you can no longer see them? As a result of the only non-linear non-unitary non-differentiable non-CPT-symmetric acausal faster-than-light informally-specified phenomenon in all of physics? Just because, by sheer historical contingency, the stupid version of the theory was proposed first?

Yudkowsky has to accept that even if he thinks that it is silly, the idea that large parts of the wavefunction disappear is still part of Science. An empiricist could believe it, or affect agnosticism, and it can only really be ruled out completely by Yudkowsky’s principles around anthropocentrism and so on. I do not think that he denigrates empirical evidence, but he does think that following these principles is guaranteed in theory to get to the right answer when you have limited information. The MWI example is not really for him a show of force for how a FOOM-ing AI might function. But I think that it at least makes it more plausible for a rationalist that a sufficiently intelligent AI given limited information could outstrip humans.

Of course when we come to AI today, a lot of LessWrongers (and others, too) think that a neural network is something like an approximation of Solomonoff induction. The idea dates to Jürgen Schmidhuber who thinks that intelligence is equivalent to compression, that Solomonoff induction is intelligence, and what neural networks do is to compress really hard. So it perhaps is plausible to think that an LLM trained on vast amounts of textual data, if given enough data and computing power, would be able to FOOM past humanity in numerous ways. While I am myself skeptical of FOOM scenarios, the idea that you can extract substantial information from existing data that we have is plausible. It applies better to protein folding than to three frames of an apple falling.