Towards a Formal Scientific Epistemology

Reevaluating Garrabrant induction

In my post “Why I’m not a Bayesian”, I argued that the Bayesian approach of assigning credences to propositions with binary truth values only works in simple and restricted domains. Instead, I claimed, a better approach to epistemology is to assign degrees of truth to models of the world.

This approach is broadly inspired by science, which is the domain from which we have the most evidence about which epistemological practices allow us to solve very hard problems. We don’t currently have a complete theory of scientific epistemology, but we can identify some important differences between scientific epistemology and Bayesian epistemology. Central examples of Bayesian epistemology (such as Solomonoff induction) assume that the truth lies within the class of hypotheses being considered. Conversely, in central examples of scientific research, the truth is definitely not already under consideration: the main problem is to come up with any hypothesis that explains existing data.

Another way of putting this point: Bayesian epistemology is entirely about empirical updates, whereas science is mostly about the process of constructing new theories. In some cases, once you’ve constructed a theory, you can be confident that it’s close to the truth merely from how well it fits existing data. But scientific theories are only fully accepted after they make successful advance predictions. That’s another difference compared with Bayesian epistemology, which treats retrodictions as equivalent to predictions.

In general I think scientific epistemology is far superior as a guide for thinking about difficult problems (like AI alignment) than Bayesian epistemology. However, scientific epistemology has mostly been described informally—e.g. by Popper, Kuhn, Feyerabend, etc. Popper did attempt to formally define a metric for degrees of truth, but it wasn’t very successful. I’d like to be able to describe scientific epistemology as formally as we can describe Bayesian epistemology (and ideally to unify them in a single framework).

---

I think that Garrabrant induction (also known as logical induction) is a major step towards formalizing scientific epistemology. This is an update compared with my position in my previous post, in which I critiqued Garrabrant induction in passing for its focus on assigning credences to propositions. However, in the process of assigning credences to propositions, Garrabrant induction also assigns something like degrees of truth (which it calls “wealth”) to something like models of the world (which it calls “traders”). So my critique was pretty off-base, in a way which I’m surprised nobody called out in the LessWrong comments. (Indeed, I’d even identified some of Garrabrant induction’s nice properties in a previous comment. This is a useful lesson on the pitfalls of writing posts about what you’re against rather than what you’re for.)

The key idea of Garrabrant induction is a market mechanism which sets credences for logical statements (including statements about the Garrabrant inductor itself) as the prices in a prediction market on whether those statements will eventually be proved. The traders in this prediction market are simply all polynomial-time algorithms, iteratively enumerated and given some starting wealth. Traders who are more successful will end up with more wealth, giving them greater power to move market prices.

Traders share a number of properties with scientific theories (which Bayesian hypotheses lack). At each point in time, most traders/theories aren’t yet under consideration. The ones that are under consideration don’t need to make predictions about everything that happens—instead, they can focus on making whichever predictions are most surprising and novel. Also, unlike Bayesian hypotheses, traders/theories aren’t mutually exclusive: an ideal reasoner would have many of them focusing on different domains.

While Garrabrant induction was formulated as a way of predicting mathematical theorems, we can imagine the same algorithm predicting a stream of input data about the world. What else would that version of Garrabrant induction need to be a good formal theory of scientific epistemology? Four things seem most prominent:

1. The ability for old evidence to support new theories.

2. The difference between traders and models.

3. The ability to modify traders.

4. The difference between wealth and degree of truth.

Abram Demski already touched on many of these points in this post and others in the same sequence. I don’t claim much novelty here, but for some reason it took me a very long time to fit Garrabrant induction into the “replacement for Bayesian epistemology” slot in my ontology—perhaps because it was originally framed more as an extension of Bayesianism than a replacement for it.¹ So further elaboration of this perspective seems helpful.

The problem of old evidence

Garrabrant induction and Solomonoff induction take very different positions on the problem of old evidence. In Solomonoff induction, there’s no distinction between old evidence and new evidence—they’re treated symmetrically. Whereas in Garrabrant induction, traders only ever gain wealth from predicting new evidence—retrodictions of old evidence are irrelevant.

Scientific epistemology takes a middle ground. Advance predictions of new evidence are weighted much more highly than retrodictions, but old evidence can still support a theory. Intuitively speaking, one reason why retrodictions should be discounted is that a theory might have been designed with that old evidence in mind, and therefore crediting it with predicting that evidence is a kind of overfitting or double counting.

Solomonoff induction doesn’t care about this, because it has a mechanism for preventing overfitting: assigning more complex hypotheses lower prior probability. One extra bit of description length might “smuggle in” information which allows the hypothesis to predict old evidence, but it’ll also halve the prior probability of that hypothesis. And if a hypothesis can more than double its probability relative to other hypotheses using just one extra bit, then it must be compressing information more efficiently, which is actually what we want.

In scientific epistemology, however, we don’t have any clear way of measuring the complexity of a given hypothesis, since it’s implemented within a big messy neural network. Even when the theory is described by precise equations, using those equations to make predictions requires the use of “auxiliary hypotheses” in which it’s often possible to hide a lot of complexity. And so in general it’s not possible to mechanistically penalize hypotheses for “smuggling in” old evidence.

However, it seems like this is the kind of thing that Garrabrant induction traders could take into account if given enough information about each other. This seems related to the concept of trading under adverse selection. In normal financial markets, other traders sometimes know more than you. So when market-making you need to set bid-ask spreads, because the expected value of a stock conditional on someone buying it from you is higher than the expected value of a stock conditional on someone selling it to you.

The implementation details do seem tricky, though. In Solomonoff induction, when you add a new hypothesis you can just go back and evaluate how it would have done on all the old evidence, which is equivalent to it having been there all along. But in Garrabrant induction, predictions move the market prices, and so intuitively it seems like you’d need to rerun the whole market. It’s also unclear how traders should be made aware that some of their competitors are “from the future”. It seems like we might need to bake in some notion of situational awareness, which seems complicated. (For more on this, see this post by Abram.)

Traders vs models

Scientific theories need to make predictions, but there’s no standard way to translate those predictions into bets. By contrast, traders in Garrabrant induction need to make bets, but those bets need not be driven by predictions. Traders in Garrabrant induction are any (and eventually every) polynomial-time algorithm. These include very simple algorithms, like ones which notice when “A OR B” is cheaper than either A or B individually, then arbitrage that difference, without having any opinion on how likely A and B actually are. (Analogously, many human actions are driven by reflexes or heuristics rather than explicit beliefs about what outcomes those actions will cause.)

However, in the long term it seems likely that the biggest wins will accrue to traders which implement models containing important insights that other traders lack, then bet that those models are right. This seems particularly true if we focus on the domain of science. Yet those traders might still use a wide range of trading strategies to convert their internally-represented beliefs into actual bets. It would be nice if we could demonstrate that almost all wealth will eventually accrue to traders which use a given kind of trading strategy (e.g. Kelly betting).

Modifying traders

Traders in Garrabrant induction are generated by enumerating every polynomial-time algorithm in order of simplicity. However, an important part of scientific epistemology is the process of identifying which new theories to consider next, especially via improving existing theories. In one sense, traders can already improve via taking their trading history into account when making new trades. But it would be nice if this were more continuous with the process of adding new traders.

One way to augment Garrabrant induction to account for the process of theory design would be if the existing traders could influence which new traders are added each day. But that doesn’t quite capture what we want, because in scientific epistemology new models evolve from old models and inherit much of their credibility. A theory that has one wrong belief can still be patched in a way which allows it to retain most of the credit for its previous correct predictions. So perhaps traders could be allowed to “donate” their wealth to other traders. More generally, if traders are allowed to invest in each other, then this allows them to represent higher-level concepts composed of the concepts represented by other traders, without needing to reimplement those same concepts internally.

However, all of this makes the concept of “trading strategies” much more complicated—now it’s about relationships between different traders. And I’m uncertain which of these suggestions are adding important innovations, versus adding unimportant details that the original formulation of Garrabrant induction correctly abstracted away.

Wealth vs truth

Making a bunch of wealth certainly suggests that a trader has an approximately-true model of the world. But the key difficulty with interpreting wealth as degree of truth is that wealth is rivalrous, whereas degree of truth isn’t. If a mostly-true theory reallocated its wealth between many slightly-different variants of itself, all of them would still be mostly true, but each of them would have much less wealth. More generally, gaining the most wealth requires betting against the consensus, and so contrarian traders might outcompete conformist traders even if they’re less correct overall than any given conformist. We could try to group traders into clusters, and talk about the degree of truth of each cluster—however, that just moves the same problem to a higher level.

When we face difficulties in defining a concept like degrees of truth, a useful question is “what do we want to use the concept for?” One answer is that traders whose models are more true should get more influence over our actions (given some mechanism for hooking up a logical inductor’s outputs to actions, which I’ll leave unspecified here). But this is still a rivalrous criterion, because our actions need to be determined by some set of traders. However, a less rivalrous version of this answer is that a model’s degree of truth affects how much we trust it to influence our actions relative to non-model-based policies. This seems to intuitively track scientific epistemology. When even our best theories of a phenomenon are quite bad, we’d prefer to rely on intuitions, habits, or traditions that have worked in the past (even when we don’t know why they work). Conversely, when we’re confident that our best theories are very close to the truth, we’re willing to follow even very counterintuitive recommendations from them.

I don’t know how to pin down the distinction between model-based and model-free traders, but it seems related to the concept of gears-level understanding. Eliezer also discussed some related points in this post (see also my comment beneath it).

1. For example, in this post Abram identifies some ways that understanding Garrabrant induction should change how Bayesians think about hypotheses. But Bayesian hypotheses are so different from Garrabrantian traders that using the same term for both seems misleading. In particular, the former are assigned credences, while the latter aren’t! That’s a much more fundamental change than the ones Abram identifies.