> "The AI 2040 authors might justify their portrayal of the deal by saying that slowing down more is not politically feasible. But in doing so, they’ve advocated for faster progress than almost anyone else endorses, as well as a more rapid handover of power to AIs than almost anyone else endorses."
Plan A is still insane! I worry that calling it Plan A normalizes that insanity (misses the mood, if you will). I respect that the authors are also trying to be realist, but I share your worry that establishing this as The Plan will bias a deferential AI safety community to not aim higher, lest the thought-leaders think that would be naive.
Interesting stuff. I find it curious that the prophets you listed (Legg, Amodei, Kokotajlo, Leike, Kurzweil, and I'll add Moravec) really run the gamut between optimism and pessimism. IIRC Legg and Kokotajlo are both very pessimistic about outcomes; Amodei is worried, but seems to spend more time in the national-competition frame than in the classical alignment frame; and Leike/Kurzweil/to some extent Moravec are optimistic, if only because the latter two are very disinterested in maintaining a 21st-century lifestyle for humankind.
Legg was middle-of-the-road back in 2011. I think he’s gotten more pessimistic since then but don’t have a source. Leike has been an optimist since 2022 ish
TLDR AI watchers have over-indexed on peak capability to the detriment of average reliability, which is much lower than often appreciated for most white collar work, and represents a huge bottleneck.
I think another big reason for the capability / practical impact gap, which you partially touch on re memorization, is that a lot of the highest quality work out there in whatever domain is only as good as it is because of its most out-of-distribution aspects, i.e. whatever makes it notably more creative or rigorous or ambitious than the field. And since AI outputs tend toward the middle of the distribution due to the probabilistic nature of LLMs, it is rare, or at minimum requires a ton of complex elicitation by a very smart human (who then himself becomes a key bottleneck), for their output to be as impressive as it would need to be to change the world beyond what the best human experts can already do.
I think a lot of folks' failure to understand this stems from not appreciating the contingency of AI performance on good data, and the related insight that we have almost no codified data on just those out-of-distribution properties distinguishing the very best work from middle-of-the-road work.
You argue that the governments of US and China may not want to race. But this misses the point that governments are not needed for a race to happen. Particularly in the US, private companies are capable and willing to race on their own. The US government would have to make an active decision to block a race, at the cost of short term wealth and national security. That is a much harder thing to rely on.
There's a nice white-pill in the cooperation point; if progress is bottlenecked by cooperation (which is very likely; see science in ancient times or renaissance compared to science during the heyday of the republic of letters), that means meaningful AI progress need to go hand in hand with advanced forms of cooperation.
AI has not achieved even one transformative objective that improves the lives of large numbers of ordinary people. Three of my real world benchmarks are curing tuberculosis, curing malaria, and engineering cereal grains to fix their own nitrogen. Truly superintelligent AI that deserved that name should be able to do all of these. If not, color me mostly unimpressed. And I say this as someone who loves the tools and uses them every day. But I’m a college professor and a research scientist, so I’m not at all representative ordinary people.
One possible complement to this argument is that most AI economics focuses on the supply side (capabilities, productivity, deployment), while giving much less attention to aggregate demand.
To paraphrase Keynes, production ultimately exists because there is demand for it. Even if AI removes most production bottlenecks, firms only automate because they expect someone to buy the output. If automation reduces labour income faster than it creates new sources of purchasing power, demand itself may become the next bottleneck. In that world, the gap between AI capabilities and economic impact isn't just explained by deployment frictions or "weak links", it's also constrained by who can actually absorb all the additional production.
This is a super interesting point "... human history, which contained many periods during which a large number of smart people failed to make significant technological progress, or actively regressed." I agree that what AI has actually done has not transformed society yet, but I hadn't thought about it following this trend.
The most optimistic part of AI2040 for me was the portrayal of how humans and gov would adapt to no longer working and paying taxes (2036: rich, happy, healthy, fine). Seemed very much like the authors saying 'don't worry AI will solve all the human problems'. I think it shows a real lack of an understanding of the chaos of this world and humans in general. I suppose this is a detail that you can't dive into when the framing is problematic, but it was striking to me.
There will be no good ending for humanity if we create a different species far more capable than us. When has that ever worked out well for another species? What is the prior?
Even in the supposed good ending, no trajectory where humanity is reduced to a useless species like pet chihuahuas will be better than the trajectory humanity has enjoyed the last 60000 or 300000 years.
Best chance for human survival? Realists in china realise this and win the next world war starting very soon and do worldwide re-education afterwards. The USA and liberal psychology itself are on the side of human extinction and disempowerement.
It's a bit strange that they didn't learn their lesson about the title, despite being criticized for it with AI 2027.
It should have just been called AI: Plan A, and I noticed that Scott Alexander did not reference the date in his blog title about it.
I especially liked this bit:
> "The AI 2040 authors might justify their portrayal of the deal by saying that slowing down more is not politically feasible. But in doing so, they’ve advocated for faster progress than almost anyone else endorses, as well as a more rapid handover of power to AIs than almost anyone else endorses."
Plan A is still insane! I worry that calling it Plan A normalizes that insanity (misses the mood, if you will). I respect that the authors are also trying to be realist, but I share your worry that establishing this as The Plan will bias a deferential AI safety community to not aim higher, lest the thought-leaders think that would be naive.
Interesting stuff. I find it curious that the prophets you listed (Legg, Amodei, Kokotajlo, Leike, Kurzweil, and I'll add Moravec) really run the gamut between optimism and pessimism. IIRC Legg and Kokotajlo are both very pessimistic about outcomes; Amodei is worried, but seems to spend more time in the national-competition frame than in the classical alignment frame; and Leike/Kurzweil/to some extent Moravec are optimistic, if only because the latter two are very disinterested in maintaining a 21st-century lifestyle for humankind.
are you sure you didn't switch up Legg and Leike? I didn't know Legg was so pessimistic? And Leike was relatively optimistic?
Legg was middle-of-the-road back in 2011. I think he’s gotten more pessimistic since then but don’t have a source. Leike has been an optimist since 2022 ish
https://aligned.substack.com/p/alignment-optimism
https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-with-shane-legg-on-risks-from-ai
On the question of why impressive capability gains haven't yielded greater practical change, this recent piece of mine may be of interest: https://arachnemag.substack.com/p/ais-reliability-gap.
TLDR AI watchers have over-indexed on peak capability to the detriment of average reliability, which is much lower than often appreciated for most white collar work, and represents a huge bottleneck.
I think another big reason for the capability / practical impact gap, which you partially touch on re memorization, is that a lot of the highest quality work out there in whatever domain is only as good as it is because of its most out-of-distribution aspects, i.e. whatever makes it notably more creative or rigorous or ambitious than the field. And since AI outputs tend toward the middle of the distribution due to the probabilistic nature of LLMs, it is rare, or at minimum requires a ton of complex elicitation by a very smart human (who then himself becomes a key bottleneck), for their output to be as impressive as it would need to be to change the world beyond what the best human experts can already do.
I think a lot of folks' failure to understand this stems from not appreciating the contingency of AI performance on good data, and the related insight that we have almost no codified data on just those out-of-distribution properties distinguishing the very best work from middle-of-the-road work.
You argue that the governments of US and China may not want to race. But this misses the point that governments are not needed for a race to happen. Particularly in the US, private companies are capable and willing to race on their own. The US government would have to make an active decision to block a race, at the cost of short term wealth and national security. That is a much harder thing to rely on.
There's a nice white-pill in the cooperation point; if progress is bottlenecked by cooperation (which is very likely; see science in ancient times or renaissance compared to science during the heyday of the republic of letters), that means meaningful AI progress need to go hand in hand with advanced forms of cooperation.
Really useful, thanks
AI has not achieved even one transformative objective that improves the lives of large numbers of ordinary people. Three of my real world benchmarks are curing tuberculosis, curing malaria, and engineering cereal grains to fix their own nitrogen. Truly superintelligent AI that deserved that name should be able to do all of these. If not, color me mostly unimpressed. And I say this as someone who loves the tools and uses them every day. But I’m a college professor and a research scientist, so I’m not at all representative ordinary people.
One possible complement to this argument is that most AI economics focuses on the supply side (capabilities, productivity, deployment), while giving much less attention to aggregate demand.
To paraphrase Keynes, production ultimately exists because there is demand for it. Even if AI removes most production bottlenecks, firms only automate because they expect someone to buy the output. If automation reduces labour income faster than it creates new sources of purchasing power, demand itself may become the next bottleneck. In that world, the gap between AI capabilities and economic impact isn't just explained by deployment frictions or "weak links", it's also constrained by who can actually absorb all the additional production.
This is a super interesting point "... human history, which contained many periods during which a large number of smart people failed to make significant technological progress, or actively regressed." I agree that what AI has actually done has not transformed society yet, but I hadn't thought about it following this trend.
The most optimistic part of AI2040 for me was the portrayal of how humans and gov would adapt to no longer working and paying taxes (2036: rich, happy, healthy, fine). Seemed very much like the authors saying 'don't worry AI will solve all the human problems'. I think it shows a real lack of an understanding of the chaos of this world and humans in general. I suppose this is a detail that you can't dive into when the framing is problematic, but it was striking to me.
There will be no good ending for humanity if we create a different species far more capable than us. When has that ever worked out well for another species? What is the prior?
Even in the supposed good ending, no trajectory where humanity is reduced to a useless species like pet chihuahuas will be better than the trajectory humanity has enjoyed the last 60000 or 300000 years.
Best chance for human survival? Realists in china realise this and win the next world war starting very soon and do worldwide re-education afterwards. The USA and liberal psychology itself are on the side of human extinction and disempowerement.