Month: March 2023

Several Rationalists objected to my recent “Against AGI Timelines” post, in which I argued that “the arguments advanced for particular timelines [to AGI]—long or short—are weak”. This disagreement is broader than predicting the arrival of one speculative technology. It illustrates a general point about when thinking in explicit probabilities is productive and illuminating vs when it’s misleading, confusing, or a big waste of effort.

These critics claim that a lack of good arguments is no obstacle to using AGI timelines, so long as those timelines are expressed as a probability distribution rather than a single date. See e.g. Scott Alexander on Reddit, Roko Mijic on Twitter, and multiple commenters on LessWrong.¹

And yes, if you must have AGI timelines, then having a probability distribution is better than just saying “2033!” and calling it a day, but even then your probability distribution is still crap and no one should use it for anything. Expressing yourself in terms of probabilities does not absolve you of the necessity of having reasons for things. These critics don’t claim to have good arguments for any particular AGI timeline. As far as I can tell, they agree with my post’s central claim, which is that there’s no solid reasoning behind any of the estimates that get thrown around.

You can use bad arguments to guess at a median date, and you will end up with noise and nonsense like “2033!”. Or you can use bad arguments to build up a probability distribution… and you will end up with noise and nonsense expressed in neat rows and figures. The output will never be better than the arguments that go into it!²

As an aside, it seems wrong to insist that I engage with people’s AGI timelines as though they represent probability distributions, when for every person who has actually sat down and thought through their 5%/25%/50%/75%/95% thresholds and spot-checked this against their beliefs about particular date ranges and etc etc in order to produce a coherent distribution of probability mass, there are dozens of people who just espouse timelines like “2033!”.

Lots of people, Rationalists especially, want the epistemic credit for moves that they could conceivably make in principle but actually have not done. This is bullshit. Despite his objection above, even Alexander—who is a lot more rigorous than most—is still perfectly happy to use single-date timelines in his arguments, and to treat others’ probability distributions as interchangeable with their median dates:

“For example, last year Metaculus thought human-like AI would arrive in 2040, and superintelligence around 2043 … If you think [AGI arrives in] 2043, the people who work on this question (“alignment researchers”) have twenty years to learn to control AI.”

Elsewhere he repeats this conflation and also claims he discards the rest of the probability distribution [emphasis mine]:

“I should end up with a distribution somewhere in between my prior and this new evidence. But where?

I . . . don’t actually care? I think Metaculus says 2040-something, Grace says 2060-something, and Ajeya [Cotra] says 2050-something, so this is basically just the average thing I already believed. Probably each of those distributions has some kind of complicated shape, but who actually manages to keep the shape of their probability distribution in their head while reasoning? Not me.”

Once you’ve established that you ignore the bulk of the probability distribution, you don’t get to fall back on it when critiqued. But if Alexander doesn’t actually have a probability distribution, then plausibly one of my other critics might, and Cotra certainly does. Some people do the real thing, so let’s end this aside about the many who gesture vaguely at “probability distributions” without putting in the legwork to use one. If this method actually works, then we only need to pay attention to the few who follow through, and I’ll return to the main argument to address that. 

Does it work? Should we use their probability distributions to guide our actions, or put in the work to develop probability distributions of our own?

Suppose we ask an insurance company to give “death of Ben Landau-Taylor timelines”. They will be able to give their answer as a probability distribution, with strong reasons and actuarial tables in support of it. This can bear a lot of weight, and is therefore used as a guide to making consequential decisions—not just insurance pricing, but I’d also use this to evaluate e.g. whether I should go ahead with a risky surgery, and you bet your ass I’d “keep the shape of the probability distribution in my head while reasoning” for something like that. Or if we ask a physicist for “radioactive decay of a carbon-14 atom timelines”, they can give a probability distribution with even firmer justification, and so we can build very robust arguments on this foundation. This is what having a probability distribution looks like when people know things—which is rarer than I’d like, but great when you can get it.

Suppose we ask a well-calibrated general or historian for “end of the Russia-Ukraine war timelines” as a probability distribution.³ Most would answer based on their judgment and experience. A few might make a database of past wars and sample from that, or something. Whatever the approach, they’ll be able to give comprehensible reasons for their position, even if it won’t be as well-justified and widely-agreed-upon as an actuarial table. People like Ukrainian refugees or American arms manufacturers would do well to put some weight on a distribution like this, while maintaining substantial skepticism and uncertainty rather than taking the numbers 100% literally. This is what having a probability distribution looks like when people have informed plausible guesses, which is a very common situation.

Suppose we ask the world’s most renowned experts for timelines to peak global population. They can indeed give you a probability distribution, but the result won’t be very reliable at all—the world’s most celebrated experts have been getting this one embarrassingly wrong for two hundred years, from Thomas Malthus to Paul Ehrlich. Their successors today are now producing timelines with probabilistic prediction intervals showing when they expect the growth of the world population to turn negative.⁴ If this were done with care then arguably it might possibly be worth putting some weight on the result, but no matter how well you do it, this would be a completely different type of object from a carbon-14 decay table, even if both can be expressed as probability distributions. The arguments just aren’t there.

The timing of breakthrough technologies like AGI are even less amenable to quantification than the peak of world population. A lot less. Again, the critics I’m addressing don’t actually dispute that we have no good arguments for this, the only people who argued with this point were advancing (bad) arguments for specific short timelines. The few people who have any probability distributions at all give reasons which are extremely weak at best, if not outright refutable, or sometimes even explicitly deny the need to have a justification.

This is not what having a probability distribution looks like when people know things! This is not what having a probability distribution looks like when people have informed plausible guesses! This is just noise! If you put weight on it then the ground will give way under your feet! Or worse, it might be quicksand, sticking you to an unjustified—but legible!—nonsense answer that’s easy to think about yet unconnected to evidence or reality.

The world is not obligated to give you a probability distribution which is better or more informative than a resigned shrug. Sometimes we have justified views, and when we do, sometimes probabilities are a good way of expressing those views and the strength of our justification. Sometimes we don’t have justified views and can’t get them. Which sucks! I hate it! But slapping unjustified numbers on raw ignorance does not actually make you less ignorant.

---

[1] While I am arguing against several individual Rationalists here, this is certainly not the position of all Rationalists. Others have agreed with my post. In 2021 ur-Rationalist Eliezer Yudkowsky wrote:

“I feel like you should probably have nearer-term bold predictions if your model [of AGI timelines] is supposedly so solid, so concentrated as a flow of uncertainty, that it’s coming up to you and whispering numbers like “2050” even as the median of a broad distribution. I mean, if you have a model that can actually, like, calculate stuff like that, and is actually bound to the world as a truth.

If you are an aspiring Bayesian, perhaps, you may try to reckon your uncertainty into the form of a probability distribution … But if you are a wise aspiring Bayesian, you will admit that whatever probabilities you are using, they are, in a sense, intuitive, and you just don’t expect them to be all that good.

…

I have refrained from trying to translate my brain’s native intuitions about this into probabilities, for fear that my verbalized probabilities will be stupider than my intuitions if I try to put weight on them.”

Separately, “Against AGI Timelines” got a couple other Rationalist critics who do claim to have good arguments for short timelines. I’m not persuaded but they are at least not making the particular mistake that I’m arguing against here. ⮐

[2] It’s not a priori impossible that there could ever be a good argument for a strong claim about AGI timelines. I’ve never found one and I’ve looked pretty damn hard, but there are lots of things that I don’t know. However, if you want to make strong claims—and “I think AGI will probably (>80%) come in the next 10 years” is definitely a strong claim—then you need to have strong reasons. ⮐

[3] The Good Judgment Project will sell you their probability distribution on the subject. If I were making big decisions about the war then I would probably buy it, and use it as one of many inputs into my thinking. ⮐

[4] I’m sure every Rationalist can explain at a glance why the UN’s 95% confidence range here is hot garbage. Consider this a parable about the dangers of applying probabilistic mathwashing to locally-popular weakly-justified assumptions. ⮐

Some of my friends have strong views on how long it will be until AGI is created. The best arguments on the subject establish that creating a superintelligent AGI is possible, and that such an AGI would by default be “unfriendly”, which is a lot worse than it sounds. So far as speculative engineering goes, this is on relatively solid ground. It’s quite possible that as research continues, we’ll learn more about what sorts of intelligence are possible and discover some reason that an AGI can’t actually be built—such discoveries have happened before in the history of science and technology—but at this point, a discovery like that would be a surprise.

The loudest voices warning of AGI also make additional claims about when AGI is coming.¹ A large contingent argue for “short timelines”, i.e., for AGI in about 5-10 years. These claims are much shakier.

Of course, short timelines don’t follow automatically from AGI being possible. There is often a very long time between when someone figures out that a technology ought to work in principle, and when it is built in reality. After Leonardo da Vinci sketched a speculative flying machine based on aerodynamic principles similar to a modern helicopter, it took about 400 years before the first flight of a powered helicopter, and AGI could well be just as far from us. Since the discovery of nuclear physics and the construction of particle accelerators about a century ago, we have known in principle how to transmute lead into gold, but this has never actually been done. Establishing timelines to AGI requires additional arguments beyond its mere possibility, and the arguments advanced for particular timelines—long or short—are weak.

Whenever I bring this up, people like to switch to the topic of what to do about AI development. That’s not what I’m discussing here. For now I’m just arguing about what we know (or don’t know) about AI development. I plan to write about the implications for action in a future post.

1.

The most common explanation² I hear for short timelines (e.g. here) goes roughly like this: Because AI tech is getting better quickly, AGI will arrive soon. Now, software capabilities are certainly getting better, but the argument is clearly incomplete. To know when you’ll arrive somewhere, you have to know not just how fast you’re moving, but also the distance. A speed of 200 miles per hour might be impressive for a journey from Shanghai to Beijing (high-speed rail is a wonder) but it’s very slow for a journey from Earth orbit to the Moon. To be valid, this argument would also need an estimate of the distance to AGI, and no one has ever provided a good one.

Some people, like in the earlier example, essentially argue “The distance is short because I personally can’t think of obstacles”. This is unpersuasive (even ignoring the commenters responding with “Well I can think of obstacles”) because the creation of essentially every technology in the history of the world is replete with unforeseen obstacles which crop up when people try to actually build the designs they’ve imagined. This is most of the reason that engineering is even hard.

Somewhat more defensibly, I’ve heard many people argue for short timelines on the basis of expert intuition. Even if most AI experts shared this intuition—which they do not, there is nothing close to a consensus in the field—this is not a reliable guide to technological progress. An engineer’s intuition might be a pretty good guide when it comes to predicting incremental improvements, like efficiency gains in batteries or the cost of photovoltaic cells. When it comes to breakthrough technologies with new capabilities, however, the track record of expert intuition is dismal. The history of artificial intelligence specifically is famously littered with experts predicting major short-term breakthroughs based on optimistic intuition, followed by widespread disappointment when those promises aren’t met. The field’s own term for this, “AI winter”, is older than I am.

It’s worth a look at the 1972 Lighthill Report, which helped usher in the first AI winter half a century ago (emphasis added): 

“Some workers in the field freely admit that originally they had very naive ideas about the potentialities of intelligent robots, but claim to recognise now what sort of research is realistic. In these circumstances it might be thought appropriate to judge the field by what has actually been achieved than by comparison with early expectations. On the other hand, some such comparison is probably justified by the fact that in some quarters wild predictions regarding the future of robot development are still being made.

When able and respected scientists write in letters to the present author that AI, the major goal of computing science, represents another step in the general process of evolution; that possibilities in the nineteen-eighties include an all-purpose intelligence on a human-scale knowledge base; that awe-inspiring possibilities suggest themselves based on machine intelligence exceeding human intelligence by the year 2000; when such predictions are made in 1972 one may be wise to compare the predictions of the past against performance as well as considering prospects for the realisation of today’s predictions in the future.”

While there has been tremendous progress in software capabilities since the Lighthill Report was written, many of the experts’ “wild predictions” for the next 20-30 years have not yet come to pass after 50. The intuition of these “able and respected scientists” is not a good guide to the pace of progress towards intelligent software.

Attempts to aggregate these predictions, in the hopes that the wisdom of crowds can extract signal from the noise of  individual predictions, are worth even less. Garbage in, garbage out. There has been a great deal of research on what criteria must be met for forecasting aggregations to be useful, and as Karger, Atanasov, and Tetlock argue, predictions of events such as the arrival of AGI are a very long way from fulfilling them. “Forecasting tournaments are misaligned with the goal of producing actionable forecasts of existential risk”.

Some people argue for short AGI timelines on the basis of secret information. I hear this occasionally from Berkeley rationalists when I see them in person. I’m pretty sure this secret information is just private reports of unreleased chatbot prototypes before they’re publicly released about 2-6 weeks later.³ I sometimes get such reports myself, as does everyone else who’s friends with engineers working on chatbot projects, and it’s easy to imagine how a game of Telephone could exaggerate this into false rumors of a new argument for short timelines rather than just one more piece of evidence for the overdetermined “the rate of progress is substantial” argument.

Edit: Twitter commenter Interstice reminds me of an additional argument, “that intelligence is mostly a matter of compute and that we should expect AGI soon since we are approaching compute levels comparable to the human brain”, and links Karnofsky’s argument. Yudkowsky’s refutation of this approach is correct.

2.

It’s worth a step back from AGI in particular to ask how well this type of speculation about future technology can ever work. Predicting the future is always hard. Predicting the future of technology is especially hard. There are lots of well-publicized, famous failures. Can this approach ever do better than chance?

When arguing for the validity of speculative engineering, short timeline proponents frequently point to the track record of the early 20^th century speculative engineers of spaceflight technology like Tsiolkovsky. This group has many very impressive successes—too many to be explained by a few lucky guesses. Before any of the technology could actually be built, they figured out a great deal of the important principles: fundamental constraints like the rocket equation, designs for spacecraft propulsion which were later used more-or-less according to the early speculative designs, etc. This does indeed prove that speculative engineering is a fruitful pursuit whose designs should be taken as serious possibilities.

However, the speculative engineers of spaceflight also produced many other possible designs which have not actually been built. According to everything we know about physics and engineering, it is perfectly feasible to build a moon base, or even an O’Neill cylinder. A space elevator should work as designed if the materials science challenges can be solved, and those challenges in turn have speculative solutions which fit the laws of physics. A mass driver should be able to launch payloads into orbit (smaller versions are even now being prototyped as naval guns). But just because one of these projects is possible under the laws of physics, it does not automatically follow that humanity will ever build one, much less that we will build one soon.

After the great advances in spaceflight of the mid-20^th century, most of the best futurists believed that progress would continue at a similar pace for generations to come. Many predicted moon bases, orbital space docks, and manned Mars missions by the early 2000s, followed by smooth progress to colonies on the moons of outer planets and city-sized orbital habitats. From our vantage today, none of this looks on track. Wernher von Braun would weep to see that since his death in 1977 we have advanced no further than Mars rovers, cheaper communication satellites, and somewhat larger manned orbital laboratories.

On the other hand, technological advances in some fields, such as materials science or agriculture, have continued steadily for generation on generation. Writers throughout the 1800s and 1900s spoke of the marvels that future progress would bring in these fields, and those expectations have mostly been realized or exceeded. If we could bring Henry Bessemer to see our alloys and plastics or Luther Burbank to see our crop yields, they would be thrilled to see achievements in line with our century-old hopes. There is no known law to tell which fields will stall out and which will continue forward.

If today’s AI prognosticators were sent back in time to the 1700s, would their “steam engine timelines” be any good? With the intellectual tools they use, they would certainly notice that steam pump technology was improving, and it’s plausible that their speculation might envision many of steam power’s eventual uses. But the intellectual tools they use to estimate the time to success—deference to prestigious theorists, listing of unsolved difficulties, the intuition of theorists and practitioners⁴—would have given the same “timelines” in 1710 as in 1770. These methods would not pick out the difference ahead of time between steam engines like Savery’s (1698) and Newcomen’s (1712), which ultimately proved to be niche curiosities of limited economic value, and the Watt steam engine (1776), which launched the biggest economic transformation since agriculture.

3.

Where does this leave us? While short AGI timelines may be popular in some circles, the arguments given are unsound. The strongest is the argument from expert intuition, and this one fails because expert intuition has an incredibly bad track record at predicting the time to breakthrough technology improvements.

This does not mean that AGI is definitely far away. Any argument for long timelines runs into the same problems as the arguments for short timelines. We simply are not in a position to know how far away AGI is. Can existing RLHF techniques with much more compute suffice to build a recursively self-improving agent which bootstraps to AGI? Is there some single breakthrough that a lone genius could make that would unlock AGI on existing machines? Does AGI require two dozen different paradigm-shifting insights in software and hardware which would take centuries to unlock, like da Vinci’s helicopter? Is AGI so fragile and difficult to create as to be effectively out of human reach? Many of these possibilities seem very unlikely, but none of them can be ruled out entirely. We just don’t know.

Addendum: Several people object that none of this presents a problem if you give timelines probabilistically rather than as a single date. See Probability Is Not A Substitute For Reasoning for my response.

---

[1] I’ve even heard a secondhand report of one-year timelines, as of February 2023. ⮐

[2] Okay, the most common explanation people give me for short timelines is that they’re deferring to subculture orthodoxy or to a handful of prestigious insiders. But this essay is about the arguments, and these are the arguments of the people they defer to. ⮐

[3] On the other hand, if the Berkeley AI alignment organizations had more substantial secret information or arguments guiding their strategy, I expect I would’ve heard it. They’ve told me a number of their organizations’ other secrets, sometimes deliberately, sometimes accidentally or casually, and on one occasion after I specifically warned my interlocutor that he was telling me sensitive information and should stop but he kept spilling the beans anyway. I very much doubt that they could keep a secret like this from me when I’m actually trying to learn it, if it were really determining their strategy. ⮐

[4] The theorists and practitioners of the 1700s didn’t predict accurate “steam engine timelines”. Watt himself thought his machine would be useful for pumping water out of mines more efficiently than the Newcomen engine, and did not expect its widespread revolutionary use. ⮐