In what important real world contexts is it both practical and useful to attempt to estimate numerical probabilities of unique future events?
Probabilities of unique events?
One can ask this basic question in contexts of individual decisions — will I be happy if I marry this particular person? — but here let us consider Big Events which are unique, or with little data on relevant past events. Is it likely that there will be a self-sustaining human colony on Mars in 100 years? Well, notwithstanding Elon Musk, to me this seems very unlikely. If forced to state a probability, I might say 1/1000 on grounds of technological infeasibility and cost, but I have no confidence in that estimate and I doubt that I would have much confidence in anyone else’s estimate of the probability.
What about more significant Big Events? One can find discussions in various specific contexts, and the main purpose of this post is to tell the reader where to find a few such specific discussions. Perhaps no general analysis of our basic question
In what important real world contexts is it both practical and useful to attempt to estimate numerical probabilities of unique future events?
is possible — certainly I have never seen one. But here are some more specific examples.
To estimate the probability of nuclear war.
Here is a short 2021 debate between “father of the internet” Vinton Cerf and “father of public key cryptography” Martin Hellman, on the question of whether it is useful to try to estimate the probability of a full-scale nuclear war. Hellman argues for a chance of (very roughly) 1/100 per year, based on the length of time without such a war and the past frequency of major superpower crises. Cerf argues
I prefer a qualitative approach because most people relate to it better.
… Quantitative estimates run either the real or perceived risk of being twisted to support whatever conclusion is desired.
….. A qualitative argument like [brief hypothetical story] will convince far more people than any mathematical reasoning.
Cerf concludes
We would do well to ask ourselves how we might accomplish risk reduction regardless of the quantified risk. If sufficient people desire risk reduction, regardless of their agreement or disagreement as to its quantification, we might succeed in making the world a safer place.
In this example I am inclined to agree with Cerf: the numerics in Hellman’s argument seem quite implausible.
To estimate probabilities for levels of greenhouse gas emissions through the 21st century?
The periodic IPCC Assessment Reports attempt a definitive and thorough account of “knowledge on climate change, its causes, potential impacts and response options”. Its analysis of possible futures relies on scenarios. This 2021 article by Roger Pielke Jr. and Justin Ritchie, though focussed on a different matter (that old scenarios continue to be used although inconsistent with present data), contains a useful history for our purposes, which I will summarize as follows.
A baseline scenario is the perception of the most likely future in the absence of actions taken to alter that future. The IPCC originally adopted one baseline and three policy scenarios, each reflecting a different mix of future climate policies. But should they assert probabilities for these alternatives? One yes argument was “policy analysts needed probability estimates to assess the seriousness of the implied impacts; otherwise they would be left to work out the implicit probability assignments for themselves.” One no argument was that assessing likelihoods of scenarios a century into the future was
fundamentally impossible and they should not do it, lest it mislead their users about the foreseeability of the future. A 2000 report finessed this issue. The report emphasized four scenarios [called RCPs below], spanning a wide range of outcomes, so that readers would not be tempted to interpret a middle scenario as representing the most likely baseline future. Its consensus was that the future is inherently unpredictable and so views will differ as to which of the representative scenarios could be more or less likely. Therefore, selecting a single “best guess” scenario was neither desirable nor possible.
Continuing to quote Pielke-Ritchie:
And yet in 2005 the IPCC ignored its own guidance. It associated the RCP scenarios with not just plausibility but also likelihoods when it labeled the scenario leading to the greatest amount of climate change, called RCP8.5, as the single business-as-usual scenario of the set. In so doing, the IPCC identified RCP8.5 as the most likely future in the absence of further policy intervention, which gave it special status among not only the RCPs but among the hundreds of baseline scenarios of the broader IPCC scenario database.
So this is one context in which it has been hard to decide firmly whether to go with scenarios or likelihoods, which leads to the next topic.
To estimate probabilities or to create scenarios?
In an article subtitled The Right Way to Think About the Future, J. Peter Scoblic and Philip E. Tetlock write
[One should] make decisions based not on simplistic extrapolations of the past but on smart estimates of the future. It involves reconciling two approaches often seen to be at philosophical loggerheads: scenario planning and probabilistic forecasting. Each approach has a fundamentally different assumption about the future. Scenario planners maintain that there are so many possible futures that one can imagine them only in terms of plausibility, not probability. By contrast, forecasters believe it is possible to calculate the odds of possible outcomes, thereby transforming amorphous uncertainty into quantifiable risk. Because each method has its strengths, the optimal approach is to combine them.
They then explain
[In geopolitical forecasting tournaments], volunteers provide probabilistic answers to sharply defined questions, such as “Will the euro fall below $1.20 in the next year?” or “Will the president of Tunisia flee to exile in the next six months?” By measuring the difference between estimates and the actual occurrence of events, one can calculate a score showing how “well-calibrated” the expectations of any given forecaster were with reality. By analyzing these data, Tetlock discovered that the key to more accurate forecasting was to take people who were naturally numerate and open-minded, train them to think probabilistically and avoid common biases, and then group them so they could leverage the “wisdom of the crowd.”
So one can combine these approaches as follows. They write
Instead of evaluating the likelihood of a long-term scenario as a whole,
question clusters allow analysts to break down potential futures into a
series of clear and forecastable signposts that are observable in the short run.
…..
Consider the scenarios RAND produced as part of its analysis of China’s grand strategy. The four scenarios envisioned for 2050 — “Triumphant China,” “Ascendant China,” “Stagnant China,” and “Imploding China” — can be roughly placed on a classic two-by-two matrix, with the strength of China’s political leadership on one axis and the strength of China’s economy on the other. A cluster of questions that would give a heads-up that history is on a “Triumphant China” trajectory might include “On December 31, 2020, will China exercise over Itu Aba (or Taiping Island) in the South China Sea (which is currently under the de facto control of Taiwan)?” “Will China’s GDP growth in 2023 exceed ten percent?” and “Among African audiences, when will the China Global Television Network have a higher weekly viewership than Voice of America?”
To estimate probabilities for medium term global economic risks?
In what contexts can we look at past estimates and see how well they turned out? Perhaps the most interesting readily accessible data comes from the annual (since 2006) Global Risks Report (GRR) published as background material for the annual World Economic Forum. Here is my own discussion of this material. The centerpiece of each Report is a graphic showing the perceived likelihood and economic effect of each of 36 risks.
So, how well have these estimates worked out? For instance, how did they do at anticipating COVID-19? In fact, pandemic risk has been featured in GRR every year in the top left quadrant (low likelihood, large impact). As many people have noted in retrospect, experts had long predicted that such a pandemic would happen sometime, while acknowledging that the chance in any given year may be small.
Initially the graphics in the Reports had numerical likelihoods and numerical dollar impacts. Over the years, the axes labels have been changed to be more qualitative: for the 2020 graphic, the experts were asked to assess likelihood and impact on a rather ill-defined scale of 1–7, and the averages are plotted. This, combined with rather vague specification of the events under consideration, complicates attempts to assess accuracy.
To estimate the chance of a major earthquake on the Hayward fault?
We have good data on the frequency and severity of earthquakes in California over the last 100 years, and so we can make plausible probabilistic forecasts of the overall California frequency and severity of future quakes over (say) the next 30 years. But can we make a believable estimate for a particular fault line, say the Hayward fault, which happens to run through a highly populated area, including U.C.Berkeley campus? This probability has most recently been estimated by USGS scientists at about 1% per year.
This 2003 article by my former Berkeley colleagues David Freedman and Philip Stark provides a technical critique of the methodology (of a previous estimate but equally applicable to subsequent ones). In particular, they argue convincingly that, because models are complex and unverifiable, the asserted error bars are totally unreliable. They conclude
Probabilities are a distraction. Instead of making forecasts, the USGS could help to improve building codes and to plan the government’s response to the next large earthquake. Bay Area residents should take reasonable precautions, including bracing and bolting their homes as well as securing water heaters, bookcases, and other heavy objects. They should keep first aid supplies, water, and food on hand. They should largely ignore the USGS probability forecast.
Given that the quake seems likely within a human lifetime, this advice for the Hayward fault is very reasonable. But what about the Cascadia subduction zone, for which a major earthquake has implicitly been regarded as less likely but potentially stronger? Does not the extent to which such measures should be advised or mandated surely depend on some assessment of likelihood?
Footnotes
- This is part of my ongoing Probability and the Real World project.
- Pedantic readers might observe that, to be useful, an activity must be practical, so saying practical and useful is a redundancy because it is logically equivalent to useful. However it is often convenient to consider practical first, if only to avoid needing value judgments for useful:
would money spent on a Mars colony be better used on Earth? - For these future Big Events, useful means useful as a factor in making decisions. In the broader realm there are several other reasons why people consider probabilities.
- To a mathematician, it is noteworthy that the logical structure of any serious discussion is somewhat analogous to that of computational complexity. If one wishes to claim that it is possible to roughly estimate a numerical probability, then one should seek to demonstrate that claim by actually outlining some way to do the estimate. But if one wishes to claim that it is not possible, then what argument can one give? The fact that you and I don’t know how to do an estimate is hardly convincing. Nor is blithely saying that the future is unpredictable, because in some contexts one can indeed give accurate probability estimates of unpredictable matters. The fascinating aspect of computational complexity theory is that one can give rigorous arguments that certain computational problems are indeed difficult in a certain precise sense. Is there any analogous convincing argument that in certain (real human world) contexts it is in principle impossible to assess probabilities?
