Contributing Editor David Hand writes:

Reading my co-columnist Xiao-Li Meng’s column The XL-Files in the January/February 2015 issue made me realise I hadn’t written a column about The Improbability Principle, my book which appeared early in 2014. Xiao-Li wrote, “Winning the lottery or being struck by lightning are both events with extremely small probabilities, events any statistically sound mind should not expect.” But the subtitle of my book (in the US) is Why Coincidences, Miracles, and Rare Events Happen Every Day, and the examples I give include both being struck by lightning and winning the lottery. So what’s going on?

One of the characteristics of probability is that it can produce results which appear counterintuitive without careful thought (and formal mathematics). This has led to a collection of results termed paradoxes or fallacies, though they’re not really paradoxes, and the fallacies lie not in probabilistic reasoning but in uninformed interpretations of probability. It has also led to a mini-genre of popular books on the subject. Think of Warren Weaver’s Lady Luck, Brian Everitt’s Chance Rules, Leonard Mlodinow’s The Drunkard’s Walk, John Haigh’s Taking Chances, Jeffrey Rosenthal’s Struck by Lightning, and Joseph Mazur’s What’s Luck Got To Do With It?. (My apologies if I have not listed your favourite – there are a lot of them.)

The particular aspect that I addressed in my book was that of highly improbable events, but events which nevertheless happened. These are things like people winning the lottery several times, or the same lottery numbers coming up in different weeks. Or people being struck by lightning multiple times (despite the old adage that “lightning never strikes twice”). Or financial crashes occurring when calculations show they shouldn’t be expected in 20 billion years. Or more mundane things like people bumping into their neighbours when travelling overseas, or meeting someone with the same name (such as the other David Hand I once met at a conference). And so on.

Surely, intuition tells us, these things are incredibly unlikely, so we shouldn’t expect to see them. Indeed, some very eminent philosophers and mathematicians have pointed this out. Jean D’Alembert questioned whether it’s possible to observe a very long sequence of heads in a random series of events in which heads and tails are equally likely. Antoine-Augustin Cournot coined the principle that an event with very small probability will not happen. And the eminent mathematician Émile Borel coined what is now called Borel’s law: events with a sufficiently small probability never occur. If an event is so improbable that we wouldn’t expect to see it in the entire history of the universe, then it is only rational – in practical terms – to regard it as impossible.

But the painful truth is that, as my book illustrates with many true stories, events which appear to have extraordinarily low probability do keep happening. As the improbability principle has it: highly improbable events are commonplace.

In my book, I have tried to create a framework for the explanations. I’ve done this through the vehicle of five “laws”, which will be recognised by professional statisticians.

The law of inevitability says that one of a complete list of outcomes must occur. It’s a sort of converse of the law of total probability.

The law of truly large numbers (which already appears in the statistical literature) says that, given enough opportunities, the most extraordinarily unlikely things can occur. Any statistician who has grappled with multiple testing problems will appreciate this only too well.

One way of expressing the law of selection is that you can make things as probable as you want if you choose after the event. If this sounds banal, it can creep up in unexpected ways—and selection bias and the related phenomenon of regression to the mean are familiar concepts to statisticians.

The law of near enough is concerned with how one defines the reference class. Make it broad enough and you can make the estimated probability as large as you like.

The law of the probability lever is to do with choosing models. With poor assumptions even highly likely events can seem very improbable. Classic examples of this concern the tails of distributions and independence assumptions.

Those five laws are not the end of the improbability principle. Sitting on top of them is the human mind, and the difficulty it often has in grasping the subtleties of probability: the base rate fallacy, the prosecutor’s fallacy, the availability heuristic, … . The list is almost endless.
Put all these things together, and it is hardly surprising that highly improbable events happen every day.

Just because it’s highly unlikely… doesn’t mean you should stay outside in a thunderstorm!
With thanks to xkcd.com for the joke.