THE scene is Jerez Grand Prix circuit, the last race of the 1997 Formula One
season. Michael Schumacher is one championship point ahead of arch-rival Jacques
Villeneuve鈥攖hanks in part to brilliant tactical driving by his Ferrari
team-mate Eddie Irvine in the previous race. Villeneuve鈥檚 Williams team-mate
Heinz-Harold Frentzen may well play the same game this time, so qualifying in
pole position is even more critical than usual . . .
So what happens? Villeneuve, Schumacher and Frentzen all lap in exactly 1
minute 21.072 seconds. The astonished commentators hailed it as an amazing
coincidence. Well, 鈥渃oincidence鈥� it surely was鈥攖he lap times coincided.
But was it truly amazing?
Questions like this do not just apply to sport. They turn up everywhere,
trivial and important. Just how surprising was it to run into Great Aunt Lottie
from Sweden in that San Francisco strip bar? Is it really unexpected that three
different people at the Christmas party are wearing the same dress? And in
science, how significant is a leukaemia cluster? Does a strong correlation
between lung cancer and having a smoker in the family really prove that passive
smoking is dangerous?
Advertisement
One of us, Jack Cohen, is a reproductive biologist. Last year he was asked to
explain two very curious statistics. While in Israel, he was told that 84 per
cent of the children of Israeli fighter pilots are girls. 鈥淲hat is it about the
life of a fighter pilot,鈥� he was asked, 鈥渢hat produces such a predominance of
daughters?鈥� The second statistic arose in connection with in vitro
fertilisation. Nowadays, IVF clinics use ultrasound to monitor ovulation, and so
can determine whether an egg鈥攁nd the resulting baby鈥攃omes from the
left or right ovary. One clinic discovered that most of the girl babies came
from the left ovary, and most of the boys from the right. A breakthrough in
choosing the sex of your children? Or just a statistical freak?
It鈥檚 not easy to decide. Gut feelings are worse than useless, because human
intuition is poor when it comes to random events. Many people believe that
lottery numbers that have so far been neglected must be more likely to come up
in future. In justification they plead the 鈥渓aw of averages鈥濃�攅verything
ought to even out in the long run. But the truth is different, and not at all
intuitive. Yes, in the long run, each lottery number is indeed just as likely to
turn up as any other. But the lottery machine has no memory. The proportions do
even out, in the long run, but you can鈥檛 say in advance how long that run will
be. In fact, if you choose any specific number of attempts, however large, then
the best prediction is that any initial imbalance will remain unchanged.
Our intuition goes even further astray when we think about coincidences. You
go to the local swimming pool, and the guy behind the counter pulls a key at
random from a drawer full of keys. You arrive in the changing room and are
relieved to find that very few lockers are in use . . . and then it turns out
that three people have been given lockers next to yours, and it鈥檚 all 鈥淪orry!鈥�
as you bang locker doors together. Or you are in Hawaii, for the only time in
your life . . . and you bump into the Hungarian you worked with at Harvard. Or
you鈥檙e on honeymoon camping in a remote part of Ireland . . . and you and your
new wife meet your department head and his new wife, walking the other way along
an otherwise deserted beach. All of which happened to Jack.
These coincidences all seem striking because we expect random events to be
evenly distributed, so statistical clumps surprise us. We think that a 鈥渢ypical鈥�
draw in Britain鈥檚 national lottery is something like 5, 14, 27, 36, 39, 45, but
that 1, 2, 3, 19, 20, 21 is far less likely. In fact, these two sets of numbers
have the same probability: 1 in 13 983 816. Sequences of six random numbers are
actually more likely to be clumpy than not.
Lapping it up
How do we know this? Probability theorists tackle such questions using
鈥渟ample spaces鈥�. A sample space contains not just the event that concerns us,
but all possible alternatives. If we are rolling a die, for instance, then the
sample space is 1, 2, 3, 4, 5, 6. For the lottery, the sample space is the set
of all sequences of six different numbers between 1 and 49. A numerical value is
assigned to each event in the sample space, called its 鈥減robability鈥�, and this
corresponds to how likely that event is to happen. For fair dice each value is
equally likely, with a probability of 1 in 6. Ditto for the lottery, but now
with a probability of 1 in 13 983 816.
Thinking about the size of the sample space is a good way to assess how
amazing an apparent coincidence really is. Take the Formula One lap times. Top
drivers all lap at roughly the same speed, so it鈥檚 reasonable to assume that the
three fastest times would fall inside the same tenth-of-a-second period. At
intervals of a thousandth of a second, there are 100 possible lap times for each
to choose from: this list determines the sample space. Assume for simplicity
that each time in that range is equally likely. Then there is a 1 in 100 chance
that the second driver laps in the same time as the first, and a 1 in 100 chance
that the third laps in the same time as the other two鈥攚hich leads to an
estimate of 1 in 10 000 as the probability of the coincidence. Low enough to be
striking, but not so low that we ought to feel truly amazed. It鈥檚 roughly as
likely as a hole-in-one in golf
Estimates like this help to explain astounding coincidences reported in
newspapers, such as bridge players getting a 鈥減erfect hand鈥�, each having the
thirteen cards in a suit. In any one game, the chances of this happening are
staggeringly small. But the number of games of bridge played every week world
wide is huge. So huge that every few weeks the actual events explore the entire
possible sample space. In other words, you should expect perfect hands to turn
up somewhere鈥攁s often as their small but non-zero probability
predicts.
The use of sample spaces, however, is not entirely straightforward.
Statisticians tend to work with the 鈥渙bvious鈥� sample space. For that question
about Israeli fighter pilots, for instance, they would naturally take the sample
space to be all children of Israeli fighter pilots. But that might well be the
wrong choice. Why? We often tend to underestimate the size of the sample
space鈥攁nd so assume that coincidences are more surprising than they really
are. It is all down to a crucial factor which we call 鈥渟elective
reporting鈥濃�� which tends to be ignored in most conventional statistics.
That perfect hand at bridge, for instance, is far more likely to make it to
the local or even national press than an imperfect one. How often do you see the
headline 鈥淣ottingham Bridge Players Get Entirely Ordinary Hand鈥�, for instance?
The human brain just can鈥檛 resist looking for patterns, and seizes on certain
events that it considers significant, whether or not they really are. And in so
doing, it ignores all the 鈥渘eighbouring鈥� events that would help it judge how
likely or unlikely the perceived coincidence actually is.
Selective reporting affects the significance of those Formula One times. If
it hadn鈥檛 been them, maybe the tennis scores in the US Open would have contained
some unusual pattern, or the snooker breaks in that week鈥檚 tournament, or the
golf . . . Any one of those would have been reported, too. But none of the
failed coincidences, the ones that didn鈥檛 quite happen, would have hit the
headlines. If you include just ten major sporting events in the list of
would-be鈥檚 that weren鈥檛, that 1 in 10 000 chance comes down to only 1 in 1000.
It鈥檚 like tossing a coin and turning up heads 10 times in a row.
So going back to the Israeli fighter pilots鈥攊s it just random chance or
is something else happening? To answer this, conventional statistics would set
up the obvious sample space (children of fighter pilots), assign probabilities
to boy and girl children, and calculate the chance of getting 84 per cent girls
in a purely random trial. But this analysis ignores selective reporting. Why did
anyone look at the sexes of Israeli fighter pilots鈥� children in the first place?
Presumably because a clump has already caught their attention. If it had been
the heights of the children of Israeli aircraft manufacturers, or the musical
abilities of the wives of Israeli air traffic controllers that showed up as a
clump, their clump-seeking brains would have pounced on those strange
circumstances instead. The conventional statistical approach tacitly excludes
many other factors that didn鈥檛 clump鈥攊t ignores part of the sample
space.
The human brain filters vast quantities of data, seeking things that appear
unusual, and only then does it send out a conscious signal: will you look at
that! The wider you cast your pattern-seeking net, the more likely it is to
catch a clump. There鈥檚 nothing wrong with that. But if you want to know how
significant the clump is, you can鈥檛 include the data that drew your attention to
the clump in the first place. In a room with 20 people there will typically be
one鈥攖he tallest鈥攚hose height puts them in the top few per cent in
the country. But you can鈥檛 then remove that person from the room, re-measure
their height, and then deduce that it is surprising that you have found somebody
with such an unusual height. That鈥檚 what you chose them for.
Extraordinary powers
This is exactly the error made in early experiments on extrasensory
perception. Thousands of subjects were asked to guess cards from a special pack
of five symbols. After several weeks of selection, anyone whose success rate had
been above average was invited back and tested some more. At first, these 鈥済ood
guessers鈥� seemed to have extraordinary powers. But as time went on, their
success rate slowly dropped back towards the average, as if their powers were
鈥渞unning down鈥�. This happened because their initial high scores鈥攖he
clumping for which they were chosen鈥攚ere included in the running total. If
these fortuitous scores had been excluded from the second set of tests, then the
scoring rate would have dropped, immediately, to near average.
And so it is with the fighter pilots, and with the left/right ovaries. The
curious figures that drew researchers鈥� attention to these particular effects may
well have been the result of selective reporting鈥攐r, much the same,
selective attention. If so, then you can make a simple prediction: 鈥淔rom now on,
the figures will revert to fifty-fifty鈥�. If this prediction fails, and if the
results instead confirm the bias that revealed the clump, then the new data can
be considered significant. But the smart money is on the prediction
succeeding.
The same fallacy can arise in conventional experimental studies, say to find
out whether certain foodstuffs give you cancer. To save time, the usual way is
to look at many different foods at once鈥攆ibre, fat, red meat, vegetables
and so on鈥攁nd see how they all compare with cancer rates. So far so good.
But now, you pick out the biggest entry鈥攖he food that is unusually closely
correlated with cancer rates. Unless you鈥檙e careful, you now forget all the
other factors, and publish a paper saying that eating red meat significantly
increases your chances of cancer. However, you chose the most significant from
the hundred different foods you tried. Of course at least one comes out to be
significant. You鈥檇 be surprised if it didn鈥檛, on statistical grounds alone, even
if all the foods were chosen at random.
Highly sensitive
A study last year by a team at London鈥檚 Institute of Child Health that
claimed to detect a 鈥済ene for sensitivity鈥� fell into this trap. The researchers
administered a questionnaire, correlated all of the answers with likely genetic
or cultural influences, and claimed that the sensitivity of human
beings鈥攈ow kind they are to others, how 鈥渇eminine鈥� their approach to the
world is鈥攊s genetically determined. The sensible prediction is that these
correlations will not be repeated in future studies. If that鈥檚 wrong, then the
alleged gene starts to look more interesting鈥攂ut not yet.
The alleged decline in the human sperm count may be another example of
selective reporting. Not by Niels Skakkebaek鈥檚 team at Copenhagen University
which published the first widely accepted evidence for a decline in 1997. The
selective reporting was done by researchers who had contrary evidence but didn鈥檛
publish it because they thought it must be wrong, by journal referees who
accepted papers that confirmed a decline more often than they accepted those
that didn鈥檛, and by the press鈥攚ho strung together a whole pile of
sex-related defects in various parts of the animal kingdom into a single
seamless story, unaware that each individual instance had an entirely reasonable
explanation that had nothing to do with falling sperm counts and often nothing
to do with sex. Sexual abnormalities in fish kept in water from sewage treatment
plants, for instance, are probably due to excess nitrites鈥攚hich all
fish-breeders know cause abnormalities of all kinds鈥攁nd not to
oestrogen-like compounds in the water, which would bolster the 鈥渟perm count鈥�
story.
The message here is that when you are estimating statistical significance,
you must tailor your choice of sample space to the experiment as it was actually
performed, not a selectively reported version. The safest way to do this is to
discard the data that led to your particular result, and repeat the experiment
to get new data. But even then you must not let the coincidence, the clump,
choose the sample space for you鈥攊f you do, you鈥檙e ignoring the surrounding
space of near-coincidences.
We decided to test this theory on a recent trip to Sweden. On the plane, Jack
predicted that a coincidence would happen at Stockholm airport. His reason:
selective reporting. If we looked hard enough, we鈥檇 be sure to find one. We got
to the bus stop outside the terminal, and no coincidences had occurred. But we
couldn鈥檛 find the right bus, so Jack went back to the enquiries desk. As he
waited, someone came up next to him鈥擲tefano, a mathematician who normally
lived in the office next door to Jack鈥檚 at Warwick University.
Prediction confirmed. But what we really wanted was to lay hands on a
near-coincidence鈥攐ne that hadn鈥檛 happened, but could have been selectively
reported if it had. For instance, if some other person we knew had shown up at
exactly the same time, but on the wrong day, or at the wrong airport, we鈥檇 never
have noticed. So for all we knew, Sweden could be overrun by acquaintances of
ours, making it virtually impossible for us not to bump into one of them at
almost any moment. Near-coincidences, by definition, are hard to observe. But we
happened to mention all of the above to Ian鈥檚 friend Ted, who visited soon after
we got back to England. 鈥淪tockholm?鈥� said Ted: 鈥淲hen?鈥� And we told him. 鈥淲hich
hotel?鈥� 鈥淭he Birger Jarl.鈥� 鈥淔unny, I was staying at the Birger Jarl one day
later than you!鈥� Had we travelled one day later then, we wouldn鈥檛 have met
Stefano鈥攂ut we would have met Ted. Selective reporting would have ensured
that we only told our friends about the one that actually happened.
Probability theory assesses how likely an event is in comparison to others
that could have happened, but somehow or another didn鈥檛. Our intuition for
probabilities is poor because the feature-detection system in our brains notices
only the things that happen. In the world around us, every event is unique.
Every meeting, every sex ratio, every bridge hand. 鈥淲hat, your telephone number
is nearly the same as your car registration? How amazing!鈥� But when you realise
that a typical citizen has several dozen significant numbers (address, postcode,
fax, mobile, credit cards . . .) a chance resemblance between some two of them
is surely mundane, rather than remarkable.
Wonderful rain
What we must not do, then, is to look back at past events and find
significance in the inevitable few that look odd. That is the way of the
pyramidologists and the tea-leaf readers. Every pattern of raindrops on the
pavement is unique. We鈥檙e not saying that if one such pattern happens to spell
your name, this is not to be wondered at鈥攂ut if your name had been written
on the pavement in Beijing during the Ming dynasty, at midnight, nobody would
have noticed. It鈥檚 no use looking at past history when assessing significance:
you need to look at all the other things that might have happened instead.
Every actual event is unique. Until you place that event in a category, you
can鈥檛 work out which background to view it against. Until you choose a
background, you can鈥檛 estimate the event鈥檚 probability. On the other hand, if
something that seems spooky to you really does turn out to have a small sample
space, that鈥檚 when you should be really amazed.