The Met Office assessment of forecasts is based on the ones that are
broadcast every day at 5.55 pm on BBC Radio 4. Copies of the forecast are
sent to weather stations in the various regions, where the parts that are
relevant to the area are assessed in terms of temperature, rainfall, wind
and so on. The overall figure of accuracy quoted is an average of the
individual scores. The Met Office could not provide me with any reports or
articles about this process. There is, it seems, no published account of the
assessment: how, then, should we judge the claim of 85 per cent accuracy?
Advertisements in The Garden (December 1992 and January 1993) offer ’80 per
cent proven accuracy’ in 12-hour weather forecasts from a simple barometer
costing £8.99 – and that claim is not substantiated, either.

A clock that was 85 per cent accurate could be spot-on all week and not work
at all on Sundays – or it could gain three-and-a-half hours every single
day. A racing tipster who was 85 per cent accurate could give six winners
out of seven; on the other hand, all his selections could run at 85 per cent
of the winner’s speed, and lose by 100 lengths every time. If you predict
that a randomly chosen adult male, provided they are not Pygmies or
Bushmen, will be 2 metres tall, your chance of being right is well over 85
per cent – if you measure height to the nearest metre. Clearly, 85 per cent
accuracy may be good or useless, easy to achieve or impossible: it all
depends on how it is measured. Weather forecasts that are 85 per cent
accurate could, like the clock, be absolutely right for six days and totally
wrong on the seventh; they could be more-or-less right nearly always; or
they could have one significant feature completely wrong every time.

So what then do ‘right’ and ‘wrong’ mean in the context of a weather
forecast? Is a forecast right if ‘strong-to-gale-force winds’ blow your roof
off? How do you rate a forecast if the rain that’s promised for lunch time
is sheeting down when you first wake up? Is there always a ‘risk of frost’
in winter? Does a forecast of ‘scattered showers’ get the benefit of the
doubt if there’s a bit of cumulus cloud about?

To get an objective assessment out of such subjective material, you must
apply exactly the same tests to an alternative forecast, and compare the
results. The obvious alternative forecast would consist of the long-term
mean values of temperature, rainfall and so on, for each weather station,
for the date to which the forecast applies. If the bundles of mean values
were only, say, 42 per cent accurate on average, the relative score for the
forecasts would be 85/42 = 2.0 – objective measure of their value. Since the
weather stations already keep the necessary records, there would be little
extra analysis involved. And if the mean values proved to be almost as good
as the forecasts, the Met Office would have a choice: they could either use
the mean values as their forecasts in future, or improve the assessment. The
first option would enable them to sell their computer – they could probably
work out the mean values on a Nintendo Gameboy – but they really ought to do
the assessment properly.

A proper assessment could differ from the current one in two fundamental
ways. First, it would be based on measurements rather than statements.
Secondly, it would penalise errors in timing, which are surely the most
common failing of next-day forecasts. For such an assessment, the forecasts
would have to be expressed in much more detailed terms than those used in
the broadcasts. Quantitative forecasts would be needed for each station, for
each of (say) five periods – perhaps four 3-hour daytime periods (9 am to 12
noon, 12 to 3 pm, 3 to 6 pm, and 6 to 9 pm), and the 12-hour overnight
period 9 pm to 9 am. The major weather features – rainfall (in millimetres),
sunshine (in hours or, preferably, minutes), temperature (in degreeC, with
daytime maximum and overnight minimum), and mean wind speed (in kilometres
per hour) – would certainly be included. Most of the other features that
could be included – relative humidity, minimum visibility, wind direction
and so on – can be expressed in measurable units, but ‘special effects’ such
as ground frost and thunderstorms could be forecast and recorded on a simple
risk-cum-severity scale (none, possible/light, likely/moderate, or
certain/heavy). Then, for each feature, in each period, the forecast value
would be compared with the measured value, and some sort of average error
would be derived.

It would not be sensible to use the simple average of errors in such
disparate quantities as hours of sunshine and wind speed, so each error
would be expressed as a percentage of the possible range of values for that
quantity. The various possible ranges – which would vary between stations,
between seasons, and between periods – can be derived from records. In
London, for example, the overnight minimum temperature in January has a
range of about 20 degreeC (from -10 to +10), so a forecast overnight minimum
of zero, when the actual value was -3, would have a percentage error of 3/20
times 100, or +15 per cent. The simple mean of these percentage errors would
balance the over-prediction of one feature with the under-prediction of
another; the ‘root mean square’ percentage error would measure the average
size of the errors, regardless of whether each forecast value was too high
or too low.

These percentage errors may not be a good measure of the quality of the
forecasts. The distribution of values of some features is highly skewed –
that is, nearly all the values lie at one end of the range, with a few
extreme values at the other. Rainfall is like this: despite the popular
impression, it is hardly ever raining at any particular time (even in
Manchester) – but you could get 50 millimetres in an hour.

It is the same with visibility, which is effectively unlimited most of the
time, but can fall to 50 metres or less. For those features, a lazy
forecaster could nearly always get it right by using the mean value (or the
highest/lowest possible value) as the predicted value for any period; there
would be little incentive to look out for the extreme values that are
generally more important. Even for features with values that are more
symmetrically distributed, like sunshine or temperature, the mean value
would still be quite a good predictor if extreme values were not too common.
This effect could be allowed for by comparing the errors of the forecast
values with the errors of the local long-term average values, using the
root-mean-square percentage error.

I hoped that the Met Office Library database of publications would include
discussions of how to assess the forecasts, but it does not. Nearly all the
reports and papers in it are concerned with ways to improve the ‘model’ –
the computer-based simulation model of the world’s atmospheric pressure
pattern – from which the forecasts are derived. That is the trouble with
modelling: those involved in it tend to become more interested in the
process than its purpose. In doing so, they verify the First Law of
Modelling: the stimulation of simulation is greater than the pleasurement of
measurement, but it makes you go blind.

Neil Duncan worked for the Transport Research Laboratory at Crawthorne,
Berkshire, for 35 years and is now retired.

The first problem is to decide which factors are relevant. You may try
to throw in everything but the kitchen sink, and prune the model later,
but you are stuck with the factors for which values are readily available.
It would be silly, for example, to model the speed of traffic in terms of
the proportion of drivers with blue eyes, if it would take an expensive
survey to identify that proportion. And you have to allow for the values
of factors to change: in economic statistics, for example, a deficit is
sometimes ‘revised’ into a surplus.

When it comes to deciding just how each factor affects the output, there
are four things to consider – shape, thresholds, interactions, and lag.
Shape refers to the basic mathematical form of the relationship; thresholds
are discontinuities in relationships; interactions are present when the
effect of a factor depends on the values of one or more of the other factors;
and lag means that the output is affected not by the current value of a
factor but by an earlier or later value.

The shape of a relationship may vary from a simple straight line to
something that needs half a page of algebra to describe it. Sometimes,
a nonlinear effect can be represented by using a simple linear expression
and transforming the factor – by taking its reciprocal, for example – but
this does not always work. Deciding the shape of the relationship is a craft
not a science.

Thresholds are levels where the effect of a factor changes. They could
be represented by a transformed factor: for a single threshold, for example,
all values of the factor below the threshold level could be replaced by
zero. A special case of the single threshold arises in ‘catastrophe theory’,
where a system can absorb all the effects of a particular factor until a
certain level of that factor is reached – when the system collapses. Thresholds
appear also in ‘chaotic systems’, where a small perturbation of a stable
or steadily-evolving condition can be so greatly magnified by a feedback
mechanism that the whole process suddenly goes berserk. Needless to say,
systems like this are hard to model reliably.

A different sort of threshold effect may occur in a system in which
the output is subject to several independent constraints. In traffic, for
example, a speed limit might be irrelevant on a very busy road, but the
level of traffic might be irrelevant where a series of bends is the main
constraint on speeds. This sort of structure is easy to build into a simulation
model, but harder to observe in real life. Going back to our traffic example,
different drivers are likely to respond differently to the various constraints,
blurring their overall effects.

Interactions between factors change their effects. For example, an increase
in traffic flow of 500 vehicles per hour may reduce the average speed of
cars by 5 kilometres per hour on a flat road but by 20 kilometres per hour
on a hill – even though the hill itself would have no noticeable effect
on car speeds. Such an interaction can easily be represented in a simulation
model or an analytical model. But in observations for an empirical model,
interactive effects could be masked or diluted – just as thresholds may
be blurred by variations between drivers.

So are empirical models inferior? Not necessarily. Simulation models
are always more detailed, but their greater detail may be inappropriate.
People adapt to circumstances, and therefore any system involving people
is likely to be fairly insensitive to changes in those circumstances. In
this case, it is not just that the detail is hard to observe: it may not
actually be there.

Lag in the effect of a factor may be more of a problem than all the
rest put together. For example, the speeds of vehicles at any point are
likely to be affected by the road layout on the approach (positive lag),
by conditions at the point (zero lag), and by the drivers’ assessments of
what lies ahead (negative lag). In some fields, lag may be crucial. It has
recently been suggested, for example, that conditions in the womb can affect
that child’s children, 25 years later; and, in economics, the seeds of a
boom or a depression can be seen (or overlooked) in various indicators for
years beforehand.

It is hard enough to detect simple lagged effects, when you do not know
how far to look back (or forwards). To make matters worse, the lag itself
may be subject to interactive or threshold effects. In a really complicated
system, the possibilities are virtually unlimited. Animal fats may be bad
for you but only if you eat more than 100 grams of saturated fat every day,
and only after 20 years if you take no exercise or 40 years if you drink
plenty of red wine, and not at all if you are Italian. Unless, of course,
your mother smoked heavily while she was carrying you, in which case . .
.

Real-life systems are complicated, and it is not surprising that future
events are so hard to predict. Also, making predictions is more difficult
in matters that involve people than in those that do not: if molecules had
drivers, chemistry would still be in its infancy.

The human element probably explains why economics and nutrition are
viewed as pseudosciences, but ‘chaos’ seems to be the most plausible excuse
for meteorologists. We should not blame those who work in these fields for
their inability to predict; their failing is their reluctance to admit it.

Neil Duncan worked for the Transport Research Laboratory at Crawthorne,
Berkshire, for 35 years and is now retired.