A campaign launched by the Royal Statistical Society has proposed 12 “rules of thumb for journalists” in order to encourage a better understanding of numbers in news.
Getstats is also calling for numeracy and statistics to be taught in journalism schools.
More details and a 12 point summary is at this link.
The full 12 rules of “number hygiene” for journalists are below:
1. You come across a number in a story or press release. Buyer beware. Before making it your own, ask who cooked it up; what are their credentials; are they selling something. What other evidence do we have (what numbers are they not showing us?); why this number, now? If the number comes from a study or research, has anyone reputable said it is any good?
2. Sniff around. Do the numbers refer to a whole group of people or things or a sample of them? If it’s a sample, are the people being questioned or the things being referred to a fair representation of the wider group? Say a company is claiming something applies to the population at large. If it is basing the story on a sample, such as a panel of internet users, the company goes back to time and again then beware: the panel may not be representative.
3. More probing. What was the sample asked? The wording of a question can hugely influence the answer you get. People’s understanding of what it means to ‘be employed’ or the nature of ‘violent crime’ may differ. What the public understands may not match the survey researcher’s idea. In government surveys bigamy was till recently classed as a violent crime. Might researchers’ choice of words have led people into a particular response?
4. One number is often used to sum up the group being measured, the average. But different averages measure different things. The mean is extremely sensitive to highs and lows: the very fact of Bill Gates coming to live in the UK would push up mean wealth. The median tells us, for example, the income of an average person – half the population get less, half more. Comparing earnings, the mode tells us the salary most people earn.
5. There is a lot of uncertainty about. We need to be sure the number on offer is a result and not just due to chance. With a sample, check the margin of error, the plus or minus 3 per cent figure, usually stated by reputable polling companies. A poll saying 52 per cent of people are in favour of something is not definitively saying half are in favour: it could be 49 per cent. Beware league tables, except in sports reports. Chelsea is higher than Arsenal for a simple and genuine reason: the side has collected more points. With hospitals or schools, a single score is a never likely valid basis for comparison (a teaching hospital may appear to have a worse score, but only because sicker patients are referred to it). Comparisons between universities or police forces are unreliable if the scores fall within margins of error. Midshires scores 650 on the ranking and Wessex 669: they could be performing at the same level or their respective positions reversed.
6. The numbers you are given show a big increase or sharp decrease. Yet a single change does not mean a trend. Blips happen often. Blips go away, so we have to ask whether the change in the numbers is just a recovery or return to normal after a one-off rise or fall (what statisticians refer to as ‘regression to the mean’). The numbers may come from a survey, like (say) ONS figures for household spending or migration. Is the change bigger than the margin of error?
7. Unless researchers carried out a controlled experiment (such as a trial of a new drug, based on a randomly chosen group, some of whom don’t know they are getting a placebo), it’s very difficult confidently to state that a causes b. Instead, the numbers may show an association (a correlation) between two things, say obesity and cancer. Beware spurious connections, which may be explained by a third or background factor. If use of mobile phones by children is associated with later behavioural disorders, the connexion could be the parents, and the way their behaviour affects both things. If the numbers suggest an association, the important thing is to assess its plausibility, on the back of other evidence. Finding a link can stimulate further study, but can’t itself be the basis for some new government policy. Recommendations for changing daily behaviour such as eating should not be based on speculative associations between particular food and medical conditions.
8. A key question for any number is ‘out of how many?’ Some events are rare — such as the death of a child. That’s why they are news, but that’s also why they deserve being put in context. Noting scarcity value is the way to reporting the significance of an event. An event’s meaning for an individual or family has to be distinguished from its public importance.
9. Billions and millionths are too big and too small to grasp. We take figures in if they are humanized. One way is comparing with, say, the whole UK; another is to plot the effect on an individual. Colourful comparisons can make risk intelligible: the risk of dying being operated on under a general anaesthetic is on average the same as the risk being killed while travelling 60 miles on a motorbike.
10. Good reporting gives a balanced view of the size of the numbers being reported. Better to focus on the most likely number rather than the most extreme, for example in stories about the effects of a flu pandemic. ‘Could be as high as’ points to an extreme; better to say ‘unlikely to be greater than’. Numbers may be misperceived so try to eliminate bias.
11. Risk is risky. ‘Eating bacon daily increases an individual’s lifetime risk of bowel cancer by 20 per cent.’ Another way of saying that is: out of 100 people eating a bacon sandwich every day one extra person will get bowel cancer. Using the first without noting the second tells a story that is both alarmist and inaccurate. If the information is available, express changes in risk in terms of the risks experienced by 100 or 100,000 people.
12. The switch from print to digital brings opportunities to present numbers more dynamically and imaginatively, for example in scatter plots. Graphics can show a trend. Stacked icons in graphs can show effects on 100 people. But the same rules of thumb apply whatever the medium: is the graphic clear; does it tell the story that is in the text.
