The four tests a poll must pass to be taken seriously

The four tests a poll must pass to be taken seriously

We all like to know what other people think. Politicians want to find out how popular they are. Non-governmental organisations want to see the impact of their campaigning. Companies want to test new product ideas. And the rest of us want to know the views of our fellow citizens.

Opinion polls are a popular way to do this. Not a week goes by without a new one being published in Thailand. But can they be trusted? This matters: not only do we want accurate information, but polls can influence opinion as well as reflect it. We are social animals, and tend to support what we think are popular choices. How can we tell whether to take a poll seriously? A good poll must pass four tests.

The first test is question design. A good poll asks a clear question in a neutral way. Small variations in wording can produce very different responses. "Do you agree that Scotland should be independent?" pushes the respondent to agree. It is much more likely to produce a "yes" than simply ''should Scotland be independent?" Not surprisingly, the Scottish National Party wanted the first question on the 2014 Scottish independence referendum, but the authorities insisted on the second.

The second test is response bias. People might not give an honest answer even if the question is clear. In some countries, if a stranger asks you whether you support the leader, you will always say yes. Even where people do not fear persecution they sometimes conceal their true beliefs. In the last US election, only one out of sixty-one tracking polls predicted Donald Trump's victory. It differed from all the others in allowing respondents to answer anonymously. It seems some voters were reluctant to admit their support for Mr Trump when they had to tell a pollster face-to-face.

The third test is random sampling. Polling involves asking a small group for their views and extrapolating the results to the whole population. For this to work, the sample group must be randomly chosen. Everyone must have an equal chance of being in it. This can be much harder than it sounds. An online poll will under-represent the old, who are less likely to use the internet. Asking people in shopping malls will under-represent those who stay at home or shop elsewhere. A 1936 telephone poll famously predicted that Franklin Roosevelt would lose the US presidential election. He won by a landslide: only the rich had phones in 1936. Choosing a genuinely random sample of respondents is usually the hardest thing to get right.

The final test is sample size. This is easier, but still important. As well as being representative, a polling sample should be big enough to predict the wider population's views with a reasonable degree of accuracy and confidence. The usual standard is 95%. So "52% of people approve of the government's performance, with a margin of error of 3%" means there is a 95% chance that the government's approval rating lies somewhere between 49% and 55%. A poll with the same margin of error in Thailand (assuming an adult population of 50 million) requires a random sample of 1,067 people. A smaller sample will mean a bigger margin of error. That's fine, as long as it is made clear.

Anyone can use these four tests to see whether a poll is worth listening to. But they can only do so if it is properly reported. A report that says "a recent poll showed 43% of people are better off this year" is not enough. Better to say "43% reported that 'My standard of living improved last year' (face-to-face interview of 850 people in 64 provinces, 3.4% margin of error)". This gives us the information to decide whether to take this poll seriously. Ideally, a poll should also include when, and by whom, it was conducted.

These four rules are essential for measuring public opinion well. Conversely, a poll that breaks even one of them will certainly be unreliable. With talk of an election in Thailand next year, interest in public attitudes will grow. By conducting and reporting opinion polls accurately, pollsters and journalists alike can help us better understand what people think.


Nigel Gould-Davies teaches at Mahidol University International College and is an Associate Fellow of the Royal Institute of International Affairs. He is completing a book on political risk.

Nigel Gould-Davies

Mahidol University International College teacher

Nigel Gould-Davies holds a PhD in Political Science from Harvard University. He teaches at Mahidol University International College.

Do you like the content of this article?
COMMENT (2)