Polls Are Still As Accurate As They Were 75 Years Ago

Here’s an idea destined to be controversial: polls are as accurate as they’ve ever been and in some ways are more accurate. That may fly in the face of the recent epic polling disasters (see: the 2016 U.S. presidential election, 2015 British general election and the 2016 Brexit vote in the U.K.), but that’s the conclusion of a new study, reports Nicola Davis at The Guardian.

To assess whether polls have lost their mojo over time, researchers looked at 30,000 voter polls conducted for 351 general elections in 45 nations over a span of 75 years. According to a press release, what they found is that the accuracy of polls have not changed much over the decades and the problematic polls of recent years are in line with historic polling.

“We find that, contrary to conventional wisdom, the recent performance of polls has not been outside the ordinary,” authors Will Jennings of the University of Southampton and Christopher Wlezien of the University of Texas at Austin write in the study, published this week in the journal Nature Human Behaviour. “Ultimately, although the polling industry faces a range of substantial challenges, we find no evidence to support the claims of a crisis in the accuracy of polling.”

Davis reports that the researchers drilled down on the data, looking at 286 elections in which polling began at least 200 days before voting. In that scenario, the mean absolute error (the absolute value of the difference between the forecasted value and actual value) decreased from about 4 percentage points that far out to approximately 3 percentage points at 50 days to just around 2 percentage points on election eve.

When looking at polls for the last week of elections in 220 national elections in 32 countries stretching back 75 years, they found the error rate was pretty steady at 2 percent.

When looking at just the 11 nations that have conducted regular polling over the entire timespan, the research showed polling had improved slightly, with error rates dropping over time. The study also isolated recent polling, looking at polls and elections between 2015 and 2017 in 11 nations including the United States and U.K. That analysis shows there’s not apocalypse in polling—the polls lined up with historical accuracy rates.

“It is one of those bits of evidence that we would like more people to be aware of, because we are aware it is not getting worse and it is not a particularly new problem, but of course the public perception is often that it is,” Anthony Wells, director of data analytics firm YouGov, not involved in the study tells Davis. “ I think it is just because obviously people remember the recent errors far more prominently than the errors from long ago. There are famous errors of the postwar US – [the] ‘Dewey defeats Truman’ election, and the polls getting it wrong then, or [in the UK] the 1970 election or the 1992 election.”

This, of course, goes against the conventional wisdom. In fact, Adam Rogers at Wired reports the American Association for Public Opinion Research issued a 50-page post-mortem of the 2016 U.S. election and a report on 2015 polls in the U.K. said polling was getting things wrong because of low response rates and consumers abandoning landlines.

But the data says something different. Jennings tells Rogers that polling during the 2016 U.S. election wasn’t the cluster many people believe it to have been. “[T]he actual national opinion polls weren’t extraordinarily wrong. They were in line with the sorts of errors we see historically,” he says. “Historically, technically advanced societies think these methods are perfect, when of course they have error built in.”

In fact, the 2016 polls did reflect the outcome of the election since candidate Hillary Clinton won the popular vote by several million ballots. Opinion polls are not designed to factor in the complications of the United States unique electoral college system.

According to the press release, the researchers say they are not claiming that polling is not facing challenges, but that so far polling firms have faced up to those challenges. That doesn’t mean the accuracy will continue. Rogers point out that while around 2000 response rates to phone polls were about one-third, today it's fallen to one call in 10 that gets picked up, meaning polls have become less random and less representative. Internet polling has similar biases. That doesn’t mean pollsters can’t produce accurate polls from the data, just that they need to do more statistical adjustments. “Pollsters are constantly struggling with issues around changing electorates and changing technology,” Jennings tells Rodgers. “Not many of them are complacent. But it’s some reassurance that things aren’t getting worse.”

But we should really take a poll about that.

Jason Daley | | READ MORE

Jason Daley is a Madison, Wisconsin-based writer specializing in natural history, science, travel, and the environment. His work has appeared in Discover, Popular Science, Outside, Men’s Journal, and other magazines.