Remember President Thomas Dewey? Or Commander-in-Chief Hillary Clinton? Or Republicans’ massive “red wave” of 2022? Neither do I. We would remember them if all the public polls had been accurate. Increasingly, election cycles are cluttered with an overabundance of polls that have a margin of error of plus or minus “a lot.”
To understand why so many polls prove inaccurate, you need to understand what goes into conducting a valid poll. Step one, get the sample universe right. Not an easy feat! Did the pollster decide to sample all registered voters, all likely voters, or simply anyone with a pulse? And if it was likely voters, as it should be, which likely voters? Those likely to vote in a presidential year? A midterm? An off-year election? It makes a huge difference. And just how big was the sample size? Obviously the larger, the more statistically valid.
The poll also needs to be demographically precise – meaning the right sample percentages by party registration, ideology, age, geography, gender, education, and marital status. Most credible pollsters don’t just release the topline numbers; they also make available the crosstabs, the side-by-side comparisons of different groups and how they answered various questions. Truth be told, this is where the real insights are to be found. It’s also where you can kick the tires and determine the quality of the survey. If the crosstabs are not released, then you have every right to be suspicious of the poll’s accuracy.
But we’re just getting started. How long did the survey take to complete? Ideally, no more than two or three nights. Beyond that, the poll is already starting to get stale. I’ve seen surveys released by universities that took as long as a month to field yet were treated by some in the press as credible and timely, generating significant headlines. And if the poll is first made public more than a week after its fielding date, due to ever-changing election events, it could already be obsolete.
And how was the poll conducted? By phone? Text? Online? And, if by phone, did the pollsters get the right number of landlines versus cellphones? And what nights were the calls made? Friday nights and weekends can produce very mixed results. Daytime calls will oversample seniors. In the South, Wednesday night calls will exclude weekly churchgoers. National holidays like Presidents Day can also skew results. And was the poll conducted before or after the most recent debate?
It should also be noted that some polls aren’t necessarily meant to be accurate – instead, they are simply meant to grab media attention and try to create momentum and fundraising opportunities for a candidate. They’re easy to spot because they’re almost always an internal campaign poll, or one conducted by a party-affiliated firm. Accuracy is rarely the top priority of this type of vanity poll.
Another problem is that perfectly valid polls are often inaccurately reported due to a reporter’s lack of understanding or a personal bias. It’s not unusual to see a headline touting a candidate’s large lead in a poll, while the reality is that the candidate has dropped from a sizable double-digit lead to a single-digit lead. The poll’s movement is the real story, not the static horserace number. Or a candidate may have a double-digit lead over a field of candidates, but “undecided” is winning in a landslide. Too often, the reported narrative of a poll story doesn’t match the reality of the numbers.
This brings us to the “push poll,” often incorrectly defined and rarely understood. Many valid surveys come with a series of questions known as “push questions.” Simply put, this is where positive or negative information about a candidate is provided, and the respondent sample is asked: “Does this make you more or less likely to vote for this candidate?”
After all the “push questions” have been completed, another ballot question is asked to see how many people changed their vote as a result. Many pollsters will then utilize “regression analysis” to measure which pieces of information were the greatest determinant of those who changed their vote. A credible public poll will only release ballot numbers before any “push questions” are asked.
But here’s where it gets confusing: utilizing “push questions” does not make a “push poll.” A “push poll” is a nefarious campaign practice where many people are called – many more than necessary for a valid poll sample – under the guise of being an independent poll. Then a series of negative “push poll” questions are read to the call’s recipients. Presented as a legitimate poll, negative information is instead being delivered to a wide-scale audience.
Public polls are certainly not going away anytime soon. But with just a little more knowledge about how they work, you can at least have some confidence about which ones to believe.