Battle of the polls: the Vance-Elway debate

Pollster Stuart Elway rebuts an allegation from The Vance Report that the former overstates Democratic numbers.

<a href="/static/story_image/elway_poll.jpg">Enlarge</a>.

Chuck Taylor

Enlarge.

Chris Vance is on a mission to kill the messenger again. Four polls last week show Patty Murray with a substantial lead over Dino Rossi, yet Vance once again has been compelled to explain to his readers that The Elway Poll “often has been somewhat off” in its polls of Washington political races. Really?

Vance says, “the only way to measure a pollster’s accuracy is to compare their final poll to actual election results, if they release a poll within a week or so of election day.”  There are many things wrong with that assertion, but let’s accept it for the sake of argument. The first thing to know that The Elway Poll has never done or published a poll within two weeks of an election. I believe that the purpose of public surveys is to explain the election dynamics, not to predict the outcome. No public purpose is served by last-day polls. I know this sounds quaint in the current atmosphere, but there it is.

Vance goes on to explain that SurveyUSA “routinely releases a poll just before the election, and in 2008 they were within 1 percentage point of the actual Washington state results for governor and president.” He also notes that Rasmussen’s final polls “came in earlier” but were also “largely accurate.”

As evidence that The Elway Poll is “off,” Vance abandons his own “last week of the election” criterion and compares those last-day polls to an Elway Poll in September of 2004 and a poll in mid October of 2008. The new criteria is that our 2004 poll was “in contrast to what other polls were showing at the time.” And the 2008 poll overestimated the size of Gregoire’s eventual victory over Rossi.

So let’s accept those criteria and look at the record. How has The Elway Poll done at foreshadowing election results?

Since 1992, The Elway Poll has published a total of 29 polls in September and October on races for President, US Senator and Governor. The eventual winner was leading in 25 of those 29 polls (we had 86 percent right). Two of the four we missed were in the 2000 Senate race — we had Gorton ahead by 1 point in September and 3 points in October. Sen. Gorton lost by 1 point, well within the margin of error, but we’ll count them as misses. Our other two misses were in the 1992 Governor’s race, where we had Eikenberry ahead of Lowry in both September and October. Note that all four of our misses involved having the Republican ahead, in contrast to the assertion that The Elway Poll has a Democrat bias.

Across those 29 polls, we have underestimated the Republican candidate 27 times — by an average of 7 points. That is, the Republican ended up with a higher percentage of the vote than he received in our poll. We under-estimated the Democrat candidate 25 times — also by an average of 7 points. How can that be? As I have noted elsewhere, we typically ask a “soft” ballot question and do not push undecided voters. This results in larger “undecided” numbers in our polls than in others. Since there are no “undecided” voters in election results, the candidates’ vote totals will be higher than the poll results.

Next, let’s look at the specific polls Vance cites as evidence of our inaccuracy. The Elway Poll in September 2004 showed Gregoire at 49 percent, which is the vote she got in November. We showed Murray at 57 percent; she got 55 percent. That may be “in sharp contrast to what other polls were showing at the time,” as Vance asserts, but it hardly supports his other assertion that our polls “dramatically overstated the vote the two Democrats would receive two months later.”

How about that other criteron for judging pollsters work, “what other polls were saying at the time”? In the 2006 Senate race, the story line all year was that it was a very tight race. There were eight polls published between late April and mid June. Our two polls at that time showed Cantwell with leads over Republican Mike McGavick of 14 and 22 points (average equals 18; remember that number). The other six polls showed Cantwell’s lead at 4-7 points. Rasmussen had it at 5 and 4. In September, there were seven published polls. Cantwell’s lead in ours was 18. The other six ranged from 6 to 17 points — both the 6 and the 17 were from Rasmussen polls, conducted two weeks apart. Our October poll (two weeks out) showed Cantwell ahead by 18. SurveyUSA’s last poll, one week later, had Cantwell ahead by 7. She won by 18 points.

In the 2008 Governor’s race our polls from July on consistently showed Gregoire well ahead of Dino Rossi, while no other poll showed a lead of more than 4 points. Our September poll, which Vance omits in his critique of our polls, had Gregoire up by 8 points. That same week, SurveyUSA had Rossi up by 1 and Rasmussen had Rossi up by 6. True, we bounced up to a 12-point Gregoire lead in October, but during the same week, SurveyUSA had her lead at 1 point and Rasmussen had it a 2 points — both “too close to call.” Gregoire won by 6. So yes, we were “in contrast to what other polls were showing at the time.”

This year’s story line is familiar. The race is believed to be tight. The national polls are all over the yard, but The Elway Poll has consistently shown a comfortable lead for the Democrat. Partisans on the short end of the data attack The Elway Poll.

Am I saying that Murray will win by 13 points? No. All I am saying is all any pollster can ever say: I am 95 percent certain that my last poll was accurate within the margin of error (4.6%) at the time it was taken. Remember, any survey is only a snapshot it time and polls do not predict the future. All that boring stuff matters about margin of error and confidence intervals and the rest.

I think the problem that Vance and his friends have with The Elway Poll is that it keeps indicating that Republicans are losing. But they keep losing. What am I supposed to do?


Topics: Washington, Politics

About the Author

H. Stuart Elway is founder and president of Elway Research, a public opinion firm that conducts The Elway Poll statewide each month. You can reach him in care of editor@crosscut.com.

Like what you just read? Support high quality local journalism. Become a member of Crosscut today!

Comments:

Posted Mon, Oct 18, 6:20 a.m. Inappropriate

It looks like Chris Vance has some explaining to do, again. Stuart has pretty well cleaned his clock.

Jan

Posted Mon, Oct 18, 8:52 a.m. Inappropriate

In the professional services world, all one has is one's reputation. Kudos, Stuart, for setting the record straight.

Rita

Posted Mon, Oct 18, 9:35 a.m. Inappropriate

Dear Chris Vance:

Exactly how would a pollster conduct a poll or release its results "within a week or so of election day" when vote-by-mail makes "election day" a meaningless anachronism of the past?

Stuart:

Thanks for a concise and logical explanation of what's what. While it's disappointing to see Crosscut continuing to publish the baseless partisan propaganda of Chris Vance (honestly, can't Crosscut find a credible conservative?) it's good to see that they'll run a fact-based, in-depth retort.

Posted Mon, Oct 18, 10:17 a.m. Inappropriate

To Super_Steve:

Most pollsters continue polling right up until the weekend before the election. In our state they will ask, "have you already voted, and if so, who did you vote for?"

It was not my intent to pick a fight with Mr. Elway. What interests me is why the polls are so inconsistent this year.

Still, Mr. Elway didn't say anything that conflicts with what I said. His poll results usually come too early to be verified by actual election results, they are often quite different than other polls done at the same time, and they seem to overstate the margin between candidates. Again, I am forced to say "seem" because unlike other pollsters, we can't compare them to actual election results.

Posted Mon, Oct 18, 11:05 a.m. Inappropriate

Chris - As the wise adage says; "Better to remain silent and be thought a fool than to speak out and remove all doubt."

By the last real Republican, A. Lincoln

Posted Tue, Oct 19, 10:22 a.m. Inappropriate

After reading Stuart Elway's effective rebuttal, then Chris Vance's comment above, one realizes it would be better for Vance to take the 5th.

No pollster is perfect, Elway included. To his credit these many past decades, he has been straight up about that. Moreover, unlike Vance, Elway's professionalism has earned him a reputation for integrity.

Mr. Vance on the other hand--as we have seen this fall in his blind shilling for his empty-suit Senate candidate--is known for spinning many Republican Titanics into would-be cruise ships. Anybody remember George Nethercutt??

Sperry

sperry

Posted Tue, Oct 19, 3:32 p.m. Inappropriate

www.538.com

Posted Sat, Oct 23, 1:28 p.m. Inappropriate

As a retired survey researcher and politcal polster, I have to put in my 2 cents worth. Ranges and confidence levels ('plus or minus so much at the 95% level of confidence') are useful, but they only tell part of the story. They assume one of two things: 1) everyone responds or 2)people that they don't reach or who refuse are no different than the people who take part. Neiter is true in the real world, especially now that many people only have cell phones. 'Robopolls', if I remember correctly, have a response rate around 2%-10%. Any poll conducted in a day or two, as are the ones released just before the election, will -- at best -- reach 25% of the numbers that are dialed. They may, by random, be right on. A weather forcaster in Phoenix who predicts rain day after day will be right some of the time.

Enough. Well -- almost enough. If I remember correctly, Stuart came into polit

Posted Sat, Oct 23, 1:29 p.m. Inappropriate

(It kicked me out. I'm a poor typist.)

Stuart's background is Republican.

Posted Mon, Oct 25, 1:19 p.m. Inappropriate

Dr. Elway remains, among pollsters, as my students used to say, "da bomb." Mr. Vance has engaged, throughout this campaign, in a lot of wishful thinking.

T.M. Sell

Login or register to add your voice to the conversation.

Join Crosscut now!
Subscribe to our Newsletter

Follow Us »