New York Times reporter Josh Katz put a lot of time and a lot of energy into trying to bust the myth that polling in Alaska is more difficult than polling elsewhere. Apparently, a bunch of people are discounting numbers coming out of Alaska, which is news to me. Perhaps Katz is responding to the blow-back from the New York Times’ new, online survey, which has been highly panned.
In any case, in order to try and bust the myth, Katz analyzed 889 polls spread over 155 Senate races across the country since 1992, He found that Alaska comes in as the fifth-most error prone state, behind Maine, New York, Maryland and Georgia. “(I)t’s not clear that we should discount numbers coming out of Alaska any more than we should numbers from, say, Georgia or Pennsylvania,” Katz writes.
There’s some glaring holes here. One, the Senate races in those other states — particularly in New York and even in Maine which shares a media market with Massachusetts — surely generated many more polls than did the races in Alaska. Secondly, until 2004 when Tony Knowles ran for Lisa Murkowski’s seat, and then in 2008, when Mark Begich ran against Ted Stevens, races were all but uncontested here. Even the New York Times’ new online survey could have called those races.
All of which leads to the real problem, which was also missed by Katz. According to Anchorage-based pollster Marc Hellenthal — who is the only pollster in the state who doesn’t contract out his polls — polling in Alaska isn’t necessarily more difficult, it’s just more expensive. And it’s more expensive because there hasn’t been enough polls conducted in Alaska to generate good call lists.
“There’ just not been the profit motive to justify building up good samples,” for other polling firms to use, Hellenthal said.
In other words, there aren’t likely even enough Alaska polls to try and analyze if we have good or bad polls.
Contact Amanda Coyne at amandamcoyne@yahoo.com
Meh, other than the cited poll in the article almost every poll the month prior to the election had Begich with a small lead over Stevens… which is exactly what happened.
Amanda, they were nice enough to share the data (I’ll email it) and it looks like Alaska’s fairly well represented, so I’m unsure there’s really a blowout factor to worry about given the measurement Josh employed.
Amanda, I’m not sure how much the blowout factor matters since the statistic at focus is the difference between a given race’s predicted margin and its actual margin. (Irrespective, I assume, of whether it was a blowout or not.) But given your question I reached out to The Upshot to ask how many Alaska races were in the data set, which may clarify things … stay tuned!
@Chris. Point taken. I was just surprised that he didn’t think to take those things into account. Analyzing polls and investigating limitations of such work is supposed to be something that he’s good enough at that the New York Times would see fit to put him on its site. You also don’t mention one of the other main points: how easy it was to call the Senate races in Alaska pre-2004. I haven’t gone through the other states, but I’d bet few state races were so consistently predictable.
The two big points made here are both valid. But I feel the article is far too quick in blowing off The Upshot’s analysis and its interpretation of the results.
Yes, the scope of the data—889 polls since 1992 spread across 50 states—doesn’t allow for a very big per-state sample size, particularly for smaller states such as Alaska. Good point.
And yes, the absence of strong, fully representative voter samples does make it sound (per Hellenthal’s comments) as though sampling bias presents a problem in Alaska polling.
So perhaps Katz subscribed too quickly to his own interpretation without effectively investigating or explaining the limitations of his work. I don’t think either of those two points—that there are too few polls for comfort, and that the presence of sampling issues makes polling difficult—justify scrapping the results. The data offers insight, despite those two limitations. Besides, after this year’s Senate race, we (The Times) can rerun the numbers and, given the attention being given to Alaska this year, help overcome the first limitation.
The New York Times is going to need a lot more than this article to recover from their blatant stupoidity.
The NYT article is a feeble attempt to recoup some credibility in the polling arena after their recent debacle. The last NYT poll was ridiculous, lacked scientific credibility and standards. It’s a shame that a paper of this magnitude allowed their stff to run contrary to their own published standards. Good report. Thanks much.