Clinton, Trump, and interval Estimates

Or subtitled:

3% margin of error? We don' need no steenking margin of error

This morning brought to TOF's attention the following factoids regarding recent polls taken in three "battleground states." Apparently, a "battleground" state is one in which the media in their anointed role as declarer of winners -- it's in the Constitution, right? -- cannot bring themselves to declare a winner just yet.

Here is the info from the story:

Donald Trump enters the Republican convention on a small roll in the three most important swing states in the country.

Clinton's biggest drop was in Florida, where last month she led Trump by 8 points. In Wednesday's poll of the state, Trump garnered 42% support to Clinton's 39% — within the poll's 3.1% margin of error.

Wednesday's poll of Pennsylvania voters, meanwhile, found Trump with 43% support to Clinton's 41%, a slight change from last month's poll which found the former secretary of state leading Trump by one point.

Ohio's race remained unchanged, as both candidates remained tied for the second month in a row.

How an unchanged tie in a sample is part of a "roll" is unexplained. The print media is more careful than broadcast, and the pollsters themselves are more careful than the media. In broadcast, we often hear that this candidate or that is "gaining" or "losing" from one month to the next when in fact we are only seeing noise in the sampling. The reason why Trump appears to lead by 3 points this month while Clinton appeared to lead by 8 points last month is that the pollsters talked to a different 1000 voters¹ this month than last, and they had different preferences than the last bunch.

Note: 1. The sample sizes in this case were 1025 for FL, 955 for Ohio, and 982 for Pennsylvania. The differences are likely due to the number of people randomly called who hung up or otherwise refused to participate. It is dangerous to assume that the uncooperative will have the same opinions as the cooperative.

Furthermore, a sample is only a part of a population and unless it is taken in a random fashion -- and this is so incredibly hard to do that most pollsters don't bother doing it -- it is unlikely to peg the actual preferences of the population of which it is a part. The sample may for example haul in more Democrats this month than last or more older people or more of the well-to-do. If the preferences differ between clusters in the population, this will affect the sample percentages in ways that are not accounted for by the so-called margin of error. The Quinnipiac sample does make adjustments for the proportions of age groups, sexes, races, counties, party affiliations, etc. harvested by the sample versus the same groups' proportions in the population from Census and other sources. There does not seem to have been an adjustment for non-response. Care was taken to correct for multiple voters using the same land line and for voters having more than one phone.

That margin of error is calculated on the assumption of a simple randomly collected sample. In a randomly collected sample, each member of the population has an equal (or at least a known) probability of entering the sample. In practice, this is seldom the case and political polls are taken with little care for good sample planning. Unlike market surveys, on which $much$ often rides, political polls are as effervescent as the bubbles in your beer. In two months, no one will care. More to the point, no one will know if you ever got it right. Except for that very last one the day before the actual vote when snarky statisticians can actually compare polls to actual election results. 

Now here's the dirty little secret: you can have a 3.1% margin of error as these polls claim bracketing a completely wrong value! Let's have fun with comparisons!

The Quinnipiac poll above sampled 982 PA voters between June 30-July 11, 2016 and found

 
Clinton...........41% interval estimate (38%-44%)
Trump.............43....................(40%-46%)
SMONE ELSE(VOL)....4
WLDN'T VOTE(VOL)...7
DK/NA..............6
 

When folks were reminded that there are other candidates running, the percentages change:

 
Clinton...........34%...................(31%-37%)
Trump.............40....................(37%-43%)
Johnson (Lib)......9
Stein (Green)......3
SMONE ELSE(VOL)....3
WLDN'T VOTE(VOL)...3
DK/NA..............8

Faced with two additional candidates, 10% jumped ship and 3rd party picked up 12%, but "someone else" dropped by only 1%, "wouldn't vote" dropped from 7% to 3%, and "don't know" increased from 6% to 8%. Go figure. Remember, the same people answered both questions.

Now here's something curious. During the same time frame, NBC/WSJ sampled 829 PA voters between July 5-10, 2016 and found the following percentages in their sample.
Clinton..........45%...interval estimate....(42%-48%)
Trump............36%........................(33%-39%)
Neither..........11%
Undecided.........6%
Other.............2%

For Clinton, the minimum and maximum likely percentages estimated by the samples were
Q,#1..............(38....xx....44) (Clinton v. Trump)
Q,#2(31....xx....37)  (Clinton v. Trump v. Johnson v. Stein)
NBC.......................(42....xx....48)

For Trump, the minimum and maximum likely percentages estimated by the samples were
Q,#1...................(40....xx....46) (Clinton v. Trump)
Q,#2.............(37....xx....43)  (Clinton v. Trump v. Johnson v. Stein)
NBC......(33....xx....39)

So the 3% margin of error is 3% around estimates that sometimes differ by more than 3% even when sampling the same population at the same time. Quinnipiac talked to a different group of people than did NBC/WSJ and so got different results. The 3% precision is trivial next to the vaster question of accuracy. In fact, given the two questions on the Quinnipiac poll, the way the question is asked makes a difference.