Election 2016           Social Desirability Bias / Trump Shaming:  Republican Primaries

The Republican primaries are a great case study for Social Desirability Bias.  We have a considerable amount of data from multiple polls in multiple states for multiple candidates.  Additionally, this data can be compared to actual results from the primaries.

It turns out that there was a fairly strong bias against Trump.  On average, the difference between live interviewer polls (‘live’) and robocalls / IVR or internet polls (‘anonymous’) was 6.1 percentage points against Trump.  In other words, people tended to answer voting intention questions very differently depending on if a live human over the phone asked the question or if they were allowed to answer in a more anonymous fashion.  Additionally, there was an anti-Trump poll bias when comparing actual primary results and anonymous polls with Trump receiving on average 7.7 percentage points more in the actual primary result than in the poll.

Another interesting fact is that the Trump discount varied according to location.  Most importantly, Trump’s live poll numbers suffered the most in traditionally Democratic states.  In other words, it appears that Trump shaming is most effective in states that are historically dominated by Democrats.

We will also see that there was the reverse effect, though to a much lesser extent, for Kasich.  As a candidate, he was very non-controversial.  In an election season that was characterized by political feuds and name calling, Kasich did his best to remain positive while trying to be ‘the adult in the room’.  The result appears to have been that more people were willing to support him publicly than those who actually liked and/or wanted to vote for him.  In Trump’s case, live polls were considerably lower than anonymous polls creating a negative bias.  In Kasich’s case, live polls were actually higher than anonymous polls creating a positive bias.  In other words, it seems like in a very emotionally charged election year, some people preferred to publicly but only superficially support the plain vanilla candidate to avoid social condemnation.  Social Desirability Bias tends to work both ways.

Some elements of the study:

• Twenty states were used.
• States divided into four sub-groups – Solid Republican States (TX, GA, LA, SC, AZ), Flipped States (NC, IN), Swing States (VA, OH, FL), and Solid Democratic States (MI, PA, NY, WI, CT, MA, NH, MD, IL, RI). Flipped States are those that changed from Democrat to Republican from 2008 to 2012.
• Poll data was taken from RealClearPolitics.
• Poll data divided into two groups, (1) ‘live’ or those using live interviewer based phone calls, and (2) ‘anonymous’ or those using non-live interviewers, such as ‘robocalls’, IVR, and internet.
• Polling methodology was confirmed by visiting poll urls or reading the actual poll.
• Poll data from one month prior to the primary vote was used to calculate averages.
• Home states were excluded from each candidate’s averages as there would be a considerably lower expected bias against them in their home state.
• Some states were excluded from the study due to lack of sufficient poll data.
• States holding voting after Cruz and Kasich left the race were excluded.
• Polls were excluded if they used a combination of live and anonymous polling with an unclear explanation of the underlying mix (such as WeAskAmerica in IL).
• Polls were excluded if the results were well out of the normal range and they produced a very limited number of polls (such as Optimus in NY).
• Iowa was excluded due to the significant impact of Rubio, who was not included in the overall study as he dropped out earlier than the others.

The following table highlights the conclusions as broken down by state group.

Table 1:  Comparing Actual Primary Results to Average Live Polls and Comparing Average Anonymous Polls to Average Live Polls, for Trump, Cruz and Kasich during 2016 Republican Primaries

Source:  Realclearpolitics.com, Wikipedia

Four main points jump out from the data: (1) Trump’s negative live polling bias, (2) Trump’s worst bias was in Solid Democratic States, (3) Kasich’s positive live polling bias, and (4) Traditional Swing States tend to have lower levels of bias.

Trump had by far the largest poll bias.   Seven of eight of his net numbers presented in the table are the highest.  The implication is that on average more people actually supporting this candidate felt that admitting to that support in public was not socially desirable than any other candidate.

Of Trump’s numbers, the most extreme appear in Solid Democratic States.  The average difference between his live poll numbers and his anonymous poll numbers was an astounding 7.1 percentage points.  The figure including actual results jumped to 10.3 percentage points.  In other words, the anonymous polls were much closer to the actual but actual results were still better.  These results make sense in that a Social Desirability Bias against Trump in traditionally Democratic states seems obvious.  Comparing the same figures to those of Cruz and Kasich for these states and it looks like Trump shaming was extremely effective in Democratic states.

Kasich appears to have the opposite bias in that he tended to do better across the board in live polls.  The net difference between his anonymous polls and live polls is actually negative.  In other words, people seem to have seen Kasich as a socially acceptable choice among Republican candidates and used Kasich as a public choice but in reality supported a different candidate.  The caveat here is that these figures are very low.  However, consistency across the groups of states supports its existence.

As a group, the figures from Swing States are noticeably lower.  Though there is not enough data to actually prove this theory, it could be that VA, FL, and OH have received so much scrutiny and polled so often that the populace is more at ease with answering polls and with political divergence.  Put another way, because they have been in the political spotlight for a considerable period of time, they might not feel as constrained about answering political questions and might not feel as much social pressure to answer one way or the other.  This would reduce the impact of Social Desirability Bias.

We will revisit these concepts in other posts so this is not the final conclusion.  For now, there are some basic takeaways, especially as they relate to the general election:

1. Live Polls Discount Trump – it seems after looking at Republican primary data, that Trump’s true support is actually much higher than live polls show. For general election polls, you probably want to put more weight on anonymous polls or use live polls while adjusting for an assumed Trump discount.
2. Northeast and Rustbelt – Trump’s true support was significantly higher in these areas as compared to live polls. For the general election, if Trump polls anywhere within 10 percentage points of Clinton in these states, he could actually have a chance at winning.
3. Swing States – polls seem to be more accurate due to an apparent lack of social pressure. Though this should be confirmed with general election polls, using just Republican primary data, it would seem that Swing State polls can more readily relied upon.