Yesterday afternoon, following a contentious press conference, President Donald Trump’s campaign team sent his supporters links to an online questionnaire called the Mainstream Media Accountability Survey. And then, as these things are wont to do, the survey spread via social media, #TrumpSurvey.
The survey asked participants to weigh in on the media’s bias against Trump, on issues like immigration, religion, the economy, and his presidency as a whole. Fair enough. However, as many critics on social media pointed out, the survey’s wording was often loaded, not to mention confusing. Any other critiques aside, questions like these are not the way to get scientific results. “A lot of these questions are designed in a way to confirm a particular hypothesis,” says Carey Morewedge, an associate professor of marketing at Boston University. “They basically are suggesting there is a correct answer.”
To get a better sense of whether the results of this survey could be used to understand the opinions of the public about the media—or even just Trump supporters’ opinions—we talked to academics like Morewedge who regularly used surveys to do research, as well as those who study the psychology of survey taking.
“1. Do you believe that the mainstream media has reported unfairly on our movement?”
The first question gets right to the point: Them against us. “Putting the ‘our movement’ is what linguists call the framing effect. They’re introducing a frame, like, ‘Come into this perspective, this space of mine, it’s our movement together,’” says Carolyn Dicey Jennings, a philosopher at University of California, Merced. “And also the ‘mainstream media’, that’s also a linguistic cue because that phrasing is used almost always in criticism of the media.”
Jennings points out that many of these questions—at least 16 of the 25—are framed in a way that signals to the reader that they are meant to be answered yes. This leads to what psychologists call acquiescence bias. Jennings says a better way of framing the question would be something like, “How has the media reported on the GOP movement?” and have response choices on a scale that runs from “fairly” to “unfairly”.
“12. Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump’s temporary restriction executive order?”
Daniel Oppenheimer, a psychologist at UCLA who’s written a book about the flaws in elections, pointed out that the survey was doing what is called “push polling”—trying to convey some information rather than asking people what they think. “This is not an attempt to get information about the media, but to sway opinions,” Oppenheimer says.
This question also takes advantage of what psychologists call ‘the bandwagon effect’—if others believe it, then you should. Jennings points out that this question is suggesting that everyone else already supports Trump’s executive order. Other questions in the survey use the same technique, but instead use an authority figure (the president) to start the bandwagon.
“14. Do you believe that contrary to what the media says, raising taxes does not create jobs?”
Question 14 has a double negation (“contrary”, then “does not”). “Which is very difficult for people to process,” says Morewedge. “If you have an affiliation with Trump, and your default response up to that point is to say yes and agree to things, you’re going to have a bias to agree with it even if you don’t understand what the question means,” Morewedge says.
“15. Do you believe that people of faith have been unfairly characterized by the media?”
In general, questions framed like this ask respondents to confirm, rather than consider, their biases, because it begins by prompting them to look in their own experience for times when they’ve noticed unfair characterizations.
“A more neutral kind of framing would be something like ‘do you believe people of faith have been fairly or unfairly characterized by the media’—that’s still kind of leading, but it’s going to be more likely to evoke both kinds of evidence,” says Morewedge. “Or you could just ask, ‘how do you feel people of faith have been treated by the media?’”
“24. Do you agree with President Trump’s media strategy to cut through the media’s noise and deliver our message straight to the people?”
Throughout the survey, questions are loaded with words—like ‘noise’—that impart a negative frame on the survey’s target: the media. “Nobody likes chaos and noise,” writes Teenie Matlock, a cognitive and information scientist at UC Merced, via email. “Framing it this way makes it sound as if Trump is saving Americans from confusion and chaos.” She noted several examples of negative framing throughout the survey.
The survey contains many other flaws. For instance, it portrays the media as a monolith. But this industry includes magazines like WIRED, cable news networks ranging from Fox News to MSNBC, newspapers of all stripes, and a potpourri of digital offerings. Also, as a survey sent primarily to Trump’s supporters, it is aimed at a predisposed audience. That means that the answers will be based on who chooses to take the survey (and who knows about the survey), rather than a true indication of the distribution of attitudes across the entire US population. Plus, there’s nothing really stopping people from taking the survey multiple times. Just make up a new name, email, and zip code, and voila, you’ve committed survey fraud.
Academics, at least, see one good use for the survey: as a cautionary tale:
Colleagues: materials for your next #sociology research methods workshops. #TrumpSurvey HT @JessieNYC https://t.co/dNfjhvhIPy pic.twitter.com/mNsHP8qWjd
— Sociology at Work (@sociologyatwork) February 17, 2017
Go Back to Top. Skip To: Start of Article.