Thinking Like a Sociologist: Deconstructing Polls

By Karen Sternheimer

A recent episode of The Daily Show satirized both left wing and right wing cable news shows for using questionable poll data in their broadcasts. Regardless of your political orientation, this clip is a great introduction to some of the core concepts sociologists use in research methods and statistics: survey construction, sampling, and margin of error. Think about these issues as you watch the video below.

 

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Poll Bearers
www.thedailyshow.com
Daily Show Full Episodes Political Humor Healthcare Protests

1. Survey construction

Did you notice the wording of the questions in the polls? Good surveys should not suggest how respondents should answer them, but some of the polls included wording that could certainly bias responses. Note how some of the questions in this clip tend to encourage one response or another, particularly the questions about taxes. Most people would respond negatively to a general rise in taxes. But there are a wide variety of taxes (sales and income taxes to name a few), and rates vary based on a variety of factors (such as location, income level, and family size). By contrast, the tax that excludes most potential respondents (raising taxes only on a small percentage of Americans) is more likely to generate favorable results.

A good survey question is worded as neutrally as possible and is specific so that respondents are very clear about the meaning of their response. A good question about taxes would be clear on what specific tax rate would change, by how much, and for whom. The word “tax” is so loaded that another word might even be used in its place, such as fee or levy.

2. Sampling

Let’s say we’ve written good survey questions. Our next step is to administer our survey to an appropriate sample.

What is a good sample? That depends on which group we are trying to learn about. If we want to get the opinion of sociology majors from across the country, then we certainly wouldn’t want to poll people who pass through the center of a couple campuses. For one, we would get a lot of non-sociology majors, and secondly, a small number of campuses would not adequately measure the nation’s sociology majors.

Many polls reported on the news imply that they represent views of Americans. That might seem like a daunting task, but it is possible to get a sense of public opinion this way.

You might be thinking that with a population around 300 million, it is nearly impossible to sample a thousand or so people and claim to know what Americans are thinking. To do so, pollsters need to construct a probability sample, which means that every American should have an equal opportunity to be chosen at random to be part of the sample.

Hypothetically, you can think of a random sample as similar to putting everyone in the population’s name in a huge hat which is shaken so anyone’s name can potentially be pulled.

But there isn’t a hat that big, nor is it practical for someone to write everyone’s name down on a slip of paper. Pollsters often rely on measures which are admittedly imperfect. One of the most common is random digit dialing, where a computer calls random phone numbers in hopes of getting respondents.

Have you ever gotten one of these calls? If you’re like me, you get a lot of calls that interrupt your busy day or evening, and caller id allows you to ignore the numbers you don’t recognize. This has created sampling challenges: if there are consistent patterns in who is more or less likely to participate then your survey results might be skewed.

So there are no perfect samples, but there are some really imperfect samples, like the ones in the Daily Show clip above. Some internet and television polls make no attempt to create a probability sample, so their responses are limited to the demographic already watching or logging onto their site.

There are also other factors which might make someone more or less likely to participate. For instance, if a television show asks you to text your response, viewers might be more likely to participate if their cell phone plan has generous texting privileges. Beyond that, some people are more motivated to respond than others. So these polls might be fun, but they are not very useful gauges of public opinion. That’s what “results not scientific” means if they flash it on the screen. But often shows do not.

3. Margin of Error

Even the polls with well-written questions and the best possible samples are approximations of public opinion, rather than a perfect reflection of what people really think about an issue. While we might not ever be 100% certain that a probability sample reflects the true opinion of a population, thanks to probability theory, we can be 95% confident that the actual opinion lies within a certain range.

Here’s an example. Gallup regularly conducts job approval surveys of political leaders. In an August 2009 survey, they found that 31% of Americans approve of the job Congress is doing. As you can see from the graph below, the approval rating appears to have fallen from previous months.

But every poll has a margin of error, since surveys are approximations rather than perfect reflections of a population’s beliefs. The margin of error is a measure that takes into account the sample size and the amount of variation of responses. In this case, the margin of error is ±4, meaning that we can be 95% certain that between 27% and 35% of Americans approve of the job that Congress is doing.

Now, if I were a member of Congress I wouldn’t be too excited about this range of scores, but because it is within the range of scores from earlier months we probably cannot say with confidence that Americans’ approval of Congress is declining. But no matter how you look at it, it’s pretty low.

You might notice that news shows reporting polls rarely discuss the meaning of the margin of error, if they mention it at all. Keeping this important measure in mind, you can use your sociological imagination to think critically about the next poll you hear about in the news.

Comments

[ostrix]

Unfortunately, I didn't manage to open the video, even when I tried from the site itself... I agree that margin error is an important factor that is rarely emphasized. While reading this post, I thought to myself that it would be good to have read it when I was attending the course of Statistics in the university :) You won't believe just how many people watch these polls and haven't got the slightest clue about methods of statistical calculations, margin errors and things like that.

[amanda funaro 3]

I agree with ostrix about how many people do not have any clue about how polls work but I also want to point my finger at the media. If these big news companies and assuming the people who are conducting the polls are aware of their sampling errors then why air a poll that has 100% of the people agree? Is it to sway our opinion? Or is there really no one who disagrees that town halls are making a difference? Whatever the case may be Americans watching should be asking themselves how accurate these polls really are. Also they should ask themselves why do these news anchors look so "surprised" when 100% of the people who took this poll agree? Lastly, I agree that the wording of these polls are poorly constructed and the answers are pretty much bias.

[Emily]

Excellent post. I'm taking a sociology class and a lot of what I read applies to what I'm learning at the moment.
I conducted my own little expieriment a while ago and made a sample survey. It really is suprising how the wording of a survey can influence a persons decision, whether intended or not.
It's unbelievable how easily people can fall into these false ratings. What's even worse is that the media is the one who reports them. Does anyone do research to see if these polls are accurate?

[soc]

Way to really emphasize how much the media really skews things. I wasn't able to watch the video, but I was still able to get the general idea of what you were saying. This post related a lot to a survey I just did in my sociology class, dealing with margin of error and how much the wording of a question can effect the responses. Really interesting to see how small things that most don't notice can make such big differences, and it's sad to see the media rely on tactics like that to get ratings.

[Paulus]

I'm glad you addressed this issue in your writing. It is not only inaccurate over-all for people to try to influence the answers of people taking their polls,but it can often be down right deceitful. When a person creating a survey uses the wording of the question to suggest the answer that they want people to choose. I wish more people understood the significance of taking a poll, and that their answer should be 100% of their own will. Thanks for enlightening us on this issue.

[Julia]

Topic:

Hi, I am a high school sociology student just learning about all the meaning behind the numbers. Your post about the research methods of sociologists really helped me to understand the problems sociologists face as they try to formulate and conduct surveys to support a study. The example of the tax questions posed by the political parties portrayed how the format of a question can manipulate and skew the results, intentionally or unintentionally. I knew that the wording of a question could sway participants to answer one way or another, but I liked how you pinpointed the connotations of the very word “tax” itself. In this economic climate, any word associated with paying money has very heavy weight indeed.

[Jaclyn]

It seems to me that the media in most ways minipulates a lot of our thoughts whether it be through newspaper articles to even those that the government puts out for society to intake on their everyday lives. Simple things are very easily twisted to make the consumer misinterpret the actual study. "Surveys" are misread very often if not at all specific. Plus, it seems that they are misinterpreted when a person may hear or see one thing in the survey they do not like, and not really look at the true questions that are being compared or asked.

[Matt]

A good sociologist doing research will create an accurate survey with relevent questions. The polling audience should cover many ages and both genders. The wider the audience the more accurate your answers will be.

[Kristen]

This article gave good insight on how you should construct a poll, and how many companies DO construct polls the 'wrong' way. I like that it brings to your attention that polls are not always exact, and it also shows you how you can make them more exact. After reading this article I now realize that just because I hear a new statistic on TV or read one in a magazine it doesn't mean that it is accurate. If you want to find out if it is an accurate stat, you should find out what the "sampling" was and how the questions were asked.

[Missy]

This was an excellent post, I'm glad that you brout up wording of a survey, this can greatly affect a persons decision. I even learnd some things i didnt know, like a lot of theese posts I'm in a sociology class. I will defenintly benefit from reading this excerpt, thanks so much.

[alena]

Great post! i'm taking a sociology class right now and just finished a lesson on statistics and sociologists doing research. You made alot of excelent points in your article that alot of people probably never thought about. I think alot of people are unaware of how much effort is needed in creating an acurate survey, and you explained how important that is.

[Sam]

Wow! I want to just point out two things.
1. People need to learn to make polls without scewing other peoples answers BEFORE putting them on the air. It just makes them look like idiots and I don't think I will be able to trust a poll like that in a long time. and..
2. Jon Stewart is halarious.
The wording of a poll also has alot to do with someones answer. I'm glad you pointed that out because it's a need to know. Great job on this post Karen. :)

[Brian Clark]

I would have to agree. I am surprised at the data that news networks and publications use. Some of this data is collected by people who try to say things that changes people's actual answer. I'm in a high school sociology class and one of our projects was to survey ten different people and one problem I ran into was word-for-word asking the same question to every person, or elaborating for someone who didn't "understand" the question. Things like this can easily skew the data for any kind of survey.

[Katelyn]

This blog entry does an excellent job at illustrating how important the set up, and use of surveys are. If one word is off, or a certain group excluded, the survey can display completly different results. This is why when reviewing a survey, one must look over the details.

[Sally]

This blog entry shows how things can go good or bad in a survey. Sociologists have to be careful how they word answers or show emotion to one answer rather than the other. It was really interesting to see how the media does their polls, especially the one where people said 100% yes to a question but the text number was the same. I'm happy I decided to view this because now when I do surveys I will be extra careful not to give any kind of influence.

[Shelby]

Great post! I'm a high school student that's taking a Sociology class. So I know about polls and surveys already with that class and other classes I've taken in the past. But this really shows me how the results can be effected by the littliest things such as the way you word it. Also, it shows me how survey or poll results can never really be 100% accurate.

[Aaron]

This really shows how easily survey results can be skewed by the wording of the question. I'm in a sociology class right now, and just learning how to conduct an effective and accurate survey. These guidelines have to be followed if you want an accurate picture of a group's general opinion.

[Abbey]

This post was very informative and interesting. As a high school sociology student learning about surveys I found this blog entry to be very eye opening. It showed me how hard it is to actually create a good poll with no biases. It even comes down to little details like word choice-using "tax" or "fee" can potentially change the way someone might answer a question. The results can become very skewed if you don't follow the guidelines listed above.