49 Designing Effective Surveys

In addition to constructing quality questions and posing clear response options, you will also need to think about how to present your written questions and response options to survey respondents.  Designing surveys takes some thought. In this section we will discuss the sorts of things you should think about as you prepare to present your well-constructed survey questions.

One of the first things to do once you have developed a set of survey questions you feel confident about is to group those questions thematically. In our example of the transition to college, perhaps we would have a few questions asking about study habits, others focused on friendships, and still others on exercise and eating habits. Those may be the themes around which we organize our questions. Or perhaps it would make more sense to present any questions we had about precollege life and habits and then present a series of questions about life after beginning college. The point here is to be deliberate about how you present your questions to respondents.

Once you have grouped similar questions together, you will need to think about the order in which to present those question groups. Most survey researchers agree that it is best to begin a survey with questions that will want to make respondents continue (Babbie, 2010; Palys & Atchison, 2014). In other words, do not bore respondents, but do not scare them away either. There is some disagreement over where on a survey to place demographic questions such as those about a person’s age, gender, and race. On the one hand, placing them at the beginning of the survey may lead respondents to think the survey is boring, unimportant, and not something they want to bother completing. They may also feel uncomfortable answering personal questions.  On the other hand, if your survey deals with some very sensitive or difficult topic, such as child sexual abuse or other criminal activity, you do not want to scare respondents away or shock them by beginning with your most intrusive questions.

In truth, the order in which questions are presented on a survey is best determined by the unique characteristics of the research. The researcher, hopefully in consultation with people who are willing to provide you with feedback, can determine how best to order the questions. It helps to think about the unique characteristics of the topic, the questions, and most importantly, the sample. Keeping in mind the characteristics and needs of the people who will be asked to complete the survey should help guide the researcher and determine the most appropriate order in which to present the questions.

Researchers also need to consider the time it will take respondents to complete the survey. Surveys vary in length, from just a page or two to a dozen or more pages, which means they also vary in the time it takes to complete them. How long to make your survey depends upon several factors. First, what is it that you wish to know? Wanting to understand how grades vary by gender and year in school certainly requires fewer questions than wanting to know how people’s experiences in college are shaped by demographic characteristics, college attended, housing situation, family background, college major, friendship networks, and extracurricular activities. Keep in mind that even if your research question requires a good number of questions be included, do your best to keep the survey as brief as possible. Any hint that you have thrown in a bunch of useless questions just for the sake of throwing them in will turn off respondents and may make them not want to complete your survey.

Second, and perhaps more important, how long are respondents likely to be willing to spend completing your survey? If you are studying college students, asking them to use their precious fun time away from studying to complete your survey may mean they will not want to spend more than a few minutes on it. However, if you have the endorsement of a professor who is willing to allow you to administer your survey in class, students may be willing to give you a little more time (though perhaps the professor will not). The time that survey researchers ask respondents to spend on surveys varies greatly.

As with question order, there is no clear-cut, always-correct answer about survey length, but the general rule is to try and keep it under 15 minutes (see Babbie, 2010). The unique characteristics of your study and your sample should be considered in order to determine how long to make your survey.  A good way to estimate the time it will take respondents to complete your survey is through pretesting. Pretesting allows you to get feedback on your survey, so you can improve it before you actually administer it. Pretesting can be expensive and time consuming if you wish to test your survey on a large sample of people who very much resemble the sample to whom you will eventually administer the finalized version of your survey. However, you can learn a lot and make great improvements to your survey simply by pretesting with a small number of people to whom you have easy access (perhaps you have a few friends you could ask). By pretesting your survey you can find out how understandable your questions are, get feedback on question wording and order, find out whether any of your questions are exceptionally boring or offensive, and learn whether there are places where you should have included filter questions, to name just a few of the benefits of pretesting. You can also time pre-testers as they take your survey. Ask them to complete the survey as though they were actually members of your sample. This will give you a good idea about what sort of time estimate to provide respondents when it comes time to actually administer your survey, and about whether you have some wiggle room to add additional items or need to cut a few items.

Perhaps this goes without saying, but your survey should also be attractive. A messy presentation style can confuse respondents or, at the very least, annoy them. Be brief, to the point, and as clear as possible. Avoid cramming too much into a single page, make your font size readable (at least 12 point), leave a reasonable amount of space between items, and make sure all instructions are exceptionally clear. Think about books, documents, articles, or web pages that you have read yourself—which were relatively easy to read and easy on the eyes and why? Try to mimic those features in the presentation of your survey questions.

Text Attributions

• This chapter has been adapted from Chapter 8.4 in Principles of Sociological Inquiry, which was adapted by the Saylor Academy without attribution to the original authors or publisher, as requested by the licensor. © Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.