8.8 Designing Effective Surveys

Designing surveys takes some thought. In addition to constructing quality questions and posing clear response options, you will also need to think about how to present your written questions and response options to survey respondents. In this section we will discuss the sorts of things you should think about as you prepare to present your well-constructed survey questions.

One of the first things to do once you feel confident about the set of survey questions you have developed is to group those questions thematically. In our example of the transition to college, perhaps we would have a few questions asking about study habits, others focused on friendships, and still others on exercise and eating habits. Those may be the themes around which we organize our questions. Or perhaps it would make more sense to present questions about precollege life and habits, then a series of questions about life after beginning college. Be deliberate about how you present your questions to respondents.

Once you have grouped similar questions together, you will need to think about the order in which to present those question groups. Most survey researchers agree that it is best to begin a survey with questions that encourage respondents continue (Babbie, 2010; Palys & Atchison, 2014), i.e., do not bore respondents, but do not scare them away either. There is some disagreement over where on a survey to place demographic questions such as those about a person’s age, gender, and race. On the one hand, placing them at the beginning of the survey may lead respondents to think the survey is boring, unimportant, and not something they want to bother completing. They may also feel uncomfortable answering personal questions. On the other hand, if your survey deals with some very sensitive or difficult topic, such as child sexual abuse or other criminal activity, you do not want to scare respondents away or shock them by beginning with your most intrusive questions.

The order in which questions are presented on a survey is best determined by the unique characteristics of the research. The researcher, hopefully in consultation with people who are willing to provide you with feedback, can determine how best to order the questions. It helps to think about the unique characteristics of the topic, the questions, and, most importantly, the sample. Keeping in mind the characteristics and needs of the people who will be asked to complete the survey should help guide the researcher and determine the most appropriate order in which to present the questions.

Researchers also need to consider the time it will take respondents to complete the survey. Surveys vary in length, from just a page or two to a dozen or more pages, which means they also vary in the time it takes to complete them. How long to make your survey depends upon several factors. First, what do you wish to know? Wanting to understand how grades vary by gender and year in school certainly requires fewer questions than wanting to know how people’s experiences in college are shaped by demographic characteristics, college attended, housing situation, family background, college major, friendship networks, and extracurricular activities. Keep in mind that even if your research question requires including many questions, do your best to keep the survey as brief as possible. Any hint that you have thrown in several useless questions just for the sake of throwing them in will turn off respondents and may make them not want to complete your survey.

Second, and  perhaps  more  important,  how  long  are  respondents  likely  to  be  willing  to  spend completing your survey? If you are studying college students, asking them to use their precious fun time away from studying to complete your survey may mean that they will not want to spend more than a few minutes on it. However, if you have the endorsement of a professor who is willing to allow you to administer your survey in class, students may be willing to give you a little more time (though perhaps the professor will not be willing). The time that survey researchers ask respondents to spend on surveys varies greatly.

As with question order, there is no clear-cut, always-correct answer about survey length, but the general rule is to try to keep the time allotted to complete it under 15 minutes (Babbie, 2010). Consider the unique characteristics of your study and your sample in order to determine how long to make your survey. A good way to estimate the time it will take respondents to complete your survey is through pre-testing. Pre-testing allows you to get feedback on your survey, so you can improve it before you actually administer it. Pre-testing can be expensive and time consuming if you wish to test your survey on a large sample of people who very much resemble the sample to whom you will eventually administer the finalized version of your survey. However, you can learn a lot and make great improvements to your survey simply by pre-testing with a small number of people to whom you have easy access (perhaps you have a few friends you could ask). By pre-testing your survey you can find out how understandable your questions are, get feedback on question wording and order, find out whether any of your questions are exceptionally boring or offensive, and learn whether or not there are places where you should have included filter questions, to name just a few of the benefits. You can also time pre-testers as they take your survey. Ask them to complete the survey as though they were actually members of your sample. This will give you a good idea about what sort of time estimate to provide respondents when you administer your actual survey, and whether you have some wiggle room to add additional items or need to cut a few items.

Your survey should also be attractive. A messy presentation style can confuse respondents or, at the very least, annoy them. Be brief, to the point, and as clear as possible. Avoid cramming too much into a single page; make your font size readable (at least 12 point); leave a reasonable amount of space between items; and make sure all instructions are exceptionally clear. Think about books, documents, articles, or web pages that you have read yourself—which were relatively easy to read and why? Try to mimic those features in the presentation of your survey questions