8.6 Designing Effective Survey Questions

We have considered several general points about surveys, including some of their pros and cons, as well as when to use surveys, and how often and in what ways to administer them. In this section we will get more specific and take a look at how to pose understandable questions that will yield useable data and how to present those questions on your survey.

Asking Effective Survey Questions.

The first thing you need to do in order to write effective survey questions is to identify what exactly it is that you wish to know. While that should go without saying, we cannot stress enough how easy it is to forget to include important questions when designing a survey. For example, suppose you want to understand how students at your school made the transition from high school to college. You wish to identify which students were comparatively more or less successful in this transition and which factors contributed to students’ success or lack thereof. To understand which factors shaped successful students’ transitions to college, you will need to include questions in your survey about all the possible factors that could contribute. Consulting the literature on the topic will certainly help, but you should also take the time to do some brainstorming on your own and to talk with others about what they think may be important in the transition to college. Perhaps time or space limitations will not allow you to include every single item you have come up with, so you will need to think about ranking your questions to be sure to include those that you view as most important.

Although we have stressed the importance of including questions on all topics you view as important to your overall research question, you do not want to take an everything-but-the-kitchen-sink approach by uncritically including every possible question that occurs to you. Doing so puts an unnecessary burden on your survey respondents. Remember that you have asked your respondents to give you their time and attention and to take care in responding to your questions; show them your respect by only asking questions that you view as important.

Once you have identified all the topics about which you would like to ask questions, you will need to actually write those questions. Questions should be as clear and to the point as possible. This is not the time to show off your creative writing skills; a survey is a technical instrument and should be written in a way that is as direct and succinct as possible. The best way to show your appreciation for your respondents´ time is to not waste it. Ensuring that your questions are clear and not overly wordy will go a long way toward showing your respondents the gratitude they deserve.

To properly value respondents’ time, make sure that every question you pose will be relevant to every person you ask to complete your survey. This means two things: first, that respondents have knowledge about your survey topic, and second, that respondents have experience with the events, behaviours, or feelings you are asking them to report. In our example of the transition to college, heeding the criterion of relevance would mean that respondents must understand what exactly you mean by “transition to college” (if you are going to use that phrase in your survey) and have actually experienced the transition to college themselves.

When developing survey questions, a researcher must consider the following aspects:

Context effects: This can be a function of funneling or be inadvertent, but questions that are asked can prime (i.e., make more salient) certain views or thoughts that then impact the way respondents answer subsequent questions. For example, if we ask you a number of questions about harm reduction and the Insite Safe Injection Site, and then ask you whether you support the Safe Injection Site, you may be more likely to support the site than if I had asked you several questions about crime in the area of the site before asking you if you support the site.

Context appropriate wording: It is important that the wording you choose is appropriate for the people who are going to be answering your questions. You should not ask people questions they cannot understand due to their age, or language barriers (including jargon). Use vocabulary appropriate for the people who are answering your survey.

Minimizing bias: Questions with loaded terms (e.g., adjectives like disgusting, dangerous, or wonderful; and terms like always or never) and non-neutral wording should be avoided. These questions ultimately lead people to the “correct” answer. The tone of the question will also impact how people answer. People answering the questions should not feel judged for their response or their opinion. If they do, they are less likely to answer the question honestly; instead, they will answer the question the way they think you want them to respond.

Ambiguity: Questions can be ambiguous in many ways. This is one area that can benefit from pilot testing (or pre-testing) your questions to determine which questions can be interpreted differently from your intended meaning. In particular, use of words like “often” or “sometimes” can result in different interpretations. However, even words that appear to be clear to the researcher can be misinterpreted by the respondents and make the question difficult for them to answer. Acronyms can also make questions difficult to answer if they are unknown to the respondents. As noted above, context appropriate wording to the audience responding to the questions should be considered; thus, acronyms are sometimes appropriate.

Meaningless responses: People can and do respond to questions about things about which they have no knowledge. As a researcher, you want responses by people who have some knowledge of the subject or ability to meaningfully answer the question.

Double-barreled questions: This type of question should be avoided at all costs – essentially this is a question where there is more than one question within it. For example: Do you enjoy biking and hiking in your free time? If a respondent enjoys biking but not hiking, how do they respond?

If you decide that you do wish to pose some questions about matters with which only a portion of respondents will have had experience, it may be appropriate to introduce a filter question into your survey. A filter question is designed to identify some subset of survey respondents who are asked additional questions that are not relevant to the entire sample.

There are some ways of asking questions that are bound to confuse survey respondents. Researchers should take great care to avoid these  kinds  of  questions.  These include: questions that pose double negatives, those that use confusing or culturally specific terms, and those that ask more than one question but are posed as a single question. Any time respondents are forced to decipher questions that utilize two forms of negation, confusion is bound to ensue. In general, avoiding negative terms in your question wording will help to increase respondent understanding. You should also avoid using terms or phrases that may be regionally or culturally specific (unless you are absolutely certain all your respondents come from the region or culture whose terms you are using).

Another thing to avoid when constructing survey questions is the problem of social desirability. We all want to look good, right? And we all probably know the politically correct response to a variety of questions, whether we agree with the politically correct response or not. In survey research, social desirability refers to the idea that respondents will try to answer questions in a way that will present them in a favourable light. Perhaps we decide that to understand the transition to college, we need to know whether respondents ever cheated on an exam in high school or college. We all know that cheating on exams is wrong, so it may be difficult to get people to admit to cheating on an exam in a survey. But if you can guarantee respondents’ confidentiality, or even better, their anonymity, chances are much better that they will be honest about having engaged in this socially undesirable behaviour. Another way to avoid problems of social desirability is to try to phrase difficult questions in the most benign way possible. Babbie (2010) offers a useful suggestion for helping you do this—simply imagine how you would feel responding to your survey questions. If you would be uncomfortable, chances are others would as well.

Finally, it is important to get feedback on your survey questions in a pre-test, from as many people as possible, especially people who are like those in your sample. Now is not the time to be shy. Ask your friends for help, ask your mentors for feedback, ask your family to take a look at your survey as well. The more feedback you can get on your survey questions, the better are the chances that you will come up with a set of questions that are understandable to a wide variety of people and, most importantly, to those in your sample.

In order to pose effective survey questions, researchers should do the following:

1. Identify what it is they wish to know.
2. Keep questions clear and succinct.
3. Make questions relevant to respondents.
4. Use filter questions when necessary.
5. Avoid questions that are likely to confuse respondents, such as those that use double negatives or culturally specific terms, or pose more than one question in the form of a single question (double-barreled question).
6. Imagine how they would feel responding to these questions themselves.
7. Get feedback, especially from people who resemble those in the researcher’s sample.