4.2 Validity

While reliability is about consistency, validity is about shared understanding. What image comes to mind for you when you hear the word alcoholic? Are you certain that the image you conjure up is similar to the image others have in mind? If not, then we may be facing a problem of validity.

To be valid, we must be certain that our measures accurately get at the meaning of our concepts. Think back to the first possible measure of alcoholism we considered in the subsection “Reliability.” There, we initially considered measuring alcoholism by asking research participants the following question: Have you ever had a problem with alcohol? We realized that this might not be the most reliable way of measuring alcoholism because the same person’s response might vary dramatically depending on how he or she is feeling that day. Likewise, this measure of alcoholism is not particularly valid. What is “a problem” with alcohol? For some, it might be having had a single regrettable or embarrassing moment that resulted from consuming too much. For others, the threshold for “problem” might be different; perhaps a person has had numerous embarrassing drunken moments but still gets out of bed for work every day and he therefore does not perceive himself as having a problem. Because what each respondent considers to be problematic could vary so dramatically, our measure of alcoholism is not likely to yield any useful or meaningful results if our aim is to objectively understand, say, how many of our research participants are alcoholics.

Here is another example: Perhaps we are interested in learning about a person’s dedication to healthy living. Most of us would probably agree that engaging in regular exercise is a sign of healthy living, so we could measure healthy living by counting the number of times per week that a person visits his local gym. At first this might seem like a reasonable measure, but if this respondent’s gym is anything like some of the gyms, there exists the distinct possibility that his gym visits include activities that are decidedly not fitness related. Perhaps he visits the gym to use the tanning beds, not a particularly good indicator of healthy living, or to flirt with potential dates or sit in the sauna. These activities, while potentially relaxing, are probably not the best indicators of healthy living. Therefore, recording the number of times a person visits the gym may not be the most valid way to measure his or her dedication to healthy living. Using this measure would not really give us an indication of a person’s dedication to healthy living and therefore, we would not really be measuring what we intended to measure.

Indeed, in the social sciences it is often not as straightforward as A causes B in the classic experiments. Frequently, there are many other variables that may occur at the same time that A and/or B cause both A and B. Therefore, a researcher must be careful to ensure that his or her study has internal validity — that it does, in fact, test the very thing it seeks to test. There are several threats to internal validity (e.g. history, maturation, testing, and regression to the mean, selection biases, and instrumentation) and ways to control for these types of threats, e.g., experiment and the use of a control or comparison groups.  We will return to the topic of internal validity in Chapter 6.

Researchers usually also want external validity, meaning that they want their study to be generic to other situations and contexts, beyond the current project. They also want it to reflect real world environments where the phenomena occur and to prove that it was not due to chance that they got the findings they did. As Palys and Atchison (2014) state, it does not, necessarily, have anything to do with the representativeness of the sample.  Rather, it depends upon the nature of the phenomenon under study and on the research objectives.

At its core, validity is about social agreement. One quick and easy way to help ensure that your measures are valid is to discuss them with others. One way to think of validity is to think of it as you would a portrait. Some portraits of people look just like the actual person they are intended to represent. But other representations of people’s images, such as caricatures and stick drawings, are not nearly as accurate. While a portrait may not be an exact representation of how a person looks, what’s important is the extent to which it approximates the look of the person it is intended to represent. The same goes for validity in measures. No measure is exact, but some measures are more accurate than others.