{"id":48,"date":"2018-03-28T19:58:49","date_gmt":"2018-03-28T23:58:49","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rmip3amed\/chapter\/5-3-experimentation-and-validity\/"},"modified":"2018-03-28T20:21:07","modified_gmt":"2018-03-29T00:21:07","slug":"5-3-experimentation-and-validity","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/rmip3amed\/chapter\/5-3-experimentation-and-validity\/","title":{"raw":"5.3 Experimentation and Validity","rendered":"5.3 Experimentation and Validity"},"content":{"raw":"\n
\n

Learning Objectives<\/h3>\n
    \n
  1. Explain what internal validity is and why experiments are considered to be high in internal validity.<\/li>\n
  2. Explain what external validity is and evaluate studies in terms of their external validity.<\/span><\/li>\n
  3. Explain the concepts of construct and statistical validity.<\/li>\n<\/ol>\n<\/div>\n

    Four Big Validities<\/span><\/strong><\/h2>\n

    When we read about psychology experiments with a critical view, one question to ask is \u201cis this study valid?\u201d However, that question is not as straightforward as it seems because, in psychology, there are many different kinds of validities. Researchers have focused on four validities to help assess whether an experiment is sound (Judd & Kenny, 1981; Morling, 2014)[footnote]Judd, C.M. & Kenny, D.A. (1981). Estimating the effects of social interventions<\/em>. Cambridge, MA: Cambridge University Press.[\/footnote][footnote]Morling, B. (2014, April). Teach your students to be better consumers. APS Observer<\/em>. Retrieved from http:\/\/www.psychologicalscience.org\/index.php\/publications\/observer\/2014\/april-14\/teach-your-students-to-be-better-consumers.html[\/footnote]: internal validity, external validity, construct validity, and statistical validity. We will explore each validity in depth.<\/span><\/p>\n

    Internal Validity<\/span><\/strong><\/h3>\n

    Two variables being statistically related does not necessarily mean that one causes the other. \u201cCorrelation does not imply causation.\u201d For example, if it were the case that people who exercise regularly are happier than people who do not exercise regularly, this implication would not necessarily mean that exercising increases people\u2019s happiness. It could mean instead that greater happiness causes people to exercise or that something like better physical health causes people to exercise<\/span> and<\/em><\/span> be happier.<\/span><\/p>\n

    The purpose of an experiment, however, is to show that two variables are statistically related and to do so in a way that supports the conclusion that the independent variable caused any observed differences in the dependent variable. The logic is based on this assumption: If the researcher creates two or more highly similar conditions and then manipulates the independent variable to produce just <\/span>one<\/span> difference between them, then any later difference between the conditions must have been caused by the independent variable. For example, because the only difference between Darley and Latan\u00e9\u2019s conditions was the number of students that participants believed to be involved in the discussion, this difference in belief must have been responsible for differences in helping between the conditions.<\/span><\/p>\n

    An empirical study is said to be high in <\/span>internal validity<\/span><\/strong> if the way it was conducted supports the conclusion that the independent variable caused any observed differences in the dependent variable. Thus experiments are high in internal validity because the way they are conducted\u2014with the manipulation of the independent variable and the control of extraneous variables\u2014provides strong support for causal conclusions. In contrast, nonexperimental research designs (e.g., correlational designs), in which variables are measured but are not manipulated by an experimenter, are low in internal validity.<\/span><\/p>\n

    <\/b>External Validity<\/b><\/h3>\n

    At the same time, the way that experiments are conducted sometimes leads to a different kind of criticism. Specifically, the need to manipulate the independent variable and control extraneous variables means that experiments are often conducted under conditions that seem artificial (Bauman, McGraw, Bartels, & Warren, 2014)[footnote]Bauman, C.W., McGraw, A.P., Bartels, D.M., & Warren, C. (2014). Revisiting external validity: Concerns about trolley problems and other sacrificial dilemmas in moral psychology. Social and Personality Psychology Compass, 8\/9<\/em>, 536-554.[\/footnote]. In many psychology experiments, the participants are all undergraduate students and come to a classroom or laboratory to fill out a series of paper-and-pencil questionnaires or to perform a carefully designed computerized task. Consider, for example, an experiment in which researcher Barbara Fredrickson and her colleagues had undergraduate students come to a laboratory on campus and complete a math test while wearing a swimsuit (Fredrickson, Roberts, Noll, Quinn, & Twenge, 1998)<\/span>[footnote]Fredrickson, B. L., Roberts, T.-A., Noll, S. M., Quinn, D. M., & Twenge, J. M. (1998). The swimsuit becomes you: Sex differences in self-objectification, restrained eating, and math performance. Journal of Personality and Social Psychology, 75<\/em>, 269\u2013284.[\/footnote]. At first, this manipulation might seem silly. When will undergraduate students ever have to complete math tests in their swimsuits outside of this experiment?<\/span><\/p>\n

    The issue we are confronting is that of <\/span>external validity<\/span><\/strong>. An empirical study is high in external validity if the way it was conducted supports generalizing the results to people and situations beyond those actually studied. As a general rule, studies are higher in external validity when the participants and the situation studied are similar to those that the researchers want to generalize to and participants encounter every day, often described as mundane realism<\/strong>.<\/span><\/a><\/sup> Imagine, for example, that a group of researchers is interested in how shoppers in large grocery stores are affected by whether breakfast cereal is packaged in yellow or purple boxes. Their study would be high in external validity and have high mundane realism if they studied the decisions of ordinary people doing their weekly shopping in a real grocery store. If the shoppers bought much more cereal in purple boxes, the researchers would be fairly confident that this increase would be true for other shoppers in other stores. Their study would be relatively low in external validity, however, if they studied a sample of undergraduate students in a laboratory at a selective university who merely judged the appeal of various colors presented on a computer screen; however, this study would have high psychological realism<\/strong> where the same mental process is used in both the laboratory and in the real world.  If the students judged purple to be more appealing than yellow, the researchers would not be very confident that this preference is relevant to grocery shoppers\u2019 cereal-buying decisions because of low external validity but they could be confident that the visual processing of colors has high psychological realism.<\/span><\/p>\n

    We should be careful, however, not to draw the blanket conclusion that experiments are low in external validity. One reason is that experiments need not seem artificial. Consider that Darley and Latan\u00e9\u2019s experiment provided a reasonably good simulation of a real emergency situation. Or consider field<\/span> experiments<\/span> that are conducted entirely outside the laboratory. In one such experiment, Robert Cialdini and his colleagues studied whether hotel guests choose to reuse their towels for a second day as opposed to having them washed as a way of conserving water and energy (Cialdini, 2005)<\/span>[footnote]Cialdini, R. (2005, April). Don\u2019t throw in the towel: Use social influence research. APS Observer<\/em>. Retrieved from http:\/\/www.psychologicalscience.org\/index.php\/publications\/observer\/2005\/april-05\/dont-throw-in-the-towel-use-social-influence-research.html[\/footnote]. <\/span>These researchers manipulated the message on a card left in a large sample of hotel rooms. One version of the message emphasized showing respect for the environment, another emphasized that the hotel would donate a portion of their savings to an environmental cause, and a third emphasized that most hotel guests choose to reuse their towels. The result was that guests who received the message that most hotel guests choose to reuse their towels, reused their own towels substantially more often than guests receiving either of the other two messages. Given the way they conducted their study, it seems very likely that their result would hold true for other guests in other hotels.<\/span><\/p>\n

    A second reason not to draw the blanket conclusion that experiments are low in external validity is that they are often conducted to learn about psychological <\/span>processes<\/span> that are likely to operate in a variety of people and situations. Let us return to the experiment by Fredrickson and colleagues. They found that the women in their study, but not the men, performed worse on the math test when they were wearing swimsuits. They argued that this gender difference was due to women\u2019s greater tendency to objectify themselves\u2014to think about themselves from the perspective of an outside observer\u2014which diverts their attention away from other tasks. They argued, furthermore, that this process of self-objectification and its effect on attention is likely to operate in a variety of women and situations\u2014even if none of them ever finds herself taking a math test in her swimsuit.<\/span><\/p>\n

    Construct Validity<\/span><\/strong><\/h3>\n

    In addition to the generalizability of the results of an experiment, another element to scrutinize in a study is the quality of the experiment\u2019s manipulations or the <\/span>construct validity<\/span><\/strong>. The research question that Darley and Latan\u00e9 started with is \u201cdoes helping behavior become diffused?\u201d They hypothesized that participants in a lab would be less likely to help when they believed there were more potential helpers besides themselves. This conversion from research question to experiment design is called <\/span>operationalization <\/span><\/strong>(see Chapter 4 for more information about the operational definition). Darley and Latan\u00e9 operationalized the independent variable of diffusion of responsibility by increasing the number of potential helpers. In evaluating this design, we would say that the construct validity was very high because the experiment\u2019s manipulations very clearly speak to the research question; there was a crisis, a way for the participant to help, and increasing the number of other students involved in the discussion, they provided a way to test diffusion.<\/span><\/p>\n

    What if the number of conditions in Darley and Latan\u00e9\u2019s study changed? Consider if there were only two conditions: one student involved in the discussion or two. Even though we may see a decrease in helping by adding another person, it may not be a clear demonstration of diffusion of responsibility, just merely the presence of others. We might think it was a form of Bandura\u2019s social <\/span>inhibition. The construct validity would be lower. However, had there been five conditions, perhaps we would see the decrease continue with more people in the discussion or perhaps it would plateau after a certain number of people. In that situation, we may not necessarily be learning more about diffusion of responsibility or it may become a different phenomenon. By adding more conditions, the construct validity may not get higher. When designing your own experiment, consider how well the research question is operationalized your study.<\/span><\/p>\n

    Statistical Validity<\/span><\/strong><\/h3>\n

    Statistical validity<\/strong> concerns the proper statistical treatment of data and the soundness of the researchers\u2019 statistical conclusions. There are many different types of inferential statistics tests (e.g., t-<\/em>tests, ANOVA, regression, correlation) and statistical validity concerns the use of the proper type of test to analyze the data. When considering the proper type of test, researchers must consider the scale of measure their dependent variable was measured on and the design of their study. Further, many of inferential statistics tests carry certain assumptions (e.g., the data are normally distributed) and statistical validity is threatened when these assumptions are not met but the statistics are used nonetheless.<\/span><\/p>\n

    One common critique of experiments is that a study did not have enough participants. The main reason for this criticism is that it is difficult to generalize about a population from a small sample. At the outset, it seems as though this critique is about external validity but there are studies where small sample sizes are not a problem (subsequent chapters will discuss how small samples, even of only 1 person, are still very illuminating for psychology research). Therefore, small sample sizes are actually a critique of <\/span>statistical validity<\/span>. The statistical validity speaks to whether the statistics conducted in the study are sound and support the conclusions that are made.<\/span><\/p>\n

    The proper statistical analysis should be conducted on the data to determine whether the difference or relationship that was predicted was found. The number of conditions and the total number of participants will determine the overall size of the effect. With this information, a power analysis can be conducted <\/span>to ascertain whether you are likely to find a real difference.<\/span> When designing a study, it is best to think about the power analysis so that the appropriate number of participants can be recruited and tested. To design a statistically valid experiment, thinking about the statistical tests at the beginning of the design will help ensure the results can be believed.<\/span><\/p>\n

    Prioritizing Validities<\/span><\/strong><\/h2>\n

    These four big validities--internal, external, construct, and statistical--are useful to keep in mind when both reading about other experiments and designing your own. However, researchers must prioritize and often it is not possible to have high validity in all four areas. In Cialdini\u2019s study on towel usage in hotels, the external validity was high but the statistical validity was more modest. This discrepancy does not invalidate the study but it shows where there may be room for improvement for future follow-up studies (Goldstein, Cialdini, & Griskevicius, 2008)[footnote]Goldstein, N. J., Cialdini, R. B., & Griskevicius, V. (2008). A room with a viewpoint: Using social norms to motivate environmental conservation in hotels. Journal of Consumer Research, 35<\/em>, 472\u2013482.[\/footnote]. Morling (2014) points out that most psychology studies have high internal and construct validity but sometimes sacrifice external validity.<\/span><\/p>\n

    \n

    Key Takeaways<\/h3>\n