{"id":71,"date":"2019-09-17T12:19:27","date_gmt":"2019-09-17T16:19:27","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rmethodspsych\/chapter\/experimental-design\/"},"modified":"2019-09-17T12:25:36","modified_gmt":"2019-09-17T16:25:36","slug":"experimental-design","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/rmethodspsych\/chapter\/experimental-design\/","title":{"raw":"Experimental Design","rendered":"Experimental Design"},"content":{"raw":"\n
\n

Learning Objectives<\/h3>\n
    \n\t
  1. Explain the difference between between-subjects and within-subjects experiments, list some of the pros and cons of each approach, and decide which approach to use to answer a particular research question.<\/span><\/li>\n\t
  2. Define random assignment, distinguish it from random sampling, explain its purpose in experimental research, and use some simple strategies to implement it.<\/span><\/li>\n\t
  3. Define what a control condition is, explain its purpose in research on treatment effectiveness, and describe some alternative types of control conditions.<\/span><\/li>\n\t
  4. Define several types of carryover effect, give examples of each, and explain how counterbalancing helps to deal with them.<\/span><\/li>\n<\/ol>\n<\/div>\n

    In this section, we look at some different ways to design an experiment. The primary distinction we will make is between approaches in which each participant experiences one level of the independent variable and approaches in which each participant experiences all levels of the independent variable. The former are called between-subjects experiments and the latter are called within-subjects experiments.<\/span><\/p>\n\n

    Between-Subjects Experiments<\/span><\/h2>\n

    In a <\/span>between-subjects experiment<\/span><\/strong>, each participant is tested in only one condition. For example, a researcher with a sample of 100 <\/span>university<\/span> students might assign half of them to write about a traumatic event and the other half write about a neutral event. Or a researcher with a sample of 60 people with severe agoraphobia (fear of open spaces) might assign 20 of them to receive each of three different treatments for that disorder. It is essential in a between-subjects experiment that the researcher assign participants to conditions so that the different groups are, on average, highly similar to each other. Those in a trauma condition and a neutral condition, for example, should include a similar proportion of men and women, and they should have similar average intelligence quotients (IQs), similar average levels of motivation, similar average numbers of health problems, and so on. This matching is a matter of controlling these extraneous participant variables across conditions so that they do not become confounding variables.<\/span><\/p>\n\n

    Random Assignment<\/span><\/h2>\n

    The primary way that researchers accomplish this kind of control of extraneous variables across conditions is called <\/span>random assignment<\/span><\/strong>, which means using a random process to decide which participants are tested in which conditions. Do not confuse random assignment with random sampling. Random sampling is a method for selecting a sample from a population, and it is rarely used in psychological research. Random assignment is a method for assigning participants in a sample to the different conditions, and it is an important element of all experimental research in psychology and other fields too.<\/span><\/p>\n

    In its strictest sense, random assignment should meet two criteria. One is that each participant has an equal chance of being assigned to each condition (e.g., a 50% chance of being assigned to each of two conditions). The second is that each participant is assigned to a condition independently of other participants. Thus one way to assign participants to two conditions would be to flip a coin for each one. If the coin lands heads, the participant is assigned to Condition A, and if it lands tails, the participant is assigned to Condition B. For three conditions, one could use a computer to generate a random integer from 1 to 3 for each participant. If the integer is 1, the participant is assigned to Condition A; if it is 2, the participant is assigned to Condition B; and if it is 3, the participant is assigned to Condition C. In practice, a full sequence of conditions\u2014one for each participant expected to be in the experiment\u2014is usually created ahead of time, and each new participant is assigned to the next condition in the sequence as he or she is tested. When the procedure is computerized, the computer program often handles the random assignment.<\/span><\/p>\n

    One problem with coin flipping and other strict procedures for random assignment is that they are likely to result in unequal sample sizes in the different conditions. Unequal sample sizes are generally not a serious problem, and you should never throw away data you have already collected to achieve equal sample sizes. However, for a fixed number of participants, it is statistically most efficient to divide them into equal-sized groups. It is standard practice, therefore, to use a kind of modified random assignment that keeps the number of participants in each group as similar as possible. One approach is <\/span>block randomization<\/span><\/strong>. In block randomization, all the conditions occur once in the sequence before any of them is repeated. Then they all occur again before any of them is repeated again. Within each of these \u201cblocks,\u201d the conditions occur in a random order. Again, the sequence of conditions is usually generated before any participants are tested, and each new participant is assigned to the next condition in the sequence. <\/span>Table 6.2<\/span> shows such a sequence for assigning nine participants to three conditions. The Research Randomizer website (<\/span>http:\/\/www.randomizer.org<\/a><\/span>) will generate block randomization sequences for any number of participants and conditions. Again, when the procedure is computerized, the computer program often handles the block randomization.<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
    Table 6.2 Block Randomization Sequence for Assigning Nine Participants to Three Conditions<\/em><\/caption>\n
    <\/b>Participant<\/b><\/td>\n<\/b>Condition<\/b><\/td>\n<\/tr>\n
    1<\/b><\/td>\nA<\/b><\/td>\n<\/tr>\n
    2<\/b><\/td>\nC<\/b><\/td>\n<\/tr>\n
    3<\/b><\/td>\nB<\/b><\/td>\n<\/tr>\n
    4<\/td>\nB<\/td>\n<\/tr>\n
    5<\/td>\nC<\/td>\n<\/tr>\n
    6<\/td>\nA<\/td>\n<\/tr>\n
    7<\/b><\/td>\nC<\/b><\/td>\n<\/tr>\n
    8<\/b><\/td>\nB<\/b><\/td>\n<\/tr>\n
    9<\/b><\/td>\nA<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Random assignment is not guaranteed to control all extraneous variables across conditions. It is always possible that just by chance, the participants in one condition might turn out to be substantially older, less tired, more motivated, or less depressed on average than the participants in another condition. However, there are some reasons that this possibility is not a major concern. One is that random assignment works better than one might expect, especially for large samples. Another is that the inferential statistics that researchers use to decide whether a difference between groups reflects a difference in the population takes the \u201cfallibility\u201d of random assignment into account. Yet another reason is that even if random assignment does result in a confounding variable and therefore produces misleading results, this confound is likely to be detected when the experiment is replicated. The upshot is that random assignment to conditions\u2014although not infallible in terms of controlling extraneous variables\u2014is always considered a strength of a research design.<\/span><\/p>\n\n

    Treatment and Control Conditions<\/span><\/h2>\n

    Between-subjects experiments are often used to determine whether a treatment works. In psychological research, a <\/span>treatment<\/span><\/strong> is any intervention meant to change people\u2019s behaviour for the better. This<\/span> intervention<\/span> includes psychotherapies and medical treatments for psychological disorders but also interventions designed to improve learning, promote conservation, reduce prejudice, and so on. To determine whether a treatment works, participants are randomly assigned to either a <\/span>treatment condition<\/span><\/strong>, in which they receive the treatment, or a <\/span>control condition<\/span><\/strong>, in which they do not receive the treatment. If participants in the treatment condition end up better off than participants in the control condition\u2014for example, they are less depressed, learn faster, conserve more, express less prejudice\u2014then the researcher can conclude that the treatment works. In research on the effectiveness of psychotherapies and medical treatments, this type of experiment is often called a <\/span>randomized clinical trial<\/span><\/strong>.<\/span><\/p>\n

    There are different types of control conditions. In a <\/span>no-treatment control condition<\/span><\/strong>, participants receive no treatment whatsoever. One problem with this approach, however, is the existence of placebo effects. A <\/span>placebo<\/span><\/strong> is a simulated treatment that lacks any active ingredient or element that should make it effective, and a <\/span>placebo effect<\/span><\/strong> is a positive effect of such a treatment. Many folk remedies that seem to work\u2014such as eating chicken soup for a cold or placing soap under the bedsheets to stop nighttime leg cramps\u2014are probably nothing more than placebos. Although placebo effects are not well understood, they are probably driven primarily by people\u2019s expectations that they will improve. Having the expectation to improve can result in reduced stress, anxiety, and depression, which can alter perceptions and even improve immune system functioning (Price, Finniss, & Benedetti, 2008)<\/span>[footnote]Price, D. D., Finniss, D. G., & Benedetti, F. (2008). A comprehensive review of the placebo effect: Recent advances and current thought. Annual Review of Psychology, 59<\/em>, 565\u2013590.[\/footnote].<\/p>\nPlacebo effects are interesting in their own right (see <\/span>Note \"The Powerful Placebo\"<\/span>), but they also pose a serious problem for researchers who want to determine whether a treatment works. <\/span>Figure 6.2<\/span> shows some hypothetical results in which participants in a treatment condition improved more on average than participants in a no-treatment control condition. If these conditions (the two leftmost bars in <\/span>Figure 6.2<\/span>) were the only conditions in this experiment, however, one could not conclude that the treatment worked. It could be instead that participants in the treatment group improved more because they expected to improve, while those in the no-treatment control condition did not.<\/span>\n\n[caption id=\"attachment_382\" align=\"aligncenter\" width=\"900\"]\"Figure<\/a> Figure 6.2 Hypothetical Results From a Study Including Treatment, No-Treatment, and Placebo Conditions[\/caption]\n

    Fortunately, there are several solutions to this problem. One is to include a <\/span>placebo control condition<\/span><\/strong>, in which participants receive a placebo that looks much like the treatment but lacks the active ingredient or element thought to be responsible for the treatment\u2019s effectiveness. When participants in a treatment condition take a pill, for example, then those in a placebo control condition would take an identical-looking pill that lacks the active ingredient in the treatment (a \u201csugar pill\u201d). In research on psychotherapy effectiveness, the placebo might involve going to a psychotherapist and talking in an unstructured way about one\u2019s problems. The idea is that if participants in both the treatment and the placebo control groups expect to improve, then any improvement in the treatment group over and above that in the placebo control group must have been caused by the treatment and not by participants\u2019 expectations. This<\/span> difference<\/span> is what is shown by a comparison of the two outer bars in <\/span>Figure 6.2<\/span>.<\/span><\/p>\n

    Of course, the principle of informed consent requires that participants be told that they will be assigned to either a treatment or a placebo control condition\u2014even though they cannot be told which until the experiment ends. In many cases the participants who had been in the control condition are then offered an opportunity to have the real treatment. An alternative approach is to use a <\/span>waitlist control condition<\/span><\/strong>, in which participants are told that they will receive the treatment but must wait until the participants in the treatment condition have already received it. This <\/span>disclosure <\/span>allows researchers to compare participants who have received the treatment with participants who are not currently receiving it but who still expect to improve (eventually). A final solution to the problem of placebo effects is to leave out the control condition completely and compare any new treatment with the best available alternative treatment. For example, a new treatment for simple phobia could be compared with standard exposure therapy. Because participants in both conditions receive a treatment, their expectations about improvement should be similar. This approach also makes sense because once there is an effective treatment, the interesting question about a new treatment is not simply \u201cDoes it work?\u201d but \u201cDoes it work better than what is already available?<\/span><\/p>\n\n

    \n

    The Powerful Placebo<\/span><\/h4>\n

    Many people are not surprised that placebos can have a positive effect on disorders that seem fundamentally psychological, including depression, anxiety, and insomnia. However, placebos can also have a positive effect on disorders that most people think of as fundamentally physiological. These include asthma, ulcers, and warts (Shapiro & Shapiro, 1999)<\/span>[footnote]Shapiro, A. K., & Shapiro, E. (1999). The powerful placebo: From ancient priest to modern physician<\/em>. Baltimore, MD: Johns Hopkins University Press.[\/footnote].<\/span> There is even evidence that placebo surgery\u2014also called \u201csham surgery\u201d\u2014can be as effective as actual surgery.<\/span><\/p>\n

    Medical researcher J. Bruce Moseley and his colleagues conducted a study on the effectiveness of two arthroscopic surgery procedures for osteoarthritis of the knee (Moseley et al., 2002)<\/span>[footnote]Moseley, J. B., O\u2019Malley, K., Petersen, N. J., Menke, T. J., Brody, B. A., Kuykendall, D. H., \u2026 Wray, N. P. (2002). A controlled trial of arthroscopic surgery for osteoarthritis of the knee. The New England Journal of Medicine, 347<\/em>, 81\u201388.[\/footnote]. <\/span>The control participants in this study were prepped for surgery, received a tranquilizer, and even received three small incisions in their knees. But they did not receive the actual arthroscopic surgical procedure. The surprising result was that all participants improved in terms of both knee pain and function, and the sham surgery group improved just as much as the treatment groups. According to the researchers, \u201cThis study provides strong evidence that arthroscopic lavage with or without d\u00e9bridement [the surgical procedures used] is not better than and appears to be equivalent to a placebo procedure in improving knee pain and self-reported function\u201d (p. 85).<\/span><\/p>\n\n<\/div>\n

    Within-Subjects Experiments<\/span><\/h2>\n

    In a <\/span>within-subjects experiment<\/span><\/strong>, each participant is tested under all conditions. Consider an experiment on the effect of a defendant\u2019s physical attractiveness on judgments of his guilt. Again, in a between-subjects experiment, one group of participants would be shown an attractive defendant and asked to judge his guilt, and another group of participants would be shown an unattractive defendant and asked to judge his guilt. In a within-subjects experiment, however, the same group of participants would judge the guilt of both an attractive <\/span>and<\/span> an unattractive defendant.<\/span><\/p>\n

    The primary advantage of this approach is that it provides maximum control of extraneous participant variables. Participants in all conditions have the same mean IQ, same socioeconomic status, same number of siblings, and so on\u2014because they are the very same people. Within-subjects experiments also make it possible to use statistical procedures that remove the effect of these extraneous participant variables on the dependent variable and therefore make the data less \u201cnoisy\u201d and the effect of the independent variable easier to detect. We will look more closely at this idea later in the <\/span>book<\/span>.  However, not all experiments can use a within-subjects design nor would it be desirable to.<\/span><\/p>\n\n

    Carryover Effects and Counterbalancing<\/span><\/h2>\nThe primary disadvantage of within-subjects designs is that they can result in carryover effects. A <\/span>carryover effect<\/span><\/strong> is an effect of being tested in one condition on participants\u2019 behaviour in later conditions. One type of carryover effect is a <\/span>practice effect<\/span><\/strong>, where participants perform a task better in later conditions because they have had a chance to practice it. Another type is a <\/span>fatigue effect<\/span><\/strong>, where participants perform a task worse in later conditions because they become tired or bored. Being tested in one condition can also change how participants perceive stimuli or interpret their task in later conditions. This<\/span> type of effect <\/span> is called a <\/span>context effect<\/span><\/strong>. For example, an average-looking defendant might be judged more harshly when participants have just judged an attractive defendant than when they have just judged an unattractive defendant. Within-subjects experiments also make it easier for participants to guess the hypothesis. For example, a participant who is asked to judge the guilt of an attractive defendant and then is asked to judge the guilt of an unattractive defendant is likely to guess that the hypothesis is that defendant attractiveness affects judgments of guilt. This<\/span> knowledge <\/span> could lead the participant to judge the unattractive defendant more harshly because he thinks this is what he is expected to do. Or it could make participants judge the two defendants similarly in an effort to be \u201cfair.\u201d<\/span>\n\nCarryover effects can be interesting in their own right. (Does the attractiveness of one person depend on the attractiveness of other people that we have seen recently?) But when they are not the focus of the research, carryover effects can be problematic. Imagine, for example, that participants judge the guilt of an attractive defendant and then judge the guilt of an unattractive defendant. If they judge the unattractive defendant more harshly, this might be because of his unattractiveness. But it could be instead that they judge him more harshly because they are becoming bored or tired. In other words, the order of the conditions is a confounding variable. The attractive condition is always the first condition and the unattractive condition the second. Thus any difference between the conditions in terms of the dependent variable could be caused by the order of the conditions and not the independent variable itself.\n\nThere is a solution to the problem of order effects, however, that can be used in many situations. It is counterbalancing<\/b>, which means testing different participants in different orders. For example, some participants would be tested in the attractive defendant condition followed by the unattractive defendant condition, and others would be tested in the unattractive condition followed by the attractive condition. With three conditions, there would be six different orders (ABC, ACB, BAC, BCA, CAB, and CBA), so some participants would be tested in each of the six orders. With counterbalancing, participants are assigned to orders randomly, using the techniques we have already discussed. Thus random assignment plays an important role in within-subjects designs just as in between-subjects designs. Here, instead of randomly assigning to conditions, they are randomly assigned to different orders of conditions. In fact, it can safely be said that if a study does not involve random assignment in one form or another, it is not an experiment.\n\nAn efficient way of counterbalancing is through a Latin square design which randomizes through having equal rows and columns. For example, if you have four treatments, you must have four versions. Like a Sudoku puzzle, no treatment can repeat in a row or column. For four versions of four treatments, the Latin square design would look like:\n\n\n\n\n\n\n
    A<\/td>\nB<\/td>\nC<\/td>\nD<\/td>\n<\/tr>\n
    B<\/td>\nC<\/td>\nD<\/td>\nA<\/td>\n<\/tr>\n
    C<\/td>\nD<\/td>\nA<\/td>\nB<\/td>\n<\/tr>\n
    D<\/td>\nA<\/td>\nB<\/td>\nC<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThere are two ways to think about what counterbalancing accomplishes. One is that it controls the order of conditions so that it is no longer a confounding variable. Instead of the attractive condition always being first and the unattractive condition always being second, the attractive condition comes first for some participants and second for others. Likewise, the unattractive condition comes first for some participants and second for others. Thus any overall difference in the dependent variable between the two conditions cannot have been caused by the order of conditions. A second way to think about what counterbalancing accomplishes is that if there are carryover effects, it makes it possible to detect them. One can analyze the data separately for each order to see whether it had an effect.\n
    \n

    When 9 Is \u201cLarger\u201d Than 221<\/b><\/h4>\nResearcher Michael Birnbaum has argued that the <\/span>lack<\/span> of context provided by between-subjects designs is often a bigger problem than the context effects created by within-subjects designs. To demonstrate this problem, he asked participants to rate two numbers on how large they were on a scale of 1-to-10 where 1 was \u201cvery very small\u201d and 10 was \u201cvery very large\u201d.  One group of participants were asked to rate the number 9 and another group was asked to rate the number 221 (Birnbaum, 1999)[footnote]Birnbaum, M.H. (1999). How to show that 9>221: Collect judgments in a between-subjects design. Psychological Methods, 4<\/em>(3), 243-249.[\/footnote].<\/span> Participants in this between-subjects design gave the number 9 a mean rating of 5.13 and the number 221 a mean rating of 3.10. In other words, they rated 9 as larger than 221! According to Birnbaum, this<\/span> difference<\/span> is because participants spontaneously compared 9 with other one-digit numbers (in which case it is <\/span>relatively <\/span><\/em>large) and compared 221 with other three-digit numbers (in which case it is <\/span>relatively<\/span><\/em> small).<\/span>\n\n<\/div>\n

    Simultaneous Within-Subjects Designs<\/span><\/h2>\n

    So far, we have discussed an approach to within-subjects designs in which participants are tested in one condition at a time. There is another approach, however, that is often used when participants make multiple responses in each condition. Imagine, for example, that participants judge the guilt of 10 attractive defendants and 10 unattractive defendants. Instead of having people make judgments about all 10 defendants of one type followed by all 10 defendants of the other type, the researcher could present all 20 defendants in a sequence that mixed the two types. The researcher could then compute each participant\u2019s mean rating for each type of defendant. Or imagine an experiment designed to see whether people with social anxiety disorder remember negative adjectives (e.g., \u201cstupid,\u201d \u201cincompetent\u201d) better than positive ones (e.g., \u201chappy,\u201d \u201cproductive\u201d). The researcher could have participants study a single list that includes both kinds of words and then have them try to recall as many words as possible. The researcher could then count the number of each type of word that was recalled. There are many ways to determine the order in which the stimuli are presented, but one common way is to generate a different random order for each participant.<\/span><\/p>\n\n

    Between-Subjects or Within-Subjects?<\/span><\/h2>\n

    Almost every experiment can be conducted using either a between-subjects design or a within-subjects design. This possibility means that researchers must choose between the two approaches based on their relative merits for the particular situation.<\/span><\/p>\n

    Between-subjects experiments have the advantage of being conceptually simpler and requiring less testing time per participant. They also avoid carryover effects without the need for counterbalancing. Within-subjects experiments have the advantage of controlling extraneous participant variables, which generally reduces noise in the data and makes it easier to detect a relationship between the independent and dependent variables.<\/span><\/p>\n

    A good rule of thumb, then, is that if it is possible to conduct a within-subjects experiment (with proper counterbalancing) in the time that is available per participant\u2014and you have no serious concerns about carryover effects\u2014this design is probably the best option. If a within-subjects design would be difficult or impossible to carry out, then you should consider a between-subjects design instead. For example, if you were testing participants in a doctor\u2019s waiting room or shoppers in line at a grocery store, you might not have enough time to test each participant in all conditions and therefore would opt for a between-subjects design. Or imagine you were trying to reduce people\u2019s level of prejudice by having them interact with someone of another race. A within-subjects design with counterbalancing would require testing some participants in the treatment condition first and then in a control condition. But if the treatment works and reduces people\u2019s level of prejudice, then they would no longer be suitable for testing in the control condition. This difficulty is true for many designs that involve a treatment meant to produce long-term change in participants\u2019 behaviour (e.g., studies testing the effectiveness of psychotherapy). Clearly, a between-subjects design would be necessary here.<\/span><\/p>\nRemember also that using one type of design does not preclude using the other type in a different study. There is no reason that a researcher could not use both a between-subjects design and a within-subjects design to answer the same research question. In fact, professional researchers often take exactly this type of mixed methods approach.<\/span>\n

    \n

    Key Takeaways<\/h3>\n