{"id":47,"date":"2018-04-16T21:55:07","date_gmt":"2018-04-17T01:55:07","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/discoverpsychology2\/?post_type=chapter&p=47"},"modified":"2018-04-26T18:25:50","modified_gmt":"2018-04-26T22:25:50","slug":"chapter-7","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/discoverpsychology2\/chapter\/chapter-7\/","title":{"raw":"The Replication Crisis in Psychology","rendered":"The Replication Crisis in Psychology"},"content":{"raw":"
By Edward Diener<\/a> and Robert Biswas-Diener<\/a>\r\n

University of Utah, University of Virginia, Portland State University<\/p>\r\n\r\n<\/header>

\r\n

In science, replication is the process of repeating research to determine the extent to which findings generalize across time and across situations. Recently, the science of psychology has come under criticism because a number of research findings do not replicate. In this module we discuss reasons for non-replication, the impact this phenomenon has on the field, and suggest solutions to the problem.<\/span><\/p>\r\n\r\n<\/section>

\u00a0<\/section>
\r\n
\r\n

Learning Objectives<\/h3>\r\n
    \r\n \t
  1. Define \u201creplication\u201d<\/li>\r\n \t
  2. Explain the difference between exact and conceptual replication<\/li>\r\n \t
  3. List 4 explanations for non-replication<\/li>\r\n \t
  4. Name 3 potential solutions to the replication crisis<\/li>\r\n<\/ol>\r\n<\/div>\r\n \r\n\r\n<\/section>
    \r\n

    \u00a0The Disturbing Problem<\/h1>\r\nIf you were driving down the road and you saw a pirate standing at an intersection you might not believe your eyes. But if you continued driving and saw a second, and then a third, you might become more confident in your observations. The more pirates you saw the less likely the first sighting would be a false positive (you were driving fast and the person was just wearing an unusual hat and billowy shirt) and the more likely it would be the result of a logical reason (there is a pirate themed conference in town). This somewhat absurd example is a real-life illustration of replication: the repeated findings of the same results.\r\n
    \"Four
    If you saw a pirate you might not believe it; but if you saw another one you would feel more confident in your observation. In science, this is the process of replication. [Image: Dave Hamster,https:\/\/goo.gl\/xg5QKi,<\/a> CC BY 2.0,https:\/\/goo.gl\/BRvSA7]<\/a><\/figcaption><\/figure>\r\nThe replication of findings is one of the defining hallmarks of science. Scientists must be able to replicate the results of studies or their findings do not become part of scientific knowledge. Replication protects against false positives (seeing a result that is not really there) and also increases confidence that the result actually exists. If you collect satisfaction data among homeless people living in Kolkata, India, for example, it might seem strange that they would report fairly high satisfaction with their food (which is exactly what we found inBiswas-Diener & Diener, 2001<\/a>). If you find the exact same result, but at a different<\/em> time, and with a different<\/em> sample of homeless people living in Kolkata, however, you can feel more confident that this result is true (as we did in Biswas-Diener & Diener, 2006<\/a>).\r\n\r\nIn modern times, the science of psychology is facing a crisis. It turns out that many studies in psychology\u2014including many highly cited studies\u2014do not replicate. In an era where news is instantaneous, the failure to replicate research raises important questions about the scientific process in general and psychology specifically. People have the right to know if they can trust research evidence. For our part, psychologists also have a vested interest in ensuring that our methods and findings are as trustworthy as possible.\r\n\r\nPsychology is not alone in coming up short on replication. There have been notable failures to replicate findings in other scientific fields as well. For instance, in 1989 scientists reported that they had produced \u201ccold fusion,\u201d achieving nuclear fusion at room temperatures. This could have been an enormous breakthrough in the advancement of clean energy. However, other scientists were unable to replicate the findings. Thus, the potentially important results did not become part of the scientific canon, and a new energy source did not materialize. In medical science as well, a number of findings have been found not to replicate\u2014which is of vital concern to all of society. The non-reproducibility of medical findings suggests that some treatments for illness could be ineffective. One example of non-replication has emerged in the study of genetics and diseases: when replications were attempted to determine whether certain gene-disease findings held up, only about 4% of the findings consistently did so.\r\n\r\nThe non-reproducibility of findings is disturbing because it suggests the possibility that the original research was done sloppily. Even worse is the suspicion that the research may have been falsified. In science, faking results is the<\/em> biggest<\/em> of sins, the unforgivable sin, and for this reason the field of psychology has been thrown into an uproar. However, as we will discuss, there are a number of explanations for non-replication, and not all are bad.\r\n

    What is Replication?<\/h1>\r\n
    \"Top
    Example of direct replication and conceptual replication of Asch's conformity experiment.<\/figcaption><\/figure>\r\nThere are different types of replication. First, there is a type called \u201cexact replication<\/a>\u201d (also called \"direct replication\"). In this form, a scientist attempts to exactly recreate the scientific methods used in\u00a0conditions of an earlier study to determine whether the results come out the same. If, for instance, you wanted to exactly replicate Asch\u2019s (1956<\/a>) classic findings on conformity, you would follow the original methodology: you would use only male participants, you would use groups of 8, and you would present the same stimuli (lines of differing lengths) in the same order. The second type of replication is called \u201cconceptual replication<\/a>.\u201d This occurs when\u2014instead of an exact replication, which reproduces the methods of the earlier study as closely as possible\u2014a scientist tries to confirm the previous findings using a different set of specific methods that test the same idea. The same hypothesis is tested, but using a different set of methods and measures. A conceptual replication of Asch\u2019s research might involve both male and female confederates<\/a> purposefully misidentifying types of fruit to investigate conformity\u2014rather than only males misidentifying line lengths.\r\n\r\nBoth exact and conceptual replications are important because they each tell us something new. Exact replications tell us whether the original findings are true, at least under the exact conditions tested. Conceptual replications help confirm whether the theoretical idea behind the findings is true, and under what conditions these findings will occur. In other words, conceptual replication offers insights into how generalizable the findings are.\r\n

    Enormity of the Current Crisis<\/h1>\r\n
    \"Percentage
    Table 1: The Reproducibility of Psychological Science<\/figcaption><\/figure>\r\nRecently, there has been growing concern as psychological research fails to replicate. To give you an idea of the extent of non-replicability of psychology findings, below are data reported in 2015 by the Open Science Collaboration project, led by University of Virginia psychologist Brian Nosek (Open Science Collaboration, 2015<\/a>). Because these findings were reported in the prestigious journal,Science,<\/em> they received widespread attention from the media. Here are the percentages of research that replicated\u2014selected from several highly prestigious journals:\r\n\r\nClearly, there is a very large problem when only about 1\/3 of the psychological studies in premier journals replicate! It appears that this problem is particularly pronounced for social psychology but even the 53% replication level of cognitive psychology is cause for concern.\r\n\r\nThe situation in psychology has grown so worrisome that the Nobel Prize-winning psychologist Daniel Kahneman called on social psychologists to clean up their act (Kahneman, 2012<\/a>). The Nobel laureate spoke bluntly of doubts about the integrity of psychology research, calling the current situation in the field a \u201cmess.\u201d His missive was pointed primarily at researchers who study social \u201cpriming,\u201d but in light of the non-replication results that have since come out, it might be more aptly directed at the behavioral sciences in general.\r\n

    Examples of Non-replications in Psychology<\/h1>\r\nA large number of scientists have attempted to replicate studies on what might be called \u201cmetaphorical priming,\u201d and more often than not these replications have failed. Priming<\/a> is the process by which a recent reference (often a subtle, subconscious cue) can increase the accessibility of a trait. For example, if your instructor says, \u201cPlease put aside your books, take out a clean sheet of paper, and write your name at the top,\u201d you might find your pulse quickening. Over time, you have learned that this cue means you are about to be given a pop quiz. This phrase primes all the features associated with pop quizzes: they are anxiety-provoking, they are tricky, your performance matters.\r\n
    \"A
    In one study, researchers enhanced test performance by priming participants with stereotypes of intelligence. But subsequent studies have not been able to replicate those results. [Image: CC0 Public Domain, https:\/\/goo.gl\/m25gce]<\/a><\/figcaption><\/figure>\r\nOne example of a priming study that, at least in some cases, does not replicate, is the priming of the idea of intelligence. In theory, it might be possible to prime people to actually become more intelligent (or perform better on tests, at least). For instance, in one study, priming students with the idea of a stereotypical professor versus soccer hooligans led participants in the \u201cprofessor\u201d condition to earn higher scores on a trivia game (Dijksterhuis & van Knippenberg, 1998<\/a>). Unfortunately, in several follow-up instances this finding has not replicated (Shanks et al, 2013<\/a>). This is unfortunate for all of us because it would be a very easy way to raise our test scores and general intelligence. If only it were true.\r\n\r\nAnother example of a finding that seems not to replicate consistently is the use of spatial distance cues to prime people\u2019s feelings of emotional closeness to their families (Williams & Bargh, 2008<\/a>). In this type of study, participants are asked to plot points on graph paper, either close together or far apart. The participants are then asked to rate how close they are to their family members. Although the original researchers found that people who plotted close-together points on graph paper reported being closer to their relatives, studies reported on PsychFileDrawer\u2014an internet repository of replication attempts\u2014suggest that the findings frequently do not replicate. Again, this is unfortunate because it would be a handy way to help people feel closer to their families.\r\n\r\nAs one can see from the examples, some of the studies that fail to replicate report extremely interesting findings\u2014even counterintuitive findings that appear to offer new insights into the human mind. Critics claim that psychologists have become too enamored with such newsworthy, surprising \u201cdiscoveries\u201d that receive a lot of media attention. Which raises the question of timing: might the current crisis of non-replication be related to the modern, media-hungry context in which psychological research (indeed, all research) is conducted? Put another way: is the non-replication crisis new?\r\n\r\nNobody has tried to systematically replicate studies from the past, so we do not know if published studies are becoming less replicable over time. In 1990<\/a>, however, Amir and Sharon were able to successfully replicate most of the main effects of six studies from another culture, though they did fail to replicate many of the interactions. This particular shortcoming in their overall replication may suggest that published studies are becoming less replicable over time, but we cannot be certain. What we can be sure of is that there is a significant problem with replication in psychology, and it\u2019s a trend the field needs to correct. Without replicable findings, nobody will be able to believe in scientific psychology.\r\n

    Reasons for Non-replication<\/h1>\r\nWhen findings do not replicate, the original scientists sometimes become indignant and defensive, offering reasons or excuses for non-replication of their findings\u2014including, at times, attacking those attempting the replication. They sometimes claim that the scientists attempting the replication are unskilled or unsophisticated, or do not have sufficient experience to replicate the findings. This, of course, might be true, and it is one possible reason for non-replication.\r\n\r\nOne reason for defensive responses is the unspoken implication that the original results might have been falsified. Faked results are only one reason studies may not replicate, but it is the most disturbing reason. We hope faking is rare, but in the past decade a number of shocking cases have turned up. Perhaps the most well-known come from social psychology. Diederik Stapel, a renowned social psychologist in the Netherlands, admitted to faking the results of a number of studies. Marc Hauser, a popular professor at Harvard, apparently faked results on morality and cognition. Karen Ruggiero at the University of Texas was also found to have\u00a0falsified<\/a> a number of her results (proving that bad behavior doesn\u2019t have a gender bias). Each of these psychologists\u2014and there are quite a few more examples\u2014was believed to have faked data. Subsequently, they all were disgraced and lost their jobs.\r\n\r\nAnother reason for non-replication is that, in studies with small sample sizes<\/a>, statistically-significant results may often be the result of chance. For example, if you ask five people if they believe that aliens from other planets visit Earth and regularly abduct humans, you may get three people who agree with this notion\u2014simply by chance. Their answers may, in fact, not be at all representative of the larger population. On the other hand, if you survey one thousand people, there is a higher probability that their belief in alien abductions reflects the actual attitudes of society. Now consider this scenario in the context of replication: if you try to replicate the first study\u2014the one in which you interviewed only five people\u2014there is only a small chance that you will randomly draw five new people with exactly the same (or similar) attitudes. It\u2019s far more likely that you will be able to replicate the findings using another large sample, because it is simply more likely that the findings are accurate.\r\n\r\nAnother reason for non-replication is that, while the findings in an original study may be true, they may only be true for some people in some circumstances and not necessarily universal or enduring. Imagine that a survey in the 1950s found a strong majority of respondents to have trust in government officials. Now imagine the same survey administered today, with vastly different results. This example of non-replication does not invalidate the original results. Rather, it suggests that attitudes have shifted over time.\r\n\r\nA final reason for non-replication relates to the quality of the replication rather than the quality of the original study. Non-replication might be the product of scientist-error, with the newer investigation not following the original procedures closely enough. Similarly, the attempted replication study might, itself, have too small a sample size or insufficient statistical power to find significant results.\r\n

    In Defense of Replication Attempts<\/h1>\r\nFailures in replication are not all bad and, in fact, some non-replication should be expected in science. Original studies are conducted when an answer to a question is uncertain. That is to say, scientists are venturing into new territory. In such cases we should expect some answers to be uncovered that will not pan out in the long run. Furthermore, we hope that scientists take on challenging new topics that come with some amount of risk. After all, if scientists were only to publish safe results that were easy to replicate, we might have very boring studies that do not advance our knowledge very quickly. But, with such risks, some non-replication of results is to be expected.\r\n
    \"A
    Researchers use specialized statistical software to store, analyze, and share data. Saving data over time and sharing data with others can be useful in conducting replications. [Image: Kwantlen Polytechnic University Psychology Department, CC BY 2.0, https:\/\/goo.gl\/BRvSA7]<\/figcaption><\/figure>\r\nA recent example of risk-taking can be seen in the research of social psychologist Daryl Bem. In 2011<\/a>, Bem published an article claiming he had found in a number of studies that future events could influence the past. His proposition turns the nature of time, which is assumed by virtually everyone except science fiction writers to run in one direction, on its head. Needless to say, attacks on Bem\u2019s article came fast and furious, including attacks on his statistics and methodology (Ritchie, Wiseman & French, 2012<\/a>). There were attempts at replication and most of them failed, but not all. A year after Bem\u2019s article came out, the prestigious journal where it was published, Journal of Personality and Social Psychology<\/em>, published another paper in which a scientist failed to replicate Bem\u2019s findings in a number of studies very similar to the originals (Galak, Lebeouf, Nelson & Simmons, 2012<\/a>).\r\n\r\nSome people viewed the publication of Bem\u2019s (2011) original study as a failure in the system of science. They argued that the paper should not have been published. But the editor and reviewers of the article had moved forward with publication because, although they might have thought the findings provocative and unlikely, they did not see obvious flaws in the methodology. We see the publication of the Bem paper, and the ensuing debate, as a strength of science. We are willing to consider unusual ideas if there is evidence to support them: we are open-minded. At the same time, we are critical and believe in replication. Scientists should be willing to consider unusual or risky hypotheses but ultimately allow good evidence to have the final say, not people\u2019s opinions.\r\n

    Solutions to the Problem<\/h1>\r\n

    Dissemination of Replication Attempts<\/h2>\r\n