{"id":2083,"date":"2019-10-25T19:18:49","date_gmt":"2019-10-25T23:18:49","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/simplestats\/?post_type=chapter&p=2083"},"modified":"2019-11-06T17:04:57","modified_gmt":"2019-11-06T22:04:57","slug":"8-4-level-of-significance-and-the-p-value","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/simplestats\/chapter\/8-4-level-of-significance-and-the-p-value\/","title":{"raw":"8.4 Level of Significance and the p-Value","rendered":"8.4 Level of Significance and the p-Value"},"content":{"raw":"[latexpage]\r\n\r\nThe concept\u00a0level of significance<\/em> is used to adjudicate whether the probability (of our results if the null hypothesis is true) is too high to dismiss the null hypothesis or low enough to allow us to reject the null hypothesis. In other words, the level of significance is what we use to proclaim results as statistically significant (when we reject the null hypothesis) or not statistically significant (when we fail to reject the null hypothesis).\r\n\r\n \r\n\r\nThink about it this way: recall that with confidence intervals we had selected 95% certainty and 99% certainty as meaningful levels of confidence. What is left is 5% and 1% \"uncertainty\", as it were, which we agree to tolerate. These 5% or 1% are distributed equally between the two tails of the normal distribution (2.5% on each side or 0.5% on each side, respectively). They also correspond to z<\/em>=1.96 and z<\/em>=2.58. Following the logic of Example 8.2 (A) from the preivious section, in order to reject a null hypothesis, we want the probability to be lower that these 5% or 1% (so that we can \"feel confident enough\").\r\n\r\n \r\n\r\nAnd this is exactly it: When we put it that way, saying that we want the probability (of the null hypothesis being true) -- called a p-value<\/em> -- to be less than 5%, we have essentially set the level of significance at 0.05. If we want the probability to be less than 1%, we have set the level of significance at 0.01. We can go even further: we might want to be extra cautious and to want\u00a0a \"confidence\" of 99.99%<\/span>, so that we want the probability to be less than 0.01% -- then we have set the level of significance at 0.001.\u00a0\u00a0<\/span>\r\n\r\n \r\n\r\nThese three numbers -- 0.05, 0.01, and 0.001 -- are the most commonly used levels of significance. The level of significance is denoted by the small-case Greek letter a<\/em>, i.e.,\u00a0\u03b1<\/em>, thus we usually choose one of the following:\r\n\r\n \r\n\r\n$\\alpha=0.05$\r\n\r\n$\\alpha=0.01$\r\n\r\n$\\alpha=0.001$\r\n\r\n \r\n\r\nYou can think of the significance level as the acceptable probability of being wrong<\/strong> -- and what is acceptable is left to the discretion of the researcher, subject to the purposes of the particular study.\r\n\r\n \r\n\r\nFollowing the logic presented in Example 8.2(A) then, if the probability of the result under the null hypothesis -- the p-value -- is smaller than a pre-selected significance level\u00a0\u03b1<\/em>, the null hypothesis is rejected and the result is considered statistically significant<\/strong>[footnote]Note the difference between \u03b1<\/em> and the p<\/em>-value. While \u03b1<\/em> indicates what probability of being wrong we are willing to tolerate, the actual p<\/em>-value we obtain is not<\/em> the probability of being wrong. The p<\/em>-value, again, is the probability of our result if the null hypothesis were true; in other words, if the null hypothesis is in fact true, and our p<\/em>-value is, say, 0.03, we'd obtain our results 3% of the time simply due to random sampling error. [\/footnote]. This is denoted in one of the following ways:\r\n\r\n \r\n\r\np \u2264 0.05\r\n\r\np \u2264 0.01\r\n\r\np \u2264 0.001[footnote]In published research you will find results marked by one asterisk, two asterisks, and three asterisks. These correspond to their significance based on the level used: \u03b1<\/em>=0.05, \u03b1<\/em>=0.01, and \u03b1<\/em>=0.001, respectively. The smaller the level of significance, the more strongly statistically significant the result is (i.e., most consider\u00a0\u03b1<\/em>=0.001 to indicate \"highly statistically significant\" results). (If you happen upon a dagger (\u2020), it indicates significance at \u03b1<\/em>=0.1 level, or 10% probability of being wrong, which most researchers consider too high, but some still use.[\/footnote]\r\n\r\n \r\n\r\nTo summarize, when a hypothesis is tested, we end up with an associated p<\/em>-value (again, the probability of the observed sample statistics if the null hypothesis is true). We compare the p<\/em>-value to the pre-selected significance level\u00a0\u03b1<\/em>: if p<\/em>\u00a0\u2264 \u03b1<\/em>, the results are statistically significant and therefore generalizable to the population.<\/strong>\r\n\r\n \r\n\r\nSo far so good? Good. However, unfortunately this isn't all (sorry!). What I have presented above is the most conventional treatment of how to use and interpret\u00a0p<\/em>-values. It is attractively straightforward\u00a0 -- but it's also arbitrary, and its true<\/em> interpretation is subject of an ongoing debate. As an introduction to the topic, I will leave it at that but you should be aware that there's more to the p<\/em>-value, and that its usage has been (rightfully) questioned and\/or challenged in recent years.[footnote] You can find plenty of information on the topic online; from journals banning the use of p<\/em>-values and hypothesis testing in favour of effect size (the Journal of Applied and Social Psychology<\/em>, see Trafimow & Marks, 2015 https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/01973533.2015.1012991), to calls to abandon statistical significance (e.g., McShane, Gal, Gelman, Robert & Tackett, 2019\u00a0https:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/00031305.2018.1527253), to others calling for its and p<\/em>-values' defense (e.g., Kuffner & Walker, 2016\u00a0https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/00031305.2016.1277161?src=recsys; Greenland, 2019 https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/00031305.2018.1529625?src=recsys). One thing is clear: p<\/em>-values and levels of significance have become increasingly controversial. Still, the American Statistical Association's position is that although caution against over-reliance on a single indicator is necessary, p<\/em>-values can still be used, alongside with other appropriate methods<\/em>: \"Researchers should recognize that a\u00a0<\/span>p<\/i>-value without context or other evidence provides limited information. For example, a\u00a0<\/span>p<\/i>-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large\u00a0<\/span>p<\/i>-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data. For these reasons, data analysis should not end with the calculation of a\u00a0<\/span>p<\/i>-value when other approaches are appropriate and feasible\" (Wasserstein & Lazar, 2016https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/00031305.2016.1154108?src=recsys). Finally, if you really want to not to overstate what the p<\/em>-value actually shows, see Greenland et al. (2016) for a of common misinterpretations and over-interpretations of the p<\/em>-value, of confidence intervals, and tests significance (here: https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4877414\/<\/a>). Because of its enormity, the topic is still conventionally taught as I presented it above (as it goes way beyond the scope of this book), at least at introductory level.<\/span>[\/footnote].\r\n\r\n \r\n\r\nGoing back to our example from the preivious section, let's see how the p<\/em>-values can change due to particular features of the study, like the sample size. Example 8.2(B) illustrates.\r\n\r\n \r\n
\r\n

Example 8.2(B) Employee Productivity (Finding Statistically Non-significant Results, N=25)<\/em><\/p>\r\n\r\n<\/header>\r\n

\r\n\r\n \r\n\r\nImagine that we had the same information as in Example 8.2(A), however, 25 employees took the training course instead of 100 and their average score was 620. The we have:\r\n\r\n \r\n\r\n$\\mu=600$\r\n\r\n$\\sigma=100$\r\n\r\n$\\overline{x}=620$\r\n\r\n$N=25$\r\n\r\n \r\n\r\nWe still want to know the probability of a score of 620 if the training course didn't contribute to the gain, i.e., the probability of a score of 620 under the condition of the null hypothesis<\/em>.<\/strong>\r\n