{"id":6,"date":"2021-01-18T15:55:54","date_gmt":"2021-01-18T20:55:54","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/2021\/01\/18\/appendix\/"},"modified":"2025-09-17T19:46:31","modified_gmt":"2025-09-17T23:46:31","slug":"appendix","status":"publish","type":"back-matter","link":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/","title":{"raw":"Appendix: Fundamental Statistics","rendered":"Appendix: Fundamental Statistics"},"content":{"raw":"This appendix covers the fundamental statistical concepts necessary to critically appraise [pb_glossary id=\"704\"]randomized controlled trials (RCTs)[\/pb_glossary] and [pb_glossary id=\"1099\"]systematic reviews[\/pb_glossary]\/[pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary].\r\n<h1>P-Value Interpretation<\/h1>\r\nP-values are sometimes misinterpreted to mean \"the probability that the results occurred by chance\". This is problematic on at least two counts: the probability of any particular result occurring by chance will be extremely low, and also \"by chance\" requires further definition to be meaningful. A more technical definition is that the p-value is the probability of finding a result at least as extreme as the observed result if the [pb_glossary id=\"828\"]null hypothesis[\/pb_glossary] (usually \"no difference\") is correct and all assumptions used to compute the p-value are met.\r\n<div class=\"textbox shaded\"><em>E.g. #1 A [pb_glossary id=\"704\"]RCT[\/pb_glossary] finds a mean difference in pain of 2.3 on a 10-point scale between treatment A and treatment B (p=0.04). This means that, if there truly were no difference between treatment A and B, the probability of finding a mean difference of \u22652.3 by chance alone is 4%.<\/em><\/div>\r\n<div class=\"textbox shaded\"><em>E.g. #2 A [pb_glossary id=\"704\"]RCT[\/pb_glossary] of hydralazine-nitrate vs. placebo demonstrated a [pb_glossary id=\"109\"]relative risk reduction (RRR)[\/pb_glossary] of 10% for heart failure hospitalization with p=0.34. This means that, if there truly is no difference between hydralazine-nitrate vs. placebo, the probability of finding a RR [pb_glossary id=\"109\"]RRR[\/pb_glossary] of 10% or greater in heart failure hospitalization by chance alone is 34%.<\/em><\/div>\r\nErrors in p-value interpretation usually involve confusing the following two probabilities:\r\n<ul>\r\n \t<li>The probability that the treatment is ineffective given the observed evidence (the misinterpretation of a p-value)<\/li>\r\n \t<li>The probability of the observed evidence if the treatment were ineffective (what the p-value provides)\r\n<ul>\r\n \t<li>For more information on this common inference mistake (known as the \"The Prosecutor's Fallacy\") see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Westreich D et al.<\/a><\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\nBy convention, a p-value \u22640.05 is considered statistically significant, though this is increasingly recognized as an oversimplification and ignores consideration of clinical importance.\r\n\r\nThe most important takeaway from this discussion is that the typical understanding of a p-value is incorrect and such misunderstanding can lead to erroneous conclusions. For a further discussion of p-value misinterpretation in medical literature see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Price R et al<\/a>. For a more advanced discussion on why research findings are often false despite statistically significant results see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Ioannidis JPA<\/a>.\r\n<h1>Confidence Interval (CI) Interpretation<\/h1>\r\nThe technical definition of a 95% confidence interval (CI) is: If we were to repeat the study an infinite number of times, 95% of 95% CIs would contain the true effect, if all assumptions used to calculate the interval are correct. Consequently, a 95% CI does not entail \"there is a 95% chance the true value is within this range\" (a common misinterpretation). As with p-values, the true meaning is more nuanced. See <a href=\"https:\/\/seeing-theory.brown.edu\/frequentist-inference\/index.html#section2\" target=\"_blank\" rel=\"noopener\">here<\/a> for visual CI simulations as illustrative examples.\r\n\r\nThe 95% CI provides all of the information of a p-value (and is derived using the same information), but also adds information on a plausible range of the effect size. When interpreting CIs, it is important to examine both ends of the CI and judge whether there is a clinically important difference between them. For example, a point estimate of 5% [pb_glossary id=\"111\"]absolute risk reduction[\/pb_glossary] in stroke risk over 5 years with a 95% CI of 3% to 7% will include a narrow range that many clinicians would consider clinically important difference at both ends of the interval. This examination (along with considerations of [pb_glossary id=\"193\"]bias[\/pb_glossary]) will help establish the degree of uncertainty in the result. See <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\">McCormack J et al<\/a>. for a discussion of how considering only statistical significance without proper regard for CIs can cause confusion.\r\n\r\nBy convention, a 95% CI that does not include the null (e.g. a [pb_glossary id=\"119\"]relative effect[\/pb_glossary] 95% CI that includes 1.0 or an [pb_glossary id=\"111\"]absolute risk difference[\/pb_glossary] 95% CI that includes 0%) is considered statistically significant (i.e. consistent with p&lt;0.05).\r\n<h1>Sample Size Interpretation<\/h1>\r\nIt is sometimes believed that sample size (i.e. how many participants were included in the study) is a determinant of [pb_glossary id=\"105\"]internal validity[\/pb_glossary]. However, a\u00a0review (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Kjaergard LL et al.<\/a>) found that smaller trials (&lt;1,000 participants) only exaggerated treatment effects compared with larger trials (\u22651,000 participants) when they had inadequate randomization, [pb_glossary id=\"34\"]allocation concealment[\/pb_glossary], or blinding. As such, sample size itself is not indicative of [pb_glossary id=\"193\"]bias[\/pb_glossary]. Furthermore, if there were too few participants enrolled to detect a difference between groups this will be reflected in the corresponding wide CI (see <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/the-results\/\">Confidence Intervals \u2013 How precise were the estimates of treatment effect?<\/a> for a discussion of wide CIs and how they illustrate precision).\r\n<div class=\"textbox shaded\">\r\n\r\n<strong>Advanced discussion<\/strong> (included for completeness, but rarely applicable to appraisal)\r\n\r\nAn exception may be true for some \"very small\" trials - as parametric statistical tests rely on the central limit theorem and require a minimum sample size (e.g. n\u226530 is often suggested). See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Fagerland MW<\/a> for a more discussion. The details of this particular statistical concern are beyond the scope of this resource, but two simplified takeaways are that:\r\n<ul>\r\n \t<li>\"Very small\" sample size may be a concern for the proper use of certain statistical methods when measuring continuous outcomes<\/li>\r\n \t<li>A minimum sample of 30 is an arbitrary rule-of-thumb to prevent this - nonetheless 30 does provide an approximation of the sample sizes where this may be a concern (e.g. this will almost certainly not be a concern for a trial with several hundreds of participants)<\/li>\r\n<\/ul>\r\n<\/div>\r\n<h1>Absolute Risk Differences and Relative Measures of Effect<\/h1>\r\n<h2><strong>Absolute Risk Difference<\/strong><\/h2>\r\nThe [pb_glossary id=\"111\"]absolute risk difference[\/pb_glossary] between groups refers to the risk of an event in one group minus the risk in another group. Consider the following example of a theoretical 2 year trial examining insomnia rates:\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 36px\" border=\"0\"><caption>Table 16. Absolute risk difference example.<\/caption>\r\n<tbody>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\"><strong>Outcome<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Intervention Group<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Comparator Group<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Absolute Difference<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Duration<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\">Insomnia<\/td>\r\n<td style=\"width: 20%;height: 18px\">15%<\/td>\r\n<td style=\"width: 20%;height: 18px\">5%<\/td>\r\n<td style=\"width: 20%;height: 18px\">+10%<\/td>\r\n<td style=\"width: 20%;height: 18px\">2 years<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nIn this case, the [pb_glossary id=\"111\"]absolute difference[\/pb_glossary] was calculated by subtracting the intervention group event rate (15%) by the comparator group event rate (5%), which equals 10% (15% - 5%). This difference is \"absolute\" because the number (e.g. +10% risk of insomnia over 2 years) is independently meaningful.\r\n\r\n[pb_glossary id=\"111\"]Absolute differences[\/pb_glossary] also need to be communicated in the context of time. For example, a 1% [pb_glossary id=\"111\"]absolute risk reduction[\/pb_glossary] over 1 month is quite different from a 1% [pb_glossary id=\"111\"]absolute risk reduction[\/pb_glossary] over 10 years. As such, [pb_glossary id=\"111\"]absolute differences[\/pb_glossary] should be stated as a __% increase\/decrease over [timeframe].\r\n<h2><strong>Relative Measures of Effect<\/strong><\/h2>\r\nThis contrasts with [pb_glossary id=\"119\"]relative effect[\/pb_glossary] measures. One example of a [pb_glossary id=\"119\"]relative effect[\/pb_glossary] is [pb_glossary id=\"109\"]relative risk (RR)[\/pb_glossary], which is calculated by dividing the risk of event in the intervention group by that in the comparator group.\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\" border=\"0\"><caption>Table 17. Relative effect example.<\/caption>\r\n<tbody>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\"><strong>Outcome<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Intervention Group<\/strong><\/td>\r\n<td style=\"width: 20%;height: 18px\"><strong>Comparator Group<\/strong><\/td>\r\n<td style=\"width: 20%\"><strong>[pb_glossary id=\"109\"]Relative Risk (RR)[\/pb_glossary]<\/strong><\/td>\r\n<td style=\"width: 20%\"><strong>Duration<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\">Insomnia<\/td>\r\n<td style=\"width: 20%;height: 18px\">15%<\/td>\r\n<td style=\"width: 20%;height: 18px\">5%<\/td>\r\n<td style=\"width: 20%;height: 18px\">3.0<\/td>\r\n<td style=\"width: 20%\">3 months<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nIn this case, the [pb_glossary id=\"109\"]RR[\/pb_glossary] was calculated by dividing the intervention group rate (15%) by the comparator group event rate (5%), which equals 3.0 (15% \u00f7 5%). <span style=\"text-align: initial;font-size: 14pt\">This difference is \"relative\" because the number (e.g. a 3.0 [pb_glossary id=\"109\"]RR[\/pb_glossary] of experiencing insomnia) is dependent on the risk in the comparator group to be meaningful. <\/span><span style=\"font-size: 14pt;text-align: initial\">[pb_glossary id=\"109\"]RR[\/pb_glossary] 3.0 means that the risk has tripled, but without knowing the baseline risk that is being tripled, then the number is not fully interpretable.<\/span>\r\n\r\nThis dependence can be problematic if not properly considered. Consider the following example, where the [pb_glossary id=\"109\"]RR[\/pb_glossary] is identical in both cases:\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\" border=\"0\"><caption>Table 18. Relative effect dependency on baseline risk example.<\/caption>\r\n<tbody>\r\n<tr>\r\n<td style=\"width: 20%\"><strong>[pb_glossary id=\"109\"]Relative Risk (RR)[\/pb_glossary]<\/strong><\/td>\r\n<td style=\"width: 20%\">\u00a0<strong>Baseline Risk<\/strong><\/td>\r\n<td style=\"width: 20%;text-align: left\"><strong>Risk on Treatment\r\n([pb_glossary id=\"109\"]RR[\/pb_glossary] x Baseline Risk)<\/strong><\/td>\r\n<td style=\"width: 20%;text-align: left\"><strong>[pb_glossary id=\"111\"]Absolute Risk Difference[\/pb_glossary]<\/strong><\/td>\r\n<td style=\"width: 20%\"><strong>Duration<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\">0.5<\/td>\r\n<td style=\"width: 20%;height: 18px\">30%<\/td>\r\n<td style=\"width: 20%;height: 18px\">15%<\/td>\r\n<td style=\"width: 20%;height: 18px\">15%<\/td>\r\n<td style=\"width: 20%\">10 years<\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 20%;height: 18px\">0.5<\/td>\r\n<td style=\"width: 20%;height: 18px\">2%<\/td>\r\n<td style=\"width: 20%;height: 18px\">1%<\/td>\r\n<td style=\"width: 20%;height: 18px\">1%<\/td>\r\n<td style=\"width: 20%\">10 years<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nAs demonstrated, [pb_glossary id=\"109\"]RR[\/pb_glossary] considered in isolation lacks crucial information. The same concept is relevant when (responsibly) buying a product during a sale. Knowing that a particular product is 50% off is not sufficient for a rational choice, as there needs to also be information about the original price (e.g. $20 versus $20,000) before deciding if the purchase is desirable.\r\n\r\nNote: [pb_glossary id=\"109\"]RR[\/pb_glossary] is just one relative measure - see discussion below for information on [pb_glossary id=\"109\"]relative risks[\/pb_glossary], [pb_glossary id=\"103\"]odds ratios[\/pb_glossary], and [pb_glossary id=\"110\"]hazard ratios[\/pb_glossary].\r\n<h1>Number Needed to Treat or Harm<\/h1>\r\nBoth number needed to treat (NNT) and number needed to harm (NNH) are measures of how many patients have to receive the treatment of interest for one additional person to experience the outcome of interest (NNT being for beneficial outcomes, and NNH for harmful outcomes).\r\n\r\nIt is calculated as: 100 \u00f7 [pb_glossary id=\"111\"]Absolute risk difference[\/pb_glossary], with the result always rounded up.\r\n\r\nFor example, if a treatment has a 7% [pb_glossary id=\"111\"]absolute risk increase[\/pb_glossary] of causing urinary retention over 3 months then the NNH is 15 (100 \u00f7 7 = 14.3, then round up to 15). This means that 15 patients will have to be treated for one of them to have urinary retention over the next 3 months (always including timeframe, as with [pb_glossary id=\"111\"]absolute risk differences[\/pb_glossary]).\r\n\r\nThis is an alternative way to understand [pb_glossary id=\"111\"]absolute risk differences[\/pb_glossary] that may be more intuitive to some (though it is more poorly understood by patients than other measures, as discussed <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/the-results\/\" target=\"_blank\" rel=\"noopener\">here<\/a>).\r\n<h1>Relative Risk, Odds Ratios, and Hazard Ratios<\/h1>\r\nBefore delving into the details of each type of [pb_glossary id=\"119\"]relative effect[\/pb_glossary] it should be noted that all of them have the following features:\r\nAny relative measure = 1.0 means there was no difference between groups\r\nAny relative measure &gt; 1.0 means the outcome was more likely with the intervention than the comparator\r\nAny relative measure &lt; 1.0 means the outcome was less likely with the intervention than the comparator\r\n\r\nTo demonstrate the differences between these measures of [pb_glossary id=\"119\"]relative effect[\/pb_glossary], consider the following table:\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 61.0469%;height: 54px\" border=\"0\"><caption>Table 19. Example 2x2 chart of aspirin vs. placebo for stroke prevention.<\/caption>\r\n<tbody>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 16.4884%;height: 18px\"><\/td>\r\n<td style=\"width: 19.9431%;height: 18px\"><strong>Stroke<\/strong><\/td>\r\n<td style=\"width: 19.4498%;height: 18px\"><strong>No stroke<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 16.4884%;height: 18px\"><strong>Aspirin<\/strong><\/td>\r\n<td style=\"width: 19.9431%;height: 18px\">10 (A)<\/td>\r\n<td style=\"width: 19.4498%;height: 18px\">90 (B)<\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 16.4884%;height: 18px\"><strong>Placebo<\/strong><\/td>\r\n<td style=\"width: 19.9431%;height: 18px\">20 (C)<\/td>\r\n<td style=\"width: 19.4498%;height: 18px\">80 (C)<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h2><strong>Relative Risk (RR)<\/strong><\/h2>\r\nCalculating the [pb_glossary id=\"109\"]RR[\/pb_glossary] consists of dividing the risk of event in the aspirin group by risk of event in the placebo group.\r\n\r\nUsing the above table:\r\nThe risk of event in the treatment group: A \u00f7 (A+B)\r\nThe risk of event in the comparator group: C \u00f7 (C+D)\r\nWith numbers imputed: 10 \u00f7 100 = 0.1 (or 10%) in the aspirin group and 20 \u00f7 100 = 0.2 (or 20%) in the placebo group.\r\nThe [pb_glossary id=\"109\"]RR[\/pb_glossary] is then 0.1 \u00f7 0.2 = 0.5.\r\n<h2><strong>Odds Ratio (OR)<\/strong><\/h2>\r\nCalculating the [pb_glossary id=\"103\"]OR[\/pb_glossary] consists of dividing the odds of an event in the aspirin group by the odds of an event in the placebo group.\r\n\r\nUsing the above table:\r\nThe odds of event in intervention group: A \u00f7 B\r\nThe odds of event in the comparator group: C \u00f7 D\r\nWith numbers imputed: 10 \u00f7 90 = 0.11 in the aspirin group and 20 \u00f7 80 = 0.25 in the placebo group.\r\nThe [pb_glossary id=\"103\"]OR[\/pb_glossary] is then 0.11 \u00f7 0.25 = 0.44.\r\n\r\n[pb_glossary id=\"103\"]OR[\/pb_glossary] are similar to [pb_glossary id=\"109\"]RR[\/pb_glossary] when events are rare (A \u00f7 (A+B) \u2248 A \u00f7 B when A is very small) (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Holcomb WL et al.<\/a>). As events become more common, these measures diverge and [pb_glossary id=\"103\"]ORs[\/pb_glossary] will overestimate [pb_glossary id=\"109\"]RRs[\/pb_glossary] (such as in this example where RR=0.5 and OR=0.44). The <a href=\"https:\/\/clincalc.com\/Stats\/ConvertOR.aspx\" target=\"_blank\" rel=\"noopener\">ClinCalc tool<\/a> can be used to convert [pb_glossary id=\"103\"]OR[\/pb_glossary] to [pb_glossary id=\"109\"]RR[\/pb_glossary].\r\n<h2><strong>Hazard Ratio (HR)<\/strong><\/h2>\r\n[pb_glossary id=\"110\"]Hazard ratios (HRs)[\/pb_glossary] represent the average of the instantaneous incidence rate at every point during a trial. Consider an example of a 5-year trial that has a [pb_glossary id=\"110\"]HR[\/pb_glossary] of 0.70 for the outcome of death comparing an intervention against some comparator. This means that a participant assigned to intervention will be 30% less likely to die relative to the comparator at any point during the trial:\r\n<ul>\r\n \t<li>Year 1: If 5% have died in the comparator group, then 3.5% are expected to have died in the intervention group (5% * 0.70 = 3.5%)<\/li>\r\n \t<li>Year 2: If 10% have died in the comparator group, then 7% are expected to have died in the intervention group (10% * 0.70 = 7%)<\/li>\r\n \t<li>Year 5: If 20% have died in the comparator group, 14% are expected to have died in the intervention group (20% * 0.70 = 14%)<\/li>\r\n<\/ul>\r\nThe same is approximately true at any given timepoint during the trial follow-up. These are all approximations as the [pb_glossary id=\"110\"]HR[\/pb_glossary] is an average, it (almost certainly) will not be exactly true at every time point. For instance, the final [pb_glossary id=\"110\"]HR[\/pb_glossary] might be 0.70, but it could be 0.80 during the first half of the trial and 0.60 during the latter half.\r\n\r\nFor example, consider the unadjusted analysis from an observational study (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turgeon RD, Koshman SL, et al.<\/a>) that compared the use of ticagrelor vs. clopidogrel in patients who had undergone percutaneous coronary intervention following acute coronary syndrome (ACS). For the outcome of survival without major adverse coronary events (MACE) the [pb_glossary id=\"110\"]HR[\/pb_glossary] was 0.84 before adjustment for potential confounding variables, depicted visually below:\r\n<div class=\"textbox\">\r\n\r\n<img class=\"alignnone size-full wp-image-1452\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1.jpg\" alt=\"\" width=\"760\" height=\"420\" \/>\r\n\r\nGraph 3. Kaplan Meier curve of survival without major adverse coronary events\r\n\r\n<\/div>\r\nWhile not exactly true at every time point, past the first 100 days the cumulative proportion patients experiencing death or MACE in the ticagrelor group appears to be roughly 84% of the cumulative proportion in the clopidogrel group fairly consistently (see \"Kaplan Meier Curves\" for more information below on how to interpret these types of graphs). This coheres with the [pb_glossary id=\"110\"]HR[\/pb_glossary] of 0.84 discussed above.\r\n\r\n<span style=\"text-align: initial\"><span style=\"font-size: 1em\">[pb_glossary id=\"110\"]HRs[\/pb_glossary] are usually similar to [pb_glossary id=\"109\"]RRs[\/pb_glossary] (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Sutradhar R et al.<\/a>). [pb_glossary id=\"110\"]HRs[\/pb_glossary] examine multiple timepoints over trial follow-up, whereas [pb_glossary id=\"109\"]RRs[\/pb_glossary] evaluate cumulative proportions at the end of the trial (or at another single timepoint). <\/span><\/span><span style=\"text-align: initial\"><span style=\"font-size: 1em\">[pb_glossary id=\"110\"]HRs[\/pb_glossary] can account for differential follow-up times, and contain more information than [pb_glossary id=\"109\"]RRs[\/pb_glossary]\/[pb_glossary id=\"103\"]ORs[\/pb_glossary] since they include the added dimension of time (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Guyatt G et al.<\/a>). [pb_glossary id=\"110\"]HRs[\/pb_glossary] are limited in their ability to convey fluctuations in effect over time, as a [pb_glossary id=\"110\"]HR[\/pb_glossary] of 1.0 could mean that there was consistently no effect, or it could mean that there was beneficial effect during the first half and a proportional detrimental effect during the latter half (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Hern\u00e1n MA<\/a>). However, some of these limitations can be overcome by combining a [pb_glossary id=\"110\"]HR[\/pb_glossary] with the use of a Kaplan Meier curve, as discussed below.<\/span><\/span>\r\n<h1>Kaplan Meier Curves<\/h1>\r\n<h2><strong>Cumulative Hazards<\/strong><\/h2>\r\nKaplan Meier curves are graphical representations comparing event accrual between two groups over time. Consider another example from the aforementioned study comparing ticagrelor vs. clopidogrel:\r\n<div class=\"textbox\">\r\n\r\n<img class=\"alignnone size-full wp-image-1545\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final.png\" alt=\"\" width=\"1341\" height=\"777\" \/>\r\nGraph 4. Kaplan Meier curve of survival without acute coronary syndrome (ACS)\r\n\r\n<\/div>\r\nEach curve displays the cumulative proportion of patients in that group who have experienced the outcome of interest. As time passes more participants experience the outcome and the curve progresses downwards. The [pb_glossary id=\"119\"]relative differences[\/pb_glossary] in outcome accumulation can be expressed as a [pb_glossary id=\"110\"]HR[\/pb_glossary], as discussed above. Note that, while not displayed, each curve has a <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">CI<\/a> surrounding it at every time point.\r\n<h2><strong>Onset of Benefit<\/strong><\/h2>\r\nConsider the above Kaplan Meier curve. During the first 50 days, the curves for the two groups overlap. However, after this point the two curves begin to separate. This curve provides insight into the onset of benefit of the intervention and if the benefit is sustained over time. In this case, onset of benefit begins after approximately 50 days and is sustained as time elapses.\r\n<h2><strong>Course of Condition<\/strong><\/h2>\r\nThe graph also gives insight into event rates over time. As can be seen above, the curve is steepest initially - indicating that the risk of death or ACS is highest immediately following the intervention. The slope of the curve then flattens and remains relatively stable - indicating the event rate after the initial period is relatively constant during the first year. This demonstrates how Kaplan Meier curves can be useful to understand the course of a condition over time.\r\n<h2><strong>Total at Risk (or Number at Risk)<\/strong><\/h2>\r\nConsider another curve from the same study, this one examining survival without major bleeds:\r\n<div class=\"textbox\">\r\n\r\n<img class=\"alignnone size-full wp-image-1454\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1.jpg\" alt=\"\" width=\"848\" height=\"472\" \/>\r\n\r\nGraph 5. Kaplan Meier curve of survival without major bleeding\r\n\r\n<\/div>\r\nAs depicted, sometimes there is also a \"Total at risk\" (sometimes called \"Number at risk\") table beneath the curve. This can give additional information regarding the participants as they progressed through the trial. All participants begin \"at risk\", but as the study progresses the number decreases. Below are the following reasons the number may decrease, as well as possible implications if the decreases are not balanced between groups:\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 75px\" border=\"0\"><caption>Table 20. Reasons and implications of total at risk decreases.<\/caption>\r\n<tbody>\r\n<tr style=\"height: 15px\">\r\n<td style=\"width: 22.0812%;height: 15px\"><strong>Reasons for total at risk decreasing<\/strong><\/td>\r\n<td style=\"width: 77.9188%;height: 15px\"><strong>Implications if imbalanced between groups<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 15px\">\r\n<td style=\"width: 22.0812%;height: 15px\">Outcome of interest occurred<\/td>\r\n<td style=\"width: 77.9188%;height: 15px\">If there is a difference in effect between the intervention and comparator, then the total at risk may decrease more quickly in one group. This is evidence of effect, not [pb_glossary id=\"193\"]bias[\/pb_glossary].<\/td>\r\n<\/tr>\r\n<tr style=\"height: 15px\">\r\n<td style=\"width: 22.0812%;height: 15px\">Death<\/td>\r\n<td style=\"width: 77.9188%;height: 15px\">If there are differences in mortality rates then this should prompt consideration of the relative safety of the comparators, as well as consideration of death as a competing event within the analysis.<\/td>\r\n<\/tr>\r\n<tr style=\"height: 15px\">\r\n<td style=\"width: 22.0812%;height: 15px\">[pb_glossary id=\"121\"]Loss to follow-up[\/pb_glossary]<\/td>\r\n<td style=\"width: 77.9188%;height: 15px\">This could result in systematic [pb_glossary id=\"193\"]bias[\/pb_glossary] if the reasons for loss to follow-up are not random (for more see the discussion on [pb_glossary id=\"121\"]loss to follow-up[\/pb_glossary] <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/chapter-1\/\" target=\"_blank\" rel=\"noopener\">here<\/a>).<\/td>\r\n<\/tr>\r\n<tr style=\"height: 15px\">\r\n<td style=\"width: 22.0812%;height: 15px\">The study ended before the participant had outcome data at that time point<\/td>\r\n<td style=\"width: 77.9188%;height: 15px\">If by chance there is a difference in how many patients were enrolled early in one group (and thus had more time to accrue events) this could [pb_glossary id=\"193\"]bias[\/pb_glossary] a [pb_glossary id=\"109\"]RR[\/pb_glossary] or [pb_glossary id=\"103\"]OR[\/pb_glossary]. For example, by chance one group might have patients enrolled for an average of 4 years and another group had patients enrolled for an average of 5 years. However, since the [pb_glossary id=\"110\"]HR[\/pb_glossary] incorporates the timing of events, this should not result in [pb_glossary id=\"193\"]bias[\/pb_glossary].<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nThus the total at risk table can serve as a clue that further examination should be undertaken to see if there is [pb_glossary id=\"193\"]bias[\/pb_glossary].\r\n<h1>Forest Plots<\/h1>\r\nForest plots are used in [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary] to graphically depict the effects of an intervention across multiple studies. Consider the labeled example below from the \"Beta-blockers for hypertension\" Cochrane Review (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Wiysonge CS et al.<\/a>):\r\n\r\n<img class=\"wp-image-50 size-full\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1552\/2021\/11\/Forest-plot-new-new.png\" alt=\"\" width=\"1604\" height=\"466\" data-wp-editing=\"1\" \/>Plot 4. Forest plot of beta-blockers vs. placebo in patients with hypertension for the outcome of mortality.\r\n\r\nAs showcased, forest plots show information about each individual study included for that outcome, and also the combined results. This information is displayed visually as well as numerically. Trials with more events or participants are generally given greater weight. [pb_glossary id=\"107\"]Heterogeneity[\/pb_glossary] is typically measured via I<sup>2<\/sup>, which is 0% in this case. For more information on [pb_glossary id=\"107\"]heterogeneity[\/pb_glossary] see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/results-of-the-meta-analysis-i-e-what-do-the-pooled-results-of-the-trials-show\/\" target=\"_blank\" rel=\"noopener\">here<\/a>.\r\n<h1>Standardized Mean Difference Interpretation<\/h1>\r\nThe [pb_glossary id=\"113\"]standardized mean difference (SMD)[\/pb_glossary] is a method of combining multiple continuous outcome scoring systems into one measurement. For example, when performing a meta-analysis on the effects of antidepressants on depression symptom reduction, trials may use different scales to rate depression symptoms (HAM-D, PHQ-9, etc.). [pb_glossary id=\"113\"]SMD[\/pb_glossary] will allow the aggregation of the results of all these studies. Notably, using [pb_glossary id=\"113\"]SMD[\/pb_glossary] assumes that differences between studies are due to differences in scales (and not in intervention\/population characteristics). Arbitrary \"rule-of-thumb\" cutoffs (e.g. [pb_glossary id=\"113\"]SMD[\/pb_glossary] of 0.2 = \"small effect\") may not reflect the minimal important difference.\r\n\r\nAn alternative approach is to transform the [pb_glossary id=\"113\"]SMD[\/pb_glossary] into a more familiar scale (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Higgins JPT et al.<\/a>). Multiply the [pb_glossary id=\"113\"]SMD[\/pb_glossary] by the standard deviation (SD) of the largest trial to convert to its scale.\r\n<div class=\"textbox shaded\">\r\n\r\n<em>E.g. These are the results of a [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] which assesses the effects of IV iron on health-related quality of life at 6 months in patients with HFrEF (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\">Turgeon RD, Barry AR, et al.<\/a>):<\/em>\r\n\r\n<em><img class=\"alignnone wp-image-51\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1552\/2021\/11\/SMD-Example-1.png\" alt=\"\" width=\"988\" height=\"317\" \/><\/em>\r\n\r\n<em>Plot 5. Forest plot of IV iron vs. placebo in patients with heart failure with reduced ejection fraction on health-related quality of life.<\/em>\r\n\r\n<em><strong>Step 1:<\/strong> Identify the trial with the most weight, FAIR-HF in this case.<\/em>\r\n\r\n<em><strong>Step 2:<\/strong> Calculate average SD. Average SD in this case is ~20.50 (the average of 16.9115 and 24.0832).\u00a0<\/em>\r\n\r\n<em><strong>Step 3:<\/strong> Multiply average SD by [pb_glossary id=\"113\"]SMD[\/pb_glossary]. In this case this gives 10.66 (20.50 * 0.52).\u00a0<\/em>\r\n\r\n<em><strong>Step 4:<\/strong> Contextualize this result in the scale used in the trial. In this case, that would be equal to a 10.66 out of 100 improvement at 6 months (per the scale used in FAIR-HF).\u00a0<\/em>\r\n\r\n<em><strong>Step 5:<\/strong> Compare this value to the [pb_glossary id=\"78\"]minimally important difference (MID)[\/pb_glossary] if known. In the case of the FAIR-HF scale the [pb_glossary id=\"78\"]MID[\/pb_glossary] was 5. Therefore the mean effect was greater than the [pb_glossary id=\"78\"]MID[\/pb_glossary].<\/em>\r\n\r\n<\/div>\r\nIdeally the review will present this information along with a comparison of the proportion of participants within each group who experienced clinically important improvement or decline (the so-called \"responder analysis\"). This is useful because calculating the mean alone will not provide any information about the distribution. The responder analysis unfortunately cannot typically be calculated by readers if it is not already reported by the reviewers.\r\n<h1>Statistical Significance Is Not Everything<\/h1>\r\nWhile this section has focused on statistical fundamentals it is important to emphasize that statistics are only one aspect of critical appraisal. Even if a study shows statistical significance, there needs to be considerations of [pb_glossary id=\"193\"]bias[\/pb_glossary], clinical significance, and [pb_glossary id=\"1653\"]generalizability[\/pb_glossary].\r\n\r\n<strong>Bias<\/strong>: P-values and CIs are contingent on all the assumptions being used to calculate them being correct. In other words, they assume there is absolutely no [pb_glossary id=\"193\"]bias[\/pb_glossary] present. As such, if the study is poorly conducted (e.g. a [pb_glossary id=\"704\"]RCT[\/pb_glossary] without adequate [pb_glossary id=\"34\"]allocation concealment[\/pb_glossary] and blinding) this will not be reflected in the statistical analysis of the results. This is why it is necessary to appraise the conduct of the trial to evaluate the credibility of the results.\r\n\r\n<strong>Clinical significance<\/strong>: Even if a result is statistically significant, it may be too small of an effect to matter to a patient. For instance, with enough participants, a 1-point reduction in pain on a 100-point scale could be statistically significant, but very unlikely to be felt by an individual patient.\r\n\r\n<strong>Generalizability<\/strong>: Even if the results are unbiased and clinically significant, they will only be useful if they can be applied to practice. If there are substantial differences between the features of the trial and your own practice, then the result may not be applicable.\r\n\r\nOther sections of this resource will go into these concepts in more depth, but these are the fundamental reasons why a comprehensive approach to appraisal is necessary, and simply looking at statistical significance in the results section will not be sufficient to understand the clinical implications of a trial.","rendered":"<p>This appendix covers the fundamental statistical concepts necessary to critically appraise <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_704\">randomized controlled trials (RCTs)<\/a> and <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_1099\">systematic reviews<\/a>\/<a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_1101\">meta-analyses<\/a>.<\/p>\n<h1>P-Value Interpretation<\/h1>\n<p>P-values are sometimes misinterpreted to mean &#8220;the probability that the results occurred by chance&#8221;. This is problematic on at least two counts: the probability of any particular result occurring by chance will be extremely low, and also &#8220;by chance&#8221; requires further definition to be meaningful. A more technical definition is that the p-value is the probability of finding a result at least as extreme as the observed result if the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_828\">null hypothesis<\/a> (usually &#8220;no difference&#8221;) is correct and all assumptions used to compute the p-value are met.<\/p>\n<div class=\"textbox shaded\"><em>E.g. #1 A <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_704\">RCT<\/a> finds a mean difference in pain of 2.3 on a 10-point scale between treatment A and treatment B (p=0.04). This means that, if there truly were no difference between treatment A and B, the probability of finding a mean difference of \u22652.3 by chance alone is 4%.<\/em><\/div>\n<div class=\"textbox shaded\"><em>E.g. #2 A <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_704\">RCT<\/a> of hydralazine-nitrate vs. placebo demonstrated a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">relative risk reduction (RRR)<\/a> of 10% for heart failure hospitalization with p=0.34. This means that, if there truly is no difference between hydralazine-nitrate vs. placebo, the probability of finding a RR <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RRR<\/a> of 10% or greater in heart failure hospitalization by chance alone is 34%.<\/em><\/div>\n<p>Errors in p-value interpretation usually involve confusing the following two probabilities:<\/p>\n<ul>\n<li>The probability that the treatment is ineffective given the observed evidence (the misinterpretation of a p-value)<\/li>\n<li>The probability of the observed evidence if the treatment were ineffective (what the p-value provides)\n<ul>\n<li>For more information on this common inference mistake (known as the &#8220;The Prosecutor&#8217;s Fallacy&#8221;) see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Westreich D et al.<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>By convention, a p-value \u22640.05 is considered statistically significant, though this is increasingly recognized as an oversimplification and ignores consideration of clinical importance.<\/p>\n<p>The most important takeaway from this discussion is that the typical understanding of a p-value is incorrect and such misunderstanding can lead to erroneous conclusions. For a further discussion of p-value misinterpretation in medical literature see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Price R et al<\/a>. For a more advanced discussion on why research findings are often false despite statistically significant results see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Ioannidis JPA<\/a>.<\/p>\n<h1>Confidence Interval (CI) Interpretation<\/h1>\n<p>The technical definition of a 95% confidence interval (CI) is: If we were to repeat the study an infinite number of times, 95% of 95% CIs would contain the true effect, if all assumptions used to calculate the interval are correct. Consequently, a 95% CI does not entail &#8220;there is a 95% chance the true value is within this range&#8221; (a common misinterpretation). As with p-values, the true meaning is more nuanced. See <a href=\"https:\/\/seeing-theory.brown.edu\/frequentist-inference\/index.html#section2\" target=\"_blank\" rel=\"noopener\">here<\/a> for visual CI simulations as illustrative examples.<\/p>\n<p>The 95% CI provides all of the information of a p-value (and is derived using the same information), but also adds information on a plausible range of the effect size. When interpreting CIs, it is important to examine both ends of the CI and judge whether there is a clinically important difference between them. For example, a point estimate of 5% <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk reduction<\/a> in stroke risk over 5 years with a 95% CI of 3% to 7% will include a narrow range that many clinicians would consider clinically important difference at both ends of the interval. This examination (along with considerations of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>) will help establish the degree of uncertainty in the result. See <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\">McCormack J et al<\/a>. for a discussion of how considering only statistical significance without proper regard for CIs can cause confusion.<\/p>\n<p>By convention, a 95% CI that does not include the null (e.g. a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative effect<\/a> 95% CI that includes 1.0 or an <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk difference<\/a> 95% CI that includes 0%) is considered statistically significant (i.e. consistent with p&lt;0.05).<\/p>\n<h1>Sample Size Interpretation<\/h1>\n<p>It is sometimes believed that sample size (i.e. how many participants were included in the study) is a determinant of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_105\">internal validity<\/a>. However, a\u00a0review (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Kjaergard LL et al.<\/a>) found that smaller trials (&lt;1,000 participants) only exaggerated treatment effects compared with larger trials (\u22651,000 participants) when they had inadequate randomization, <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_34\">allocation concealment<\/a>, or blinding. As such, sample size itself is not indicative of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>. Furthermore, if there were too few participants enrolled to detect a difference between groups this will be reflected in the corresponding wide CI (see <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/the-results\/\">Confidence Intervals \u2013 How precise were the estimates of treatment effect?<\/a> for a discussion of wide CIs and how they illustrate precision).<\/p>\n<div class=\"textbox shaded\">\n<p><strong>Advanced discussion<\/strong> (included for completeness, but rarely applicable to appraisal)<\/p>\n<p>An exception may be true for some &#8220;very small&#8221; trials &#8211; as parametric statistical tests rely on the central limit theorem and require a minimum sample size (e.g. n\u226530 is often suggested). See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Fagerland MW<\/a> for a more discussion. The details of this particular statistical concern are beyond the scope of this resource, but two simplified takeaways are that:<\/p>\n<ul>\n<li>&#8220;Very small&#8221; sample size may be a concern for the proper use of certain statistical methods when measuring continuous outcomes<\/li>\n<li>A minimum sample of 30 is an arbitrary rule-of-thumb to prevent this &#8211; nonetheless 30 does provide an approximation of the sample sizes where this may be a concern (e.g. this will almost certainly not be a concern for a trial with several hundreds of participants)<\/li>\n<\/ul>\n<\/div>\n<h1>Absolute Risk Differences and Relative Measures of Effect<\/h1>\n<h2><strong>Absolute Risk Difference<\/strong><\/h2>\n<p>The <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk difference<\/a> between groups refers to the risk of an event in one group minus the risk in another group. Consider the following example of a theoretical 2 year trial examining insomnia rates:<\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 36px\">\n<caption>Table 16. Absolute risk difference example.<\/caption>\n<tbody>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\"><strong>Outcome<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Intervention Group<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Comparator Group<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Absolute Difference<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Duration<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\">Insomnia<\/td>\n<td style=\"width: 20%;height: 18px\">15%<\/td>\n<td style=\"width: 20%;height: 18px\">5%<\/td>\n<td style=\"width: 20%;height: 18px\">+10%<\/td>\n<td style=\"width: 20%;height: 18px\">2 years<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>In this case, the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute difference<\/a> was calculated by subtracting the intervention group event rate (15%) by the comparator group event rate (5%), which equals 10% (15% &#8211; 5%). This difference is &#8220;absolute&#8221; because the number (e.g. +10% risk of insomnia over 2 years) is independently meaningful.<\/p>\n<p><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">Absolute differences<\/a> also need to be communicated in the context of time. For example, a 1% <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk reduction<\/a> over 1 month is quite different from a 1% <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk reduction<\/a> over 10 years. As such, <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute differences<\/a> should be stated as a __% increase\/decrease over [timeframe].<\/p>\n<h2><strong>Relative Measures of Effect<\/strong><\/h2>\n<p>This contrasts with <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative effect<\/a> measures. One example of a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative effect<\/a> is <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">relative risk (RR)<\/a>, which is calculated by dividing the risk of event in the intervention group by that in the comparator group.<\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\">\n<caption>Table 17. Relative effect example.<\/caption>\n<tbody>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\"><strong>Outcome<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Intervention Group<\/strong><\/td>\n<td style=\"width: 20%;height: 18px\"><strong>Comparator Group<\/strong><\/td>\n<td style=\"width: 20%\"><strong><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">Relative Risk (RR)<\/a><\/strong><\/td>\n<td style=\"width: 20%\"><strong>Duration<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\">Insomnia<\/td>\n<td style=\"width: 20%;height: 18px\">15%<\/td>\n<td style=\"width: 20%;height: 18px\">5%<\/td>\n<td style=\"width: 20%;height: 18px\">3.0<\/td>\n<td style=\"width: 20%\">3 months<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>In this case, the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> was calculated by dividing the intervention group rate (15%) by the comparator group event rate (5%), which equals 3.0 (15% \u00f7 5%). <span style=\"text-align: initial;font-size: 14pt\">This difference is &#8220;relative&#8221; because the number (e.g. a 3.0 <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> of experiencing insomnia) is dependent on the risk in the comparator group to be meaningful. <\/span><span style=\"font-size: 14pt;text-align: initial\"><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> 3.0 means that the risk has tripled, but without knowing the baseline risk that is being tripled, then the number is not fully interpretable.<\/span><\/p>\n<p>This dependence can be problematic if not properly considered. Consider the following example, where the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> is identical in both cases:<\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\">\n<caption>Table 18. Relative effect dependency on baseline risk example.<\/caption>\n<tbody>\n<tr>\n<td style=\"width: 20%\"><strong><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">Relative Risk (RR)<\/a><\/strong><\/td>\n<td style=\"width: 20%\">\u00a0<strong>Baseline Risk<\/strong><\/td>\n<td style=\"width: 20%;text-align: left\"><strong>Risk on Treatment<br \/>\n(<a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> x Baseline Risk)<\/strong><\/td>\n<td style=\"width: 20%;text-align: left\"><strong><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">Absolute Risk Difference<\/a><\/strong><\/td>\n<td style=\"width: 20%\"><strong>Duration<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\">0.5<\/td>\n<td style=\"width: 20%;height: 18px\">30%<\/td>\n<td style=\"width: 20%;height: 18px\">15%<\/td>\n<td style=\"width: 20%;height: 18px\">15%<\/td>\n<td style=\"width: 20%\">10 years<\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 20%;height: 18px\">0.5<\/td>\n<td style=\"width: 20%;height: 18px\">2%<\/td>\n<td style=\"width: 20%;height: 18px\">1%<\/td>\n<td style=\"width: 20%;height: 18px\">1%<\/td>\n<td style=\"width: 20%\">10 years<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>As demonstrated, <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> considered in isolation lacks crucial information. The same concept is relevant when (responsibly) buying a product during a sale. Knowing that a particular product is 50% off is not sufficient for a rational choice, as there needs to also be information about the original price (e.g. $20 versus $20,000) before deciding if the purchase is desirable.<\/p>\n<p>Note: <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> is just one relative measure &#8211; see discussion below for information on <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">relative risks<\/a>, <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">odds ratios<\/a>, and <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">hazard ratios<\/a>.<\/p>\n<h1>Number Needed to Treat or Harm<\/h1>\n<p>Both number needed to treat (NNT) and number needed to harm (NNH) are measures of how many patients have to receive the treatment of interest for one additional person to experience the outcome of interest (NNT being for beneficial outcomes, and NNH for harmful outcomes).<\/p>\n<p>It is calculated as: 100 \u00f7 <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">Absolute risk difference<\/a>, with the result always rounded up.<\/p>\n<p>For example, if a treatment has a 7% <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk increase<\/a> of causing urinary retention over 3 months then the NNH is 15 (100 \u00f7 7 = 14.3, then round up to 15). This means that 15 patients will have to be treated for one of them to have urinary retention over the next 3 months (always including timeframe, as with <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk differences<\/a>).<\/p>\n<p>This is an alternative way to understand <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_111\">absolute risk differences<\/a> that may be more intuitive to some (though it is more poorly understood by patients than other measures, as discussed <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/the-results\/\" target=\"_blank\" rel=\"noopener\">here<\/a>).<\/p>\n<h1>Relative Risk, Odds Ratios, and Hazard Ratios<\/h1>\n<p>Before delving into the details of each type of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative effect<\/a> it should be noted that all of them have the following features:<br \/>\nAny relative measure = 1.0 means there was no difference between groups<br \/>\nAny relative measure &gt; 1.0 means the outcome was more likely with the intervention than the comparator<br \/>\nAny relative measure &lt; 1.0 means the outcome was less likely with the intervention than the comparator<\/p>\n<p>To demonstrate the differences between these measures of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative effect<\/a>, consider the following table:<\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 61.0469%;height: 54px\">\n<caption>Table 19. Example 2&#215;2 chart of aspirin vs. placebo for stroke prevention.<\/caption>\n<tbody>\n<tr style=\"height: 18px\">\n<td style=\"width: 16.4884%;height: 18px\"><\/td>\n<td style=\"width: 19.9431%;height: 18px\"><strong>Stroke<\/strong><\/td>\n<td style=\"width: 19.4498%;height: 18px\"><strong>No stroke<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 16.4884%;height: 18px\"><strong>Aspirin<\/strong><\/td>\n<td style=\"width: 19.9431%;height: 18px\">10 (A)<\/td>\n<td style=\"width: 19.4498%;height: 18px\">90 (B)<\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 16.4884%;height: 18px\"><strong>Placebo<\/strong><\/td>\n<td style=\"width: 19.9431%;height: 18px\">20 (C)<\/td>\n<td style=\"width: 19.4498%;height: 18px\">80 (C)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2><strong>Relative Risk (RR)<\/strong><\/h2>\n<p>Calculating the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> consists of dividing the risk of event in the aspirin group by risk of event in the placebo group.<\/p>\n<p>Using the above table:<br \/>\nThe risk of event in the treatment group: A \u00f7 (A+B)<br \/>\nThe risk of event in the comparator group: C \u00f7 (C+D)<br \/>\nWith numbers imputed: 10 \u00f7 100 = 0.1 (or 10%) in the aspirin group and 20 \u00f7 100 = 0.2 (or 20%) in the placebo group.<br \/>\nThe <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> is then 0.1 \u00f7 0.2 = 0.5.<\/p>\n<h2><strong>Odds Ratio (OR)<\/strong><\/h2>\n<p>Calculating the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">OR<\/a> consists of dividing the odds of an event in the aspirin group by the odds of an event in the placebo group.<\/p>\n<p>Using the above table:<br \/>\nThe odds of event in intervention group: A \u00f7 B<br \/>\nThe odds of event in the comparator group: C \u00f7 D<br \/>\nWith numbers imputed: 10 \u00f7 90 = 0.11 in the aspirin group and 20 \u00f7 80 = 0.25 in the placebo group.<br \/>\nThe <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">OR<\/a> is then 0.11 \u00f7 0.25 = 0.44.<\/p>\n<p><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">OR<\/a> are similar to <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> when events are rare (A \u00f7 (A+B) \u2248 A \u00f7 B when A is very small) (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Holcomb WL et al.<\/a>). As events become more common, these measures diverge and <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">ORs<\/a> will overestimate <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RRs<\/a> (such as in this example where RR=0.5 and OR=0.44). The <a href=\"https:\/\/clincalc.com\/Stats\/ConvertOR.aspx\" target=\"_blank\" rel=\"noopener\">ClinCalc tool<\/a> can be used to convert <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">OR<\/a> to <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a>.<\/p>\n<h2><strong>Hazard Ratio (HR)<\/strong><\/h2>\n<p><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">Hazard ratios (HRs)<\/a> represent the average of the instantaneous incidence rate at every point during a trial. Consider an example of a 5-year trial that has a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> of 0.70 for the outcome of death comparing an intervention against some comparator. This means that a participant assigned to intervention will be 30% less likely to die relative to the comparator at any point during the trial:<\/p>\n<ul>\n<li>Year 1: If 5% have died in the comparator group, then 3.5% are expected to have died in the intervention group (5% * 0.70 = 3.5%)<\/li>\n<li>Year 2: If 10% have died in the comparator group, then 7% are expected to have died in the intervention group (10% * 0.70 = 7%)<\/li>\n<li>Year 5: If 20% have died in the comparator group, 14% are expected to have died in the intervention group (20% * 0.70 = 14%)<\/li>\n<\/ul>\n<p>The same is approximately true at any given timepoint during the trial follow-up. These are all approximations as the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> is an average, it (almost certainly) will not be exactly true at every time point. For instance, the final <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> might be 0.70, but it could be 0.80 during the first half of the trial and 0.60 during the latter half.<\/p>\n<p>For example, consider the unadjusted analysis from an observational study (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turgeon RD, Koshman SL, et al.<\/a>) that compared the use of ticagrelor vs. clopidogrel in patients who had undergone percutaneous coronary intervention following acute coronary syndrome (ACS). For the outcome of survival without major adverse coronary events (MACE) the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> was 0.84 before adjustment for potential confounding variables, depicted visually below:<\/p>\n<div class=\"textbox\">\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1452\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1.jpg\" alt=\"\" width=\"760\" height=\"420\" srcset=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1.jpg 760w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1-300x166.jpg 300w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1-65x36.jpg 65w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1-225x124.jpg 225w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/MACE-final-KM-1-350x193.jpg 350w\" sizes=\"auto, (max-width: 760px) 100vw, 760px\" \/><\/p>\n<p>Graph 3. Kaplan Meier curve of survival without major adverse coronary events<\/p>\n<\/div>\n<p>While not exactly true at every time point, past the first 100 days the cumulative proportion patients experiencing death or MACE in the ticagrelor group appears to be roughly 84% of the cumulative proportion in the clopidogrel group fairly consistently (see &#8220;Kaplan Meier Curves&#8221; for more information below on how to interpret these types of graphs). This coheres with the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> of 0.84 discussed above.<\/p>\n<p><span style=\"text-align: initial\"><span style=\"font-size: 1em\"><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HRs<\/a> are usually similar to <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RRs<\/a> (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Sutradhar R et al.<\/a>). <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HRs<\/a> examine multiple timepoints over trial follow-up, whereas <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RRs<\/a> evaluate cumulative proportions at the end of the trial (or at another single timepoint). <\/span><\/span><span style=\"text-align: initial\"><span style=\"font-size: 1em\"><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HRs<\/a> can account for differential follow-up times, and contain more information than <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RRs<\/a>\/<a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">ORs<\/a> since they include the added dimension of time (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Guyatt G et al.<\/a>). <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HRs<\/a> are limited in their ability to convey fluctuations in effect over time, as a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> of 1.0 could mean that there was consistently no effect, or it could mean that there was beneficial effect during the first half and a proportional detrimental effect during the latter half (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Hern\u00e1n MA<\/a>). However, some of these limitations can be overcome by combining a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> with the use of a Kaplan Meier curve, as discussed below.<\/span><\/span><\/p>\n<h1>Kaplan Meier Curves<\/h1>\n<h2><strong>Cumulative Hazards<\/strong><\/h2>\n<p>Kaplan Meier curves are graphical representations comparing event accrual between two groups over time. Consider another example from the aforementioned study comparing ticagrelor vs. clopidogrel:<\/p>\n<div class=\"textbox\">\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1545\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final.png\" alt=\"\" width=\"1341\" height=\"777\" srcset=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final.png 1341w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-300x174.png 300w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-1024x593.png 1024w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-768x445.png 768w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-65x38.png 65w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-225x130.png 225w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/01\/Survival-Without-ACS-Final-350x203.png 350w\" sizes=\"auto, (max-width: 1341px) 100vw, 1341px\" \/><br \/>\nGraph 4. Kaplan Meier curve of survival without acute coronary syndrome (ACS)<\/p>\n<\/div>\n<p>Each curve displays the cumulative proportion of patients in that group who have experienced the outcome of interest. As time passes more participants experience the outcome and the curve progresses downwards. The <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_119\">relative differences<\/a> in outcome accumulation can be expressed as a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a>, as discussed above. Note that, while not displayed, each curve has a <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">CI<\/a> surrounding it at every time point.<\/p>\n<h2><strong>Onset of Benefit<\/strong><\/h2>\n<p>Consider the above Kaplan Meier curve. During the first 50 days, the curves for the two groups overlap. However, after this point the two curves begin to separate. This curve provides insight into the onset of benefit of the intervention and if the benefit is sustained over time. In this case, onset of benefit begins after approximately 50 days and is sustained as time elapses.<\/p>\n<h2><strong>Course of Condition<\/strong><\/h2>\n<p>The graph also gives insight into event rates over time. As can be seen above, the curve is steepest initially &#8211; indicating that the risk of death or ACS is highest immediately following the intervention. The slope of the curve then flattens and remains relatively stable &#8211; indicating the event rate after the initial period is relatively constant during the first year. This demonstrates how Kaplan Meier curves can be useful to understand the course of a condition over time.<\/p>\n<h2><strong>Total at Risk (or Number at Risk)<\/strong><\/h2>\n<p>Consider another curve from the same study, this one examining survival without major bleeds:<\/p>\n<div class=\"textbox\">\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1454\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1.jpg\" alt=\"\" width=\"848\" height=\"472\" srcset=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1.jpg 848w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1-300x167.jpg 300w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1-768x427.jpg 768w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1-65x36.jpg 65w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1-225x125.jpg 225w, https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/11\/Major-Bleed-final-KM-1-350x195.jpg 350w\" sizes=\"auto, (max-width: 848px) 100vw, 848px\" \/><\/p>\n<p>Graph 5. Kaplan Meier curve of survival without major bleeding<\/p>\n<\/div>\n<p>As depicted, sometimes there is also a &#8220;Total at risk&#8221; (sometimes called &#8220;Number at risk&#8221;) table beneath the curve. This can give additional information regarding the participants as they progressed through the trial. All participants begin &#8220;at risk&#8221;, but as the study progresses the number decreases. Below are the following reasons the number may decrease, as well as possible implications if the decreases are not balanced between groups:<\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 75px\">\n<caption>Table 20. Reasons and implications of total at risk decreases.<\/caption>\n<tbody>\n<tr style=\"height: 15px\">\n<td style=\"width: 22.0812%;height: 15px\"><strong>Reasons for total at risk decreasing<\/strong><\/td>\n<td style=\"width: 77.9188%;height: 15px\"><strong>Implications if imbalanced between groups<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 15px\">\n<td style=\"width: 22.0812%;height: 15px\">Outcome of interest occurred<\/td>\n<td style=\"width: 77.9188%;height: 15px\">If there is a difference in effect between the intervention and comparator, then the total at risk may decrease more quickly in one group. This is evidence of effect, not <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>.<\/td>\n<\/tr>\n<tr style=\"height: 15px\">\n<td style=\"width: 22.0812%;height: 15px\">Death<\/td>\n<td style=\"width: 77.9188%;height: 15px\">If there are differences in mortality rates then this should prompt consideration of the relative safety of the comparators, as well as consideration of death as a competing event within the analysis.<\/td>\n<\/tr>\n<tr style=\"height: 15px\">\n<td style=\"width: 22.0812%;height: 15px\"><a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_121\">Loss to follow-up<\/a><\/td>\n<td style=\"width: 77.9188%;height: 15px\">This could result in systematic <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a> if the reasons for loss to follow-up are not random (for more see the discussion on <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_121\">loss to follow-up<\/a> <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/chapter-1\/\" target=\"_blank\" rel=\"noopener\">here<\/a>).<\/td>\n<\/tr>\n<tr style=\"height: 15px\">\n<td style=\"width: 22.0812%;height: 15px\">The study ended before the participant had outcome data at that time point<\/td>\n<td style=\"width: 77.9188%;height: 15px\">If by chance there is a difference in how many patients were enrolled early in one group (and thus had more time to accrue events) this could <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a> a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_109\">RR<\/a> or <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_103\">OR<\/a>. For example, by chance one group might have patients enrolled for an average of 4 years and another group had patients enrolled for an average of 5 years. However, since the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_110\">HR<\/a> incorporates the timing of events, this should not result in <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Thus the total at risk table can serve as a clue that further examination should be undertaken to see if there is <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>.<\/p>\n<h1>Forest Plots<\/h1>\n<p>Forest plots are used in <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_1101\">meta-analyses<\/a> to graphically depict the effects of an intervention across multiple studies. Consider the labeled example below from the &#8220;Beta-blockers for hypertension&#8221; Cochrane Review (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Wiysonge CS et al.<\/a>):<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-50 size-full\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1552\/2021\/11\/Forest-plot-new-new.png\" alt=\"\" width=\"1604\" height=\"466\" data-wp-editing=\"1\" \/>Plot 4. Forest plot of beta-blockers vs. placebo in patients with hypertension for the outcome of mortality.<\/p>\n<p>As showcased, forest plots show information about each individual study included for that outcome, and also the combined results. This information is displayed visually as well as numerically. Trials with more events or participants are generally given greater weight. <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_107\">Heterogeneity<\/a> is typically measured via I<sup>2<\/sup>, which is 0% in this case. For more information on <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_107\">heterogeneity<\/a> see <a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/chapter\/results-of-the-meta-analysis-i-e-what-do-the-pooled-results-of-the-trials-show\/\" target=\"_blank\" rel=\"noopener\">here<\/a>.<\/p>\n<h1>Standardized Mean Difference Interpretation<\/h1>\n<p>The <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">standardized mean difference (SMD)<\/a> is a method of combining multiple continuous outcome scoring systems into one measurement. For example, when performing a meta-analysis on the effects of antidepressants on depression symptom reduction, trials may use different scales to rate depression symptoms (HAM-D, PHQ-9, etc.). <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a> will allow the aggregation of the results of all these studies. Notably, using <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a> assumes that differences between studies are due to differences in scales (and not in intervention\/population characteristics). Arbitrary &#8220;rule-of-thumb&#8221; cutoffs (e.g. <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a> of 0.2 = &#8220;small effect&#8221;) may not reflect the minimal important difference.<\/p>\n<p>An alternative approach is to transform the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a> into a more familiar scale (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Higgins JPT et al.<\/a>). Multiply the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a> by the standard deviation (SD) of the largest trial to convert to its scale.<\/p>\n<div class=\"textbox shaded\">\n<p><em>E.g. These are the results of a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_1101\">meta-analysis<\/a> which assesses the effects of IV iron on health-related quality of life at 6 months in patients with HFrEF (<a href=\"https:\/\/pressbooks.bccampus.ca\/testforthistobedone\/back-matter\/references\/\">Turgeon RD, Barry AR, et al.<\/a>):<\/em><\/p>\n<p><em><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-51\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1552\/2021\/11\/SMD-Example-1.png\" alt=\"\" width=\"988\" height=\"317\" \/><\/em><\/p>\n<p><em>Plot 5. Forest plot of IV iron vs. placebo in patients with heart failure with reduced ejection fraction on health-related quality of life.<\/em><\/p>\n<p><em><strong>Step 1:<\/strong> Identify the trial with the most weight, FAIR-HF in this case.<\/em><\/p>\n<p><em><strong>Step 2:<\/strong> Calculate average SD. Average SD in this case is ~20.50 (the average of 16.9115 and 24.0832).\u00a0<\/em><\/p>\n<p><em><strong>Step 3:<\/strong> Multiply average SD by <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_113\">SMD<\/a>. In this case this gives 10.66 (20.50 * 0.52).\u00a0<\/em><\/p>\n<p><em><strong>Step 4:<\/strong> Contextualize this result in the scale used in the trial. In this case, that would be equal to a 10.66 out of 100 improvement at 6 months (per the scale used in FAIR-HF).\u00a0<\/em><\/p>\n<p><em><strong>Step 5:<\/strong> Compare this value to the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_78\">minimally important difference (MID)<\/a> if known. In the case of the FAIR-HF scale the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_78\">MID<\/a> was 5. Therefore the mean effect was greater than the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_78\">MID<\/a>.<\/em><\/p>\n<\/div>\n<p>Ideally the review will present this information along with a comparison of the proportion of participants within each group who experienced clinically important improvement or decline (the so-called &#8220;responder analysis&#8221;). This is useful because calculating the mean alone will not provide any information about the distribution. The responder analysis unfortunately cannot typically be calculated by readers if it is not already reported by the reviewers.<\/p>\n<h1>Statistical Significance Is Not Everything<\/h1>\n<p>While this section has focused on statistical fundamentals it is important to emphasize that statistics are only one aspect of critical appraisal. Even if a study shows statistical significance, there needs to be considerations of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a>, clinical significance, and <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_1653\">generalizability<\/a>.<\/p>\n<p><strong>Bias<\/strong>: P-values and CIs are contingent on all the assumptions being used to calculate them being correct. In other words, they assume there is absolutely no <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_193\">bias<\/a> present. As such, if the study is poorly conducted (e.g. a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_704\">RCT<\/a> without adequate <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_6_34\">allocation concealment<\/a> and blinding) this will not be reflected in the statistical analysis of the results. This is why it is necessary to appraise the conduct of the trial to evaluate the credibility of the results.<\/p>\n<p><strong>Clinical significance<\/strong>: Even if a result is statistically significant, it may be too small of an effect to matter to a patient. For instance, with enough participants, a 1-point reduction in pain on a 100-point scale could be statistically significant, but very unlikely to be felt by an individual patient.<\/p>\n<p><strong>Generalizability<\/strong>: Even if the results are unbiased and clinically significant, they will only be useful if they can be applied to practice. If there are substantial differences between the features of the trial and your own practice, then the result may not be applicable.<\/p>\n<p>Other sections of this resource will go into these concepts in more depth, but these are the fundamental reasons why a comprehensive approach to appraisal is necessary, and simply looking at statistical significance in the results section will not be sufficient to understand the clinical implications of a trial.<\/p>\n<div class=\"glossary\"><span class=\"screen-reader-text\" id=\"definition\">definition<\/span><template id=\"term_6_704\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_704\"><div tabindex=\"-1\"><p>Randomized controlled trials are those in which participants are randomly allocated to two or more groups which are given different treatments.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_1099\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_1099\"><div tabindex=\"-1\"><p>A review that systematically identifies all potentially relevant studies on a research question. The aggregate of studies is then evaluated with respect to factors such as risk of bias of individual studies or heterogeneity among results. The qualitative combination of results is a systematic review.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_1101\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_1101\"><div tabindex=\"-1\"><p>A meta-analysis is a quantitative combination of the data obtained in a systematic review.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_828\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_828\"><div tabindex=\"-1\"><p>In superiority analyses, this is the hypothesis that there is no difference in the outcome of interest between the intervention group and the comparator group. In non-inferiority analyses, this is the hypothesis that there is a difference in the outcomes of interest between the treatment group and the control group.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_109\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_109\"><div tabindex=\"-1\"><p>Relative risk (or risk ratio) is the risk in one group relative to (divided by) risk in another group. For example, if 10% in the treatment group and 20% in the placebo group have the outcome of interest, the relative risk in the treatment group is 0.5 (10% \u00f7 20%; half) the risk in the placebo group. See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for a more detailed discussion.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_111\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_111\"><div tabindex=\"-1\"><p>Absolute risk difference is the risk in one group compared to (minus) the risk in another group over a specified period of time. For example, if the absolute risk of myocardial infarction over 5 years was 15% for the comparator and 10% for the intervention, then the absolute risk difference was 5% (15% - 10%) over 5 years. See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for further discussion.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_193\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_193\"><div tabindex=\"-1\"><p>Systematic deviation of an estimate from the truth (either an overestimation or underestimation) caused by a study design or conduct feature. See the <a href=\"https:\/\/catalogofbias.org\/biases\/\">Catalog of Bias<\/a> for specific biases, explanations, and examples.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_119\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_119\"><div tabindex=\"-1\"><p>Calculates the effect of an intervention via a fractional comparison with the comparator group (i.e. intervention group measure \u00f7 comparator group measure). Used for binary outcomes. Relative risk, odds ratio, or hazards ratio are all expressions of relative effect. For example, if the risk of developing neuropathy was 1% in the treatment group and 2% in the comparator group, then the relative risk is 0.5 (1 \u00f7 2). See the Absolute Risk Differences and Relative Measures of Effect discussion <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for more information.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_105\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_105\"><div tabindex=\"-1\"><p>The extent to which the study results are attributable to the intervention and not to bias. If internal validity is high, there is high confidence that the results are due to the effects of treatment (with low internal validity entailing low confidence).<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_34\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_34\"><div tabindex=\"-1\"><p>Refers to the process that prevents patients, clinicians, and researchers from predicting which intervention group the patient will be assigned. This is different from blinding; allocation concealment refers to patients\/clinicians\/outcome assessors\/etc. being unaware of group allocation prior to randomization, whereas blinding refers to remaining unaware of group allocation after randomization. Allocation concealment is a necessary condition for blinding. It is always feasible to implement.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_103\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_103\"><div tabindex=\"-1\"><p>Odds ratios are the ratio of odds (events divided by non-events) in the intervention group to the odds in the comparator group. For example, if the odds of an event in the treatment group is 0.2 and the odds in the comparator group is 0.1, then the OR is 2 (0.2\/0.1). See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for a more detailed discussion.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_110\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_110\"><div tabindex=\"-1\"><p>Hazard ratios are a relative measure of effect. Hazards refer to average instantaneous incidence rate at every point during the trial. This differentiates it from other measures, such as relative risk, which rely only on cumulative event rates. See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for a more detailed discussion.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_121\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_121\"><div tabindex=\"-1\"><p>Loss to follow-up may occur when participants stop coming to study follow-up visits, do not answer follow-up phone calls, and cannot otherwise be assessed for study outcomes. This leads to missing data from the time they became \"lost\". Underlying reasons may include leaving the trial without informing investigators, moving to a new location, debilitation due to illness, or death.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_107\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_107\"><div tabindex=\"-1\"><p>Refers to variability between studies in a systematic review. It can refer to clinical differences, methodological differences, or variable results between studies. Heterogeneity occurs on a continuum and, in the case of heterogeneity amongst results, can be expressed numerically via measures of statistical heterogeneity. See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/results-of-the-meta-analysis-i-e-what-do-the-pooled-results-of-the-trials-show\/\">here<\/a> for a further discussion of statistical heterogeneity.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_113\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_113\"><div tabindex=\"-1\"><p>Transformation of continuous data that consists of dividing the difference in means between two groups by the standard deviation of the variable. In clinical research, this is often used to summarize and\/or pool continuous outcomes that are measured in several ways. For example, a meta-analysis of antidepressants may need to use the SMD if trials used different scales (e.g. Beck Depression Inventory, Hamilton Depression Rating Scale) to report change in depression symptoms. See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for further discussion on SMD interpretation.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_78\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_78\"><div tabindex=\"-1\"><p>The minimum difference in a value that would be of importance to a patient. There are various methods of calculating a minimally important difference.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_6_1653\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_6_1653\"><div tabindex=\"-1\"><p>Refers to the extent to which the trial results are applicable beyond the patients included in the study. Also known as external validity.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><\/div>","protected":false},"author":1226,"menu_order":1,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"back-matter-type":[27],"contributor":[],"license":[],"class_list":["post-6","back-matter","type-back-matter","status-publish","hentry","back-matter-type-appendix"],"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter\/6","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/types\/back-matter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/users\/1226"}],"version-history":[{"count":26,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter\/6\/revisions"}],"predecessor-version":[{"id":1925,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter\/6\/revisions\/1925"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter\/6\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/media?parent=6"}],"wp:term":[{"taxonomy":"back-matter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/back-matter-type?post=6"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/contributor?post=6"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/license?post=6"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}