{"id":44,"date":"2021-05-21T00:25:10","date_gmt":"2021-05-21T04:25:10","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/?post_type=chapter&p=44"},"modified":"2024-03-12T21:53:03","modified_gmt":"2024-03-13T01:53:03","slug":"the-results","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/the-results\/","title":{"raw":"Interpreting the results","rendered":"Interpreting the results"},"content":{"raw":"When examining the results of a trial, it is necessary to consider more than statistical significance. Not all statistically significant results are clinically important. Similarly, failure to find a statistically significant difference does not necessarily rule out that there is a clinically important difference. In addition, when making judgements of clinical relevance, it is necessary to examine not only the [pb_glossary id=\"119\"]relative effect[\/pb_glossary] of treatment, but also the [pb_glossary id=\"111\"]absolute effect[\/pb_glossary] (see here<\/a> for further discussion).\r\n\r\nIn terms of effectively communicating results to patients, a review (Zipkin DA et al.<\/a>) found that:\r\n- Any type of difference ([pb_glossary id=\"111\"]absolute[\/pb_glossary] or [pb_glossary id=\"119\"]relative[\/pb_glossary]) is understood more accurately when baseline risk is provided;\r\n- [pb_glossary id=\"111\"]Absolute differences[\/pb_glossary] are understood more accurately than [pb_glossary id=\"119\"]relative differences[\/pb_glossary];\r\n- Numbers needed to treat (NNTs)<\/a> are often misunderstood and are inferior to reporting absolute differences;\r\n- Addition of visual displays to numerical information increase understanding.\r\n

Checklist Questions<\/h1>\r\n\r\n\r\n\r\n\r\n\r\n\r\n
What was the magnitude of effect for efficacy and harms?<\/td>\r\n<\/tr>\r\n
How precise were the estimates of treatment effect?<\/td>\r\n<\/tr>\r\n
Is the difference clinically important?<\/td>\r\n<\/tr>\r\n
Are these results consistent with other evidence?<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n

Point estimate: What was the magnitude of effect for efficacy and harms?<\/span><\/span><\/h1>\r\nLook at both the [pb_glossary id=\"111\"]absolute effect[\/pb_glossary] and the [pb_glossary id=\"119\"]relative effect[\/pb_glossary]. [pb_glossary id=\"119\"]Relative effects[\/pb_glossary] are typically assumed to be reasonably consistent across populations, whereas [pb_glossary id=\"111\"]absolute effects[\/pb_glossary] depend on baseline risk.\r\n
E.g. The [pb_glossary id=\"1111\"]relative risk reduction[\/pb_glossary] of statins on all-cause mortality is similar in primary prevention (i.e. prevention in patients without cardiovascular disease) and secondary prevention (patients with cardiovascular disease), but the [pb_glossary id=\"111\"]absolute effect[\/pb_glossary] is greater in secondary prevention (<\/em>Wilt TJ et al., <\/a>Tonelli M et al.<\/a>):\r\n<\/em><\/em>\r\n\r\n\r\n\r\n\r\n
Table 8. Comparison of absolute risk reductions in primary and secondary prevention patients treated with statins.<\/caption>\r\n
Population<\/strong><\/em><\/td>\r\nRelative risk reduction<\/strong><\/em><\/td>\r\nAbsolute Risk Reduction Over 5 Years<\/strong><\/em><\/td>\r\n<\/tr>\r\n
Primary prevention patients<\/em>\r\n- No coronary artery disease\r\n- A predicted <20% risk of a cardiovascular event in the next 10 years<\/em><\/td>\r\n10-15%<\/em><\/td>\r\n0.4%<\/em><\/td>\r\n<\/tr>\r\n
Secondary prevention patients<\/em>\r\n- Coronary artery disease<\/em><\/td>\r\n10-15%<\/em><\/td>\r\n2%<\/em><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

Confidence interval:\u00a0How precise were the estimates of treatment effect?<\/span><\/span><\/h1>\r\nConfidence intervals (CIs)<\/a> provide information regarding the uncertainty of the results. The wider the CI<\/a>, the greater the uncertainty. The width is based on the difference between the two ends of the CI<\/a>. Wide and narrow do not have exact definitions.\r\n
E.g. #1 Narrow 95% CI<\/a>: [pb_glossary id=\"109\"]RR[\/pb_glossary] 0.90 (0.85 to 0.95)<\/em>\r\nE.g. #2 Wide 95% CI<\/a>: [pb_glossary id=\"103\"]OR[\/pb_glossary] 1.25 (0.2 to 5)<\/em><\/div>\r\n
E.g. #3 A meta-analysis by Ortiz-Orendain J et al.<\/a> illustrates visually how CI<\/a> have varying widths:<\/em>\r\n\"\"<\/em>\r\nPlot 1. Forest plot of any antipsychotic plus atypical antipsychotic vs. atypical antipsychotic in patients with schizophrenia on the outcome of no clinically important response.<\/em><\/div>\r\nThe relevance of this uncertainty depends on whether the CI<\/a> includes clinically important differences (see the following section). This involves examining both ends of the CI<\/a>, and judging whether there is a meaningful difference between the two.\r\n
E.g. #4 In CAPRIE (CAPRIE Steering Committee<\/a>), the lower end of the [pb_glossary id=\"1111\"]relative risk reduction[\/pb_glossary] 95% CI<\/a> (\u201cworst-case\u201d) was 0.3% and the upper end (\u201cbest-case\u201d) was 16.5%, corresponding to a NNT<\/a> of 5555 and 105 per year, respectively.<\/em><\/div>\r\n

Is the difference clinically important?<\/h1>\r\nClinical importance is determined by looking at the [pb_glossary id=\"111\"]absolute risk difference[\/pb_glossary], rather than a [pb_glossary id=\"1111\"]relative risk reduction[\/pb_glossary]. Note that clinical importance is dependent on an individual's preferences and values, therefore opinions will differ based on clinician and patient preferences, patient situation, intervention characteristics (e.g. adverse events, cost, convenience), and other factors.\r\n
E.g. #1 A<\/span><\/span><\/span> [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] (ATT Collaboration<\/a>) <\/span><\/span><\/span>comparing ASA vs. placebo <\/span><\/span><\/span>in<\/span><\/span><\/span>\u00a0secondary prevention\u00a0<\/span><\/span><\/span>patients<\/span><\/span><\/span>\u00a0fo<\/span><\/span><\/span>und<\/span><\/span><\/span>\u00a0a<\/span><\/span><\/span>\u00a0<\/span><\/span><\/span>statistically significant <\/span><\/span><\/span>1.<\/span><\/span><\/span>5<\/span><\/span><\/span>% per <\/span><\/span><\/span>year<\/span><\/span><\/span>\u00a0<\/span><\/span><\/span>[pb_glossary id=\"111\"]absolute r<\/span><\/span><\/span>isk reduction<\/span><\/span><\/span>[\/pb_glossary] in<\/span><\/span><\/span>\u00a0serious vascular event<\/span><\/span><\/span>s<\/span><\/span><\/span> (myocardial infarction, st<\/span><\/span><\/span>roke, or vascular death)<\/span><\/span><\/span>. The same [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary]\u00a0<\/span>also examined ASA vs. placebo in primary prevention patients found a much smaller, but still statistically significant, 0.07% per year [pb_glossary id=\"111\"]absolute reduction[\/pb_glossary] for the same outcome. Many patients and clinicians would consider the benefit of ASA in secondary prevention to be clinically meaningful, whereas far fewer would be willing to take ASA for primary prevention knowing these numbers.<\/span><\/span><\/span><\/em><\/div>\r\nConsider comparing the results with [pb_glossary id=\"111\"]absolute risk reductions[\/pb_glossary] or NNTs<\/a> achieved with other interventions used in a similar patient population.\r\n
E.g. #2 In HPS (Heart Protection Study Collaborative Group<\/a>), the [pb_glossary id=\"111\"]absolute reduction[\/pb_glossary] in risk of death over 5 years with simvastatin vs. placebo was 1.8% in a high-risk population. For comparison, ramipril reduced the 5-year risk of death by 1.7% versus placebo in a similar patient population within the HOPE trial (Yusuf S et al.<\/a>).<\/em><\/div>\r\n

Are these results consistent with other evidence?<\/h1>\r\n
Differences between groups in any given study may occur by chance. Replication of the consistent results in multiple studies increases the confidence that the difference represents a true effect of the study intervention. Searching for a [pb_glossary id=\"1099\"]systematic review[\/pb_glossary] on the topic can efficiently provide insight into the context of surrounding literature and consistency between studies.<\/span><\/span>
<\/span><\/div>\r\n
\r\n
E.g. ASPEN (Knopp RH et al.<\/a>), a 2006 [pb_glossary id=\"704\"]RCT[\/pb_glossary], did not demonstrate a statistically significant difference in cardiovascular events between atorvastatin versus placebo in patients with diabetes ([pb_glossary id=\"110\"]HR[\/pb_glossary] 0.9; 95% CI<\/a>, 0.7 to 1.1). In contrast, the 2004 CARDS trial (Colhoun HM et al.<\/a>), also comparing atorvastatin vs. placebo in patients with diabetes, had previously shown benefit in a similar population ([pb_glossary id=\"110\"]HR[\/pb_glossary] 0.6; 95% CI<\/a> 0.5 to 0.8) for a similar [pb_glossary id=\"1517\"]primary endpoint[\/pb_glossary]. The neutral findings of ASPEN should be understood in the context of the CARDS (and also the dozens of other trials that demonstrated benefits of statins vs. placebo for the prevention of cardiovascular events (Cholesterol Treatment Trialists\u2019 (CTT) Collaborators<\/a>)).<\/em><\/div>\r\n<\/div>","rendered":"

When examining the results of a trial, it is necessary to consider more than statistical significance. Not all statistically significant results are clinically important. Similarly, failure to find a statistically significant difference does not necessarily rule out that there is a clinically important difference. In addition, when making judgements of clinical relevance, it is necessary to examine not only the relative effect<\/a> of treatment, but also the absolute effect<\/a> (see here<\/a> for further discussion).<\/p>\n

In terms of effectively communicating results to patients, a review (Zipkin DA et al.<\/a>) found that:
\n– Any type of difference (absolute<\/a> or relative<\/a>) is understood more accurately when baseline risk is provided;
\n– Absolute differences<\/a> are understood more accurately than relative differences<\/a>;
\n– Numbers needed to treat (NNTs)<\/a> are often misunderstood and are inferior to reporting absolute differences;
\n– Addition of visual displays to numerical information increase understanding.<\/p>\n

Checklist Questions<\/h1>\n\n\n\n\n\n\n
What was the magnitude of effect for efficacy and harms?<\/td>\n<\/tr>\n
How precise were the estimates of treatment effect?<\/td>\n<\/tr>\n
Is the difference clinically important?<\/td>\n<\/tr>\n
Are these results consistent with other evidence?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Point estimate: What was the magnitude of effect for efficacy and harms?<\/span><\/span><\/h1>\n

Look at both the absolute effect<\/a> and the relative effect<\/a>. Relative effects<\/a> are typically assumed to be reasonably consistent across populations, whereas absolute effects<\/a> depend on baseline risk.<\/p>\n

E.g. The relative risk reduction<\/a> of statins on all-cause mortality is similar in primary prevention (i.e. prevention in patients without cardiovascular disease) and secondary prevention (patients with cardiovascular disease), but the absolute effect<\/a> is greater in secondary prevention (<\/em>Wilt TJ et al., <\/a>Tonelli M et al.<\/a>):
\n<\/em><\/em><\/p>\n\n\n\n\n\n
Table 8. Comparison of absolute risk reductions in primary and secondary prevention patients treated with statins.<\/caption>\n
Population<\/strong><\/em><\/td>\nRelative risk reduction<\/strong><\/em><\/td>\nAbsolute Risk Reduction Over 5 Years<\/strong><\/em><\/td>\n<\/tr>\n
Primary prevention patients<\/em>
\n– No coronary artery disease
\n– A predicted <20% risk of a cardiovascular event in the next 10 years<\/em><\/td>\n		10-15%<\/em><\/td>\n0.4%<\/em><\/td>\n<\/tr>\n
Secondary prevention patients<\/em>
\n– Coronary artery disease<\/em><\/td>\n		10-15%<\/em><\/td>\n2%<\/em><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Confidence interval:\u00a0How precise were the estimates of treatment effect?<\/span><\/span><\/h1>\n

Confidence intervals (CIs)<\/a> provide information regarding the uncertainty of the results. The wider the CI<\/a>, the greater the uncertainty. The width is based on the difference between the two ends of the CI<\/a>. Wide and narrow do not have exact definitions.<\/p>\n

E.g. #1 Narrow 95% CI<\/a>: RR<\/a> 0.90 (0.85 to 0.95)<\/em>
\nE.g. #2 Wide 95% CI<\/a>: OR<\/a> 1.25 (0.2 to 5)<\/em><\/div>\n
E.g. #3 A meta-analysis by Ortiz-Orendain J et al.<\/a> illustrates visually how CI<\/a> have varying widths:<\/em>
\n\"\"<\/em>
\nPlot 1. Forest plot of any antipsychotic plus atypical antipsychotic vs. atypical antipsychotic in patients with schizophrenia on the outcome of no clinically important response.<\/em><\/div>\n

The relevance of this uncertainty depends on whether the CI<\/a> includes clinically important differences (see the following section). This involves examining both ends of the CI<\/a>, and judging whether there is a meaningful difference between the two.<\/p>\n

E.g. #4 In CAPRIE (CAPRIE Steering Committee<\/a>), the lower end of the relative risk reduction<\/a> 95% CI<\/a> (\u201cworst-case\u201d) was 0.3% and the upper end (\u201cbest-case\u201d) was 16.5%, corresponding to a NNT<\/a> of 5555 and 105 per year, respectively.<\/em><\/div>\n

Is the difference clinically important?<\/h1>\n

Clinical importance is determined by looking at the absolute risk difference<\/a>, rather than a relative risk reduction<\/a>. Note that clinical importance is dependent on an individual’s preferences and values, therefore opinions will differ based on clinician and patient preferences, patient situation, intervention characteristics (e.g. adverse events, cost, convenience), and other factors.<\/p>\n

E.g. #1 A<\/span><\/span><\/span> meta-analysis<\/a> (ATT Collaboration<\/a>) <\/span><\/span><\/span>comparing ASA vs. placebo <\/span><\/span><\/span>in<\/span><\/span><\/span>\u00a0secondary prevention\u00a0<\/span><\/span><\/span>patients<\/span><\/span><\/span>\u00a0fo<\/span><\/span><\/span>und<\/span><\/span><\/span>\u00a0a<\/span><\/span><\/span>\u00a0<\/span><\/span><\/span>statistically significant <\/span><\/span><\/span>1.<\/span><\/span><\/span>5<\/span><\/span><\/span>% per <\/span><\/span><\/span>year<\/span><\/span><\/span>\u00a0<\/span><\/span><\/span>absolute r<\/span><\/span><\/span>isk reduction<\/span><\/span><\/span><\/a> in<\/span><\/span><\/span>\u00a0serious vascular event<\/span><\/span><\/span>s<\/span><\/span><\/span> (myocardial infarction, st<\/span><\/span><\/span>roke, or vascular death)<\/span><\/span><\/span>. The same meta-analysis<\/a>\u00a0<\/span>also examined ASA vs. placebo in primary prevention patients found a much smaller, but still statistically significant, 0.07% per year absolute reduction<\/a> for the same outcome. Many patients and clinicians would consider the benefit of ASA in secondary prevention to be clinically meaningful, whereas far fewer would be willing to take ASA for primary prevention knowing these numbers.<\/span><\/span><\/span><\/em><\/div>\n

Consider comparing the results with absolute risk reductions<\/a> or NNTs<\/a> achieved with other interventions used in a similar patient population.<\/p>\n

E.g. #2 In HPS (Heart Protection Study Collaborative Group<\/a>), the absolute reduction<\/a> in risk of death over 5 years with simvastatin vs. placebo was 1.8% in a high-risk population. For comparison, ramipril reduced the 5-year risk of death by 1.7% versus placebo in a similar patient population within the HOPE trial (Yusuf S et al.<\/a>).<\/em><\/div>\n

Are these results consistent with other evidence?<\/h1>\n
Differences between groups in any given study may occur by chance. Replication of the consistent results in multiple studies increases the confidence that the difference represents a true effect of the study intervention. Searching for a systematic review<\/a> on the topic can efficiently provide insight into the context of surrounding literature and consistency between studies.<\/span><\/span>
<\/span><\/div>\n
\n
E.g. ASPEN (Knopp RH et al.<\/a>), a 2006 RCT<\/a>, did not demonstrate a statistically significant difference in cardiovascular events between atorvastatin versus placebo in patients with diabetes (HR<\/a> 0.9; 95% CI<\/a>, 0.7 to 1.1). In contrast, the 2004 CARDS trial (Colhoun HM et al.<\/a>), also comparing atorvastatin vs. placebo in patients with diabetes, had previously shown benefit in a similar population (HR<\/a> 0.6; 95% CI<\/a> 0.5 to 0.8) for a similar primary endpoint<\/a>. The neutral findings of ASPEN should be understood in the context of the CARDS (and also the dozens of other trials that demonstrated benefits of statins vs. placebo for the prevention of cardiovascular events (Cholesterol Treatment Trialists\u2019 (CTT) Collaborators<\/a>)).<\/em><\/div>\n<\/div>\n
definition<\/span>