{"id":91,"date":"2021-05-22T23:28:02","date_gmt":"2021-05-23T03:28:02","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/?post_type=chapter&p=91"},"modified":"2022-01-11T00:23:36","modified_gmt":"2022-01-11T05:23:36","slug":"truncated-studies-was-the-trial-stopped-early-for-overwhelming-evidence-of-benefit-or-futility","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/truncated-studies-was-the-trial-stopped-early-for-overwhelming-evidence-of-benefit-or-futility\/","title":{"raw":"Truncated studies: Was the trial stopped early for \u201coverwhelming\u201d evidence of benefit or futility?","rendered":"Truncated studies: Was the trial stopped early for \u201coverwhelming\u201d evidence of benefit or futility?"},"content":{"raw":"Studies may be stopped early for efficacy as part of an ethical obligation to not expose participants to less effective treatment (or placebo) any longer than necessary. In other words, once it is sufficiently clear that an intervention is efficacious, there is reason to end the trial.\r\n\r\nHowever stopping early runs the risk of overestimating the effect size of the intervention. The estimate of effect will randomly vary around the true effect over time (with more fluctuation with fewer events early in the trial), so interim looks may lead to premature stop due an exaggerated estimate of the true effect size.\r\n\r\nConsider the following simulated trial where there is no true difference between the groups (i.e. [pb_glossary id=\"109\"]RR[\/pb_glossary] = 1.0):\r\n\r\n[caption id=\"attachment_781\" align=\"alignnone\" width=\"1024\"]\"\" Graph 2. Relative risk vs. number of events in a simulated trial. Created via Microsoft Excel using the RAND function to generate randomized event-data for two groups.[\/caption]\r\n\r\nAs depicted in Graph 2 above, there is random deviation from the true effect as events accumulate. If the trial had interim analyses for benefit every 100 events, and the threshold for statistical significance was kept at\u00a0 the standard p<\/a><0.05 without accounting for interim looks, then the trial may have stopped at 100 events when the [pb_glossary id=\"109\"]RR[\/pb_glossary] was 1.3, which we know to be an exaggeration of the true effect ([pb_glossary id=\"109\"]RR[\/pb_glossary] = 1.0, i.e. no effect).\r\n\r\nAs a simplified example, imagine studying a chess player and trying to assess if they are an above-average player (and by what margin) by judging their win percentage. One approach is to wait 50 matches, then assess their win percentage and judge accordingly. However, this could waste time as it might be unnecessary to wait that long if they are quite skilled (e.g. winning 90% of their first 10 games). So instead there could be an assessment of skill every 5 matches (up to a maximum of 50 matches). If they seem sufficiently impressive at one of these midpoint assessments, then the observation could be stopped. While this might save time, it also has a risk: if by pure chance the player goes on a win streak, then the observation is likely to end early. Even if our player is truly above-average in skill, an early stop is most likely to occur when they are on such a hot streak, consequently introducing [pb_glossary id=\"193\"]bias[\/pb_glossary] into our assessment (e.g. assessing their win probability to be 80% due to the win streak, when in fact it is only 60%).\r\n\r\nThis is the major concern with stopping rules: there is a systematic tendency for an early stop to be an overestimation. While such precautions cannot prevent [pb_glossary id=\"193\"]bias[\/pb_glossary] towards overestimation, they can help reduce the extent of this [pb_glossary id=\"193\"]bias[\/pb_glossary], as discussed below.\r\n

Checklist Questions<\/h1>\r\n\r\n\r\n\r\n\r\n\r\n
Was there a predefined interim analysis plan with a stopping rule?<\/td>\r\n<\/tr>\r\n
Did the stopping rule involve few interim looks and a stringent p-value<\/a> (e.g. <0.001)?<\/td>\r\n<\/tr>\r\n
Did enough endpoint events occur?<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n

Was there a predefined interim analysis plan with a stopping rule?<\/h1>\r\n
If there is no pre-planned stopping rule then there is no assurance that sufficient safeguards were in place to minimize [pb_glossary id=\"193\"]bias[\/pb_glossary] from early stops.<\/div>\r\n
E.g.<\/span> In JUPITER (Ridker PM et al.<\/a>), a [pb_glossary id=\"704\"]RCT[\/pb_glossary] <\/span><\/span>of rosuvastatin vs. placebo in a <\/span>highly-selected<\/span> primary cardiovascular prevention population, the pre-planned stopping rule was mentioned, though poorly described, in an early report: \u201c<\/span><\/span>Frequency of interim efficacy analyses and rules for early trial termination have been prespecified and approved by all members of this board.\u201d<\/span><\/span>\u00a0<\/span><\/em><\/div>\r\n

Did the\u00a0<\/span><\/span>stopping\u00a0<\/span><\/span>rule involve few interim looks and a stringent p-value (<\/span>e.g.<\/span>\u00a0<0.001)?<\/span><\/span>\u00a0<\/span><\/h1>\r\n
\r\n\r\nAs the number of interim looks increases, then the probability of finding a false positive or overestimation also increases. This can be mitigated by (1) minimizing the number of interim looks and (2) having a stricter threshold for statistical significance that accounts for these multiple interim analyses.\r\n\r\nSome common interim analysis strategies used (Schulz KF et al.<\/a>) are:\r\nPocock<\/strong>: To keep the overall trial p-value<\/a> threshold (alpha) = 0.05, the number of interim analyses are pre-defined & all have the same adjusted statistical significance threshold (i.e. p<0.029 for 2 planned analyses, p<0.016 for 5 planned analyses, and so forth).\r\nPeto<\/strong>: Assign the final analysis p-value<\/a> threshold = 0.05 (like in a conventional trial), but have a more stringent threshold (i.e. p<\/a><0.001) for the interim analyses.\r\nO'Brien-Fleming<\/strong>: Begin with stringent interim analyses that start conservatively and then successively ease as they approach the final analysis (e.g. for 3 interim analyses & a final analysis, sequence of p-value<\/a> thresholds 0.0001, 0.004, 0.019, 0.043)\r\nLan-DeMets:<\/strong> An adaptable approach where the significance level changes and analysis timing changes in accordance to previously observed information.\r\n\r\n<\/div>\r\n
\r\n
E.g.\u00a0JUPITER (Ridker PM et al.<\/a>) <\/span><\/span>was stopped after the first of two interim analyses using \u201cO\u2019Brien-Fleming stopping boundaries determined by means of the Lan-DeMets approach,\u201d (which requires a p-value<\/a> <0.005). The actual p-value<\/a> for the [pb_glossary id=\"1517\"]primary endpoint[\/pb_glossary] was <0.00001.\u00a0<\/em><\/div>\r\n<\/div>\r\n

Did enough endpoint events occur?<\/span><\/span>\u00a0<\/span><\/h1>\r\n
\r\n\r\nTrials stopped early for benefit exaggerate the [pb_glossary id=\"119\"]relative effect[\/pb_glossary] of an intervention by an average 29% compared with trials that conclude as planned (Bassler D et al.<\/a>). As events accumulate, the fluctuations in effect size measures will become smaller and there will be less risk of [pb_glossary id=\"193\"]bias[\/pb_glossary] (see graph above). Optimally \u2265500 events (<\/span>Bassler D et al.<\/a>) should occur before stopping, after which the exaggeration decreases to an average of 12%.<\/span>\r\n\r\n<\/div>\r\n
\r\n\r\nFor these reasons, skepticism is warranted for any [pb_glossary id=\"1111\"]relative risk reduction (RRR)[\/pb_glossary] \u226550% generated in truncated trials with <100 events (Pocock SJ et al., Montori VM et al.<\/a>). The larger the number of events and the more plausible the [pb_glossary id=\"1111\"]RRR[\/pb_glossary] (e.g. ~20-30% is typical for the impact of cardiovascular pharmacotherapy on cardiovascular events), the more believable the results.\r\n\r\n<\/div>\r\n
\r\n
E.g. In JUPITER (Ridker PM et al.<\/a>)<\/span><\/span>, 393 [pb_glossary id=\"290\"]primary[\/pb_glossary] ([pb_glossary id=\"359\"]composite[\/pb_glossary]) endpoint events occurred between the two groups by the interim analysis. The [pb_glossary id=\"1111\"]RRR[\/pb_glossary] for the [pb_glossary id=\"1517\"]primary endpoint[\/pb_glossary] was 44%, and the [pb_glossary id=\"1111\"]RRRs[\/pb_glossary] for individual components ranged from 18-54%.\u00a0<\/em><\/div>\r\n<\/div>","rendered":"

Studies may be stopped early for efficacy as part of an ethical obligation to not expose participants to less effective treatment (or placebo) any longer than necessary. In other words, once it is sufficiently clear that an intervention is efficacious, there is reason to end the trial.<\/p>\n

However stopping early runs the risk of overestimating the effect size of the intervention. The estimate of effect will randomly vary around the true effect over time (with more fluctuation with fewer events early in the trial), so interim looks may lead to premature stop due an exaggerated estimate of the true effect size.<\/p>\n

Consider the following simulated trial where there is no true difference between the groups (i.e. RR<\/a> = 1.0):<\/p>\n

\"\"
Graph 2. Relative risk vs. number of events in a simulated trial. Created via Microsoft Excel using the RAND function to generate randomized event-data for two groups.<\/figcaption><\/figure>\n

As depicted in Graph 2 above, there is random deviation from the true effect as events accumulate. If the trial had interim analyses for benefit every 100 events, and the threshold for statistical significance was kept at\u00a0 the standard p<\/a><0.05 without accounting for interim looks, then the trial may have stopped at 100 events when the RR<\/a> was 1.3, which we know to be an exaggeration of the true effect (RR<\/a> = 1.0, i.e. no effect).<\/p>\n

As a simplified example, imagine studying a chess player and trying to assess if they are an above-average player (and by what margin) by judging their win percentage. One approach is to wait 50 matches, then assess their win percentage and judge accordingly. However, this could waste time as it might be unnecessary to wait that long if they are quite skilled (e.g. winning 90% of their first 10 games). So instead there could be an assessment of skill every 5 matches (up to a maximum of 50 matches). If they seem sufficiently impressive at one of these midpoint assessments, then the observation could be stopped. While this might save time, it also has a risk: if by pure chance the player goes on a win streak, then the observation is likely to end early. Even if our player is truly above-average in skill, an early stop is most likely to occur when they are on such a hot streak, consequently introducing bias<\/a> into our assessment (e.g. assessing their win probability to be 80% due to the win streak, when in fact it is only 60%).<\/p>\n

This is the major concern with stopping rules: there is a systematic tendency for an early stop to be an overestimation. While such precautions cannot prevent bias<\/a> towards overestimation, they can help reduce the extent of this bias<\/a>, as discussed below.<\/p>\n

Checklist Questions<\/h1>\n\n\n\n\n\n
Was there a predefined interim analysis plan with a stopping rule?<\/td>\n<\/tr>\n
Did the stopping rule involve few interim looks and a stringent p-value<\/a> (e.g. <0.001)?<\/td>\n<\/tr>\n
Did enough endpoint events occur?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Was there a predefined interim analysis plan with a stopping rule?<\/h1>\n
If there is no pre-planned stopping rule then there is no assurance that sufficient safeguards were in place to minimize bias<\/a> from early stops.<\/div>\n
E.g.<\/span> In JUPITER (Ridker PM et al.<\/a>), a RCT<\/a> <\/span><\/span>of rosuvastatin vs. placebo in a <\/span>highly-selected<\/span> primary cardiovascular prevention population, the pre-planned stopping rule was mentioned, though poorly described, in an early report: \u201c<\/span><\/span>Frequency of interim efficacy analyses and rules for early trial termination have been prespecified and approved by all members of this board.\u201d<\/span><\/span>\u00a0<\/span><\/em><\/div>\n

Did the\u00a0<\/span><\/span>stopping\u00a0<\/span><\/span>rule involve few interim looks and a stringent p-value (<\/span>e.g.<\/span>\u00a0<0.001)?<\/span><\/span>\u00a0<\/span><\/h1>\n
\n

As the number of interim looks increases, then the probability of finding a false positive or overestimation also increases. This can be mitigated by (1) minimizing the number of interim looks and (2) having a stricter threshold for statistical significance that accounts for these multiple interim analyses.<\/p>\n

Some common interim analysis strategies used (Schulz KF et al.<\/a>) are:
\nPocock<\/strong>: To keep the overall trial p-value<\/a> threshold (alpha) = 0.05, the number of interim analyses are pre-defined & all have the same adjusted statistical significance threshold (i.e. p<0.029 for 2 planned analyses, p<0.016 for 5 planned analyses, and so forth).
\nPeto<\/strong>: Assign the final analysis p-value<\/a> threshold = 0.05 (like in a conventional trial), but have a more stringent threshold (i.e. p<\/a><0.001) for the interim analyses.
\nO’Brien-Fleming<\/strong>: Begin with stringent interim analyses that start conservatively and then successively ease as they approach the final analysis (e.g. for 3 interim analyses & a final analysis, sequence of p-value<\/a> thresholds 0.0001, 0.004, 0.019, 0.043)
\nLan-DeMets:<\/strong> An adaptable approach where the significance level changes and analysis timing changes in accordance to previously observed information.<\/p>\n<\/div>\n

\n
E.g.\u00a0JUPITER (Ridker PM et al.<\/a>) <\/span><\/span>was stopped after the first of two interim analyses using \u201cO\u2019Brien-Fleming stopping boundaries determined by means of the Lan-DeMets approach,\u201d (which requires a p-value<\/a> <0.005). The actual p-value<\/a> for the primary endpoint<\/a> was <0.00001.\u00a0<\/em><\/div>\n<\/div>\n

Did enough endpoint events occur?<\/span><\/span>\u00a0<\/span><\/h1>\n
\n

Trials stopped early for benefit exaggerate the relative effect<\/a> of an intervention by an average 29% compared with trials that conclude as planned (Bassler D et al.<\/a>). As events accumulate, the fluctuations in effect size measures will become smaller and there will be less risk of bias<\/a> (see graph above). Optimally \u2265500 events (<\/span>Bassler D et al.<\/a>) should occur before stopping, after which the exaggeration decreases to an average of 12%.<\/span><\/p>\n<\/div>\n