{"id":93,"date":"2021-05-22T23:30:54","date_gmt":"2021-05-23T03:30:54","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/?post_type=chapter&#038;p=93"},"modified":"2022-01-12T23:28:37","modified_gmt":"2022-01-13T04:28:37","slug":"search","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/chapter\/search\/","title":{"raw":"Search","rendered":"Search"},"content":{"raw":"A comprehensive search is at the core of all [pb_glossary id=\"1099\"]systematic reviews[\/pb_glossary], and is essential to ensure that all relevant trials were included. A search that is not sufficiently thorough will be more vulnerable to [pb_glossary id=\"102\"]publication bias[\/pb_glossary]. In the case of [pb_glossary id=\"102\"]publication bias[\/pb_glossary] \"an ounce of prevention is worth a pound of cure\" since the tools available to identify and adjust for [pb_glossary id=\"102\"]publication bias[\/pb_glossary] are insensitive and cannot discriminate between [pb_glossary id=\"102\"]publication bias[\/pb_glossary] and alternative causes for [pb_glossary id=\"604\"]small-study effects[\/pb_glossary]. Other considerations (such as when the search was last completed) are also necessary to ensure the search is sufficiently complete.\r\n<h1>Checklist Questions<\/h1>\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\" border=\"0\">\r\n<tbody>\r\n<tr>\r\n<td style=\"width: 100%\">Were a reasonable number of relevant databases searched?<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"width: 100%\">When was the search conducted? Is it likely there have been subsequent publications that may alter the results?<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"width: 100%\">Was a sufficient effort made to find unpublished studies (or unreported results of published studies)?<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"width: 100%\">Were sources of additional published\/unpublished data sought out?<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h1><span class=\"TextRun SCXW95017315 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW95017315 BCX9\">Databases of published literature:<\/span><\/span><span class=\"TextRun SCXW95017315 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW95017315 BCX9\"> Were a reasonable number of relevant databases searched?<\/span><\/span><\/h1>\r\n<span class=\"TextRun SCXW206533241 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW206533241 BCX9\">It is important to search multiple databases to maximize the identification of all relevant studies, as no single database includes all studies. One study (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Royle P et al.<\/a>) compared three major databases to a set of relevant studies established by searching twenty-six additional databases:<\/span><\/span>\r\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 90px\" border=\"0\"><caption>Table 11. Proportion of relevant trials identified by different databases.<\/caption>\r\n<tbody>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 24.2568%;height: 18px\"><strong>Database<\/strong><\/td>\r\n<td style=\"width: 75.7432%;height: 18px\"><strong>Proportion of relevant trials identified<\/strong><\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 24.2568%;height: 18px\">MEDLINE<\/td>\r\n<td style=\"width: 75.7432%;height: 18px\">69%<\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 24.2568%;height: 18px\">EMBASE<\/td>\r\n<td style=\"width: 75.7432%;height: 18px\">65%<\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 24.2568%;height: 18px\">CENTRAL<\/td>\r\n<td style=\"width: 75.7432%;height: 18px\">79%<\/td>\r\n<\/tr>\r\n<tr style=\"height: 18px\">\r\n<td style=\"width: 24.2568%;height: 18px\">Combining all three<\/td>\r\n<td style=\"width: 75.7432%;height: 18px\">97%<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<span class=\"TextRun SCXW206533241 BCX9\" lang=\"EN-US\" style=\"text-align: initial;font-size: 14pt\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW206533241 BCX9\">The optimal selection of which (and how many) databases to search will depend on the discipline, topic area, and type of intervention. For example, studies evaluating nursing and physiotherapy interventions should at minimum include <a href=\"https:\/\/www.ebsco.com\/products\/research-databases\/cinahl-database\" target=\"_blank\" rel=\"noopener\">CINAHL<\/a> and <a href=\"https:\/\/pedro.org.au\/\" target=\"_blank\" rel=\"noopener\">PEDro<\/a>, respectively. A good rule-of-thumb is to search MEDLINE and at least 1-2 other topic-specific databases (e.g. EMBASE and CENTRAL for pharmacotherapy studies).<\/span><\/span>\r\n<h1>Timeframe: When was the search conducted? Is it likely there have been subsequent publications that may alter the results?<\/h1>\r\nThere are no strict rules as to how long is too long before a review becomes outdated, as this largely depends on the rate of evidence generation in a given field or topic area. It is important to consider the rate at which new publications are being added to the literature (i.e. considering if it is a \"hot\" topic) and whether the results would likely be sensitive to new publications (i.e. a [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] with low-to-moderate certainty). If there are already several large high-quality trials showing consistent results it is less likely that any new literature would substantially change results.\r\n<div class=\"textbox shaded\"><em>\"Hot\" topic: A living [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] (i.e. a [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] that is actively updated with new evidence) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Siemieniuk RA et al.<\/a>) of drug treatments for COVID-19 illustrates an instance of rapidly changing evidence. The first version, published July 2020, included 32 [pb_glossary id=\"704\"]RCTs[\/pb_glossary] and evaluated 17 therapies. The fourth version, published in March 2021, included 196 trials and evaluated 27 therapies. Under such circumstances, [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary] become quickly outdated.<\/em><\/div>\r\n<div class=\"textbox shaded\"><em>\"Cold\" topic: The evidence surrounding the cardiovascular risk associated with rosiglitazone has changed minimally for over a decade. [pb_glossary id=\"1101\"]Meta-analyses[\/pb_glossary] from 2007 and 2010 (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Nissen SE et al. 2007, 2010<\/a>) demonstrated increases in the risk of myocardial infarction with rosiglitazone. Both reviews had large patient sample sizes (27,847 and 35, 531 respectively), a factor which weighed in favor of their persisting relevance. As such, the evidence on this topic has remained largely unchanged since those reviews.<\/em><\/div>\r\n<h1><span class=\"TextRun SCXW81926157 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW81926157 BCX9\">Grey literature:<\/span><\/span><span class=\"TextRun SCXW81926157 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW81926157 BCX9\"> Was a sufficient effort made to find unpublished studies (or unreported results of published studies)?<\/span><\/span><\/h1>\r\n<div style=\"font-weight: 400\">\r\n\r\nA thorough search of unpublished literature aims to minimize the effects of [pb_glossary id=\"102\"]publication bias[\/pb_glossary].\r\n<div class=\"textbox shaded\">\r\n\r\n<em>E.g. In a [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Siu JT et al.<\/a>) of N-acetylcysteine for non-acetaminophen-related acute liver failure the authors searched all of:<\/em>\r\n<ul>\r\n \t<li><em>The following databases (without language restrictions): Cochrane Hepato\u2010Biliary Group Controlled Trials Register, the Cochrane Central Register of Controlled Trials, MEDLINE Ovid, Embase Ovid, LILACS, Science Citation Index Expanded, and Conference Proceedings Citation Index \u2013 Science <\/em><\/li>\r\n \t<li><em>The reference lists of all included studies and relevant papers<\/em><\/li>\r\n \t<li><em>The following online clinical trial registries: ClinicalTrial.gov, European Medicines Agency, World Health Organization International Clinical Trial Registry Platform, the Food and Drug Administration, and pharmaceutical company sources for ongoing or unpublished trials<\/em><\/li>\r\n<\/ul>\r\n<em>The authors of relevant papers were also contacted to inquire regarding any further published or unpublished work.<\/em>\r\n\r\n<\/div>\r\n<h2><strong>Why is publication bias so concerning?<\/strong><\/h2>\r\n<div style=\"font-weight: 400\">\r\n\r\nStudies with statistically significant results (\u201cpositive\u201d studies) are twice as likely to get published, and will typically get published faster (by a median of 1.3 years in one study) compared to trials with statistically non-significant results (\"neutral\" studies) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Hopewell S et al.<\/a>, <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Ioannidis JP<\/a>).\r\n<ul>\r\n \t<li>Published trials have a 15% larger estimate of effect compared to unpublished trials (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">McAuley L et al.<\/a>)<\/li>\r\n \t<li>Although more common with industry-funded trials, government-funded studies are still prone to publication bias (32% vs. 18% unpublished 5 years after completion) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Jones CW et al.<\/a>)<\/li>\r\n \t<li>In one study, 90-98% of [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary] with very large effects observed in early trials became substantially smaller once subsequent studies became available (e.g. median [pb_glossary id=\"103\"]odds ratio[\/pb_glossary] decreased from ~11 to ~4 after more trials were added to the first trial) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Pereira TV et al.<\/a>)<\/li>\r\n \t<li>In one study of 42 [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary], in 93% of cases the addition of unpublished FDA outcome data changed the efficacy summary estimate (either increased or decreased) compared to the [pb_glossary id=\"1101\"]meta-analysis[\/pb_glossary] based purely on published outcome data (<a style=\"font-size: 14pt\" href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Hart B et al.<\/a><span style=\"font-size: 14pt\">)<\/span><\/li>\r\n<\/ul>\r\n<\/div>\r\n<div style=\"font-weight: 400\">\r\n\r\n<span style=\"text-decoration: underline\">Bottom line:<\/span> [pb_glossary id=\"1101\"]Meta-analyses[\/pb_glossary] of only published trials will overestimate the effects of drugs and other interventions, especially when [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary] are conducted \"earlier on\" (before the neutral trials get published). Consequently, there is likely a greater risk of [pb_glossary id=\"102\"]publication bias[\/pb_glossary] in [pb_glossary id=\"1101\"]meta-analyses[\/pb_glossary] based on a few small studies.\r\n<div class=\"textbox shaded\">\r\n\r\n<em>E.g. A review (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turner EH et al.<\/a>) of antidepressants found that 94% of published trials demonstrated a statistically significant difference with respect to the [pb_glossary id=\"1517\"]primary outcome[\/pb_glossary]. However, when combined with unpublished FDA review data, only 51% of total trials demonstrated a statistically significant difference with respect to the [pb_glossary id=\"1517\"]primary outcome[\/pb_glossary]. Including only published studies increased the [pb_glossary id=\"119\"]relative effect[\/pb_glossary] size by 32%.<\/em>\r\n\r\n<img class=\"alignnone size-full wp-image-1733\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/05\/GIF-at-new-speed.gif\" alt=\"\" width=\"720\" height=\"540\" \/>\r\n<em>Gif 1. [pb_glossary id=\"102\"]Publication bias[\/pb_glossary] among antidepressant trials as reported by <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turner EH et al.<\/a> GIF created by Turner EH.<\/em>\r\n\r\n<\/div>\r\n<\/div>\r\n<\/div>","rendered":"<p>A comprehensive search is at the core of all <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1099\">systematic reviews<\/a>, and is essential to ensure that all relevant trials were included. A search that is not sufficiently thorough will be more vulnerable to <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a>. In the case of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a> &#8220;an ounce of prevention is worth a pound of cure&#8221; since the tools available to identify and adjust for <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a> are insensitive and cannot discriminate between <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a> and alternative causes for <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_604\">small-study effects<\/a>. Other considerations (such as when the search was last completed) are also necessary to ensure the search is sufficiently complete.<\/p>\n<h1>Checklist Questions<\/h1>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%\">\n<tbody>\n<tr>\n<td style=\"width: 100%\">Were a reasonable number of relevant databases searched?<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 100%\">When was the search conducted? Is it likely there have been subsequent publications that may alter the results?<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 100%\">Was a sufficient effort made to find unpublished studies (or unreported results of published studies)?<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 100%\">Were sources of additional published\/unpublished data sought out?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h1><span class=\"TextRun SCXW95017315 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW95017315 BCX9\">Databases of published literature:<\/span><\/span><span class=\"TextRun SCXW95017315 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW95017315 BCX9\"> Were a reasonable number of relevant databases searched?<\/span><\/span><\/h1>\n<p><span class=\"TextRun SCXW206533241 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW206533241 BCX9\">It is important to search multiple databases to maximize the identification of all relevant studies, as no single database includes all studies. One study (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Royle P et al.<\/a>) compared three major databases to a set of relevant studies established by searching twenty-six additional databases:<\/span><\/span><\/p>\n<table class=\"grid\" style=\"border-collapse: collapse;width: 100%;height: 90px\">\n<caption>Table 11. Proportion of relevant trials identified by different databases.<\/caption>\n<tbody>\n<tr style=\"height: 18px\">\n<td style=\"width: 24.2568%;height: 18px\"><strong>Database<\/strong><\/td>\n<td style=\"width: 75.7432%;height: 18px\"><strong>Proportion of relevant trials identified<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 24.2568%;height: 18px\">MEDLINE<\/td>\n<td style=\"width: 75.7432%;height: 18px\">69%<\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 24.2568%;height: 18px\">EMBASE<\/td>\n<td style=\"width: 75.7432%;height: 18px\">65%<\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 24.2568%;height: 18px\">CENTRAL<\/td>\n<td style=\"width: 75.7432%;height: 18px\">79%<\/td>\n<\/tr>\n<tr style=\"height: 18px\">\n<td style=\"width: 24.2568%;height: 18px\">Combining all three<\/td>\n<td style=\"width: 75.7432%;height: 18px\">97%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span class=\"TextRun SCXW206533241 BCX9\" lang=\"EN-US\" style=\"text-align: initial;font-size: 14pt\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW206533241 BCX9\">The optimal selection of which (and how many) databases to search will depend on the discipline, topic area, and type of intervention. For example, studies evaluating nursing and physiotherapy interventions should at minimum include <a href=\"https:\/\/www.ebsco.com\/products\/research-databases\/cinahl-database\" target=\"_blank\" rel=\"noopener\">CINAHL<\/a> and <a href=\"https:\/\/pedro.org.au\/\" target=\"_blank\" rel=\"noopener\">PEDro<\/a>, respectively. A good rule-of-thumb is to search MEDLINE and at least 1-2 other topic-specific databases (e.g. EMBASE and CENTRAL for pharmacotherapy studies).<\/span><\/span><\/p>\n<h1>Timeframe: When was the search conducted? Is it likely there have been subsequent publications that may alter the results?<\/h1>\n<p>There are no strict rules as to how long is too long before a review becomes outdated, as this largely depends on the rate of evidence generation in a given field or topic area. It is important to consider the rate at which new publications are being added to the literature (i.e. considering if it is a &#8220;hot&#8221; topic) and whether the results would likely be sensitive to new publications (i.e. a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analysis<\/a> with low-to-moderate certainty). If there are already several large high-quality trials showing consistent results it is less likely that any new literature would substantially change results.<\/p>\n<div class=\"textbox shaded\"><em>&#8220;Hot&#8221; topic: A living <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analysis<\/a> (i.e. a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analysis<\/a> that is actively updated with new evidence) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Siemieniuk RA et al.<\/a>) of drug treatments for COVID-19 illustrates an instance of rapidly changing evidence. The first version, published July 2020, included 32 <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_704\">RCTs<\/a> and evaluated 17 therapies. The fourth version, published in March 2021, included 196 trials and evaluated 27 therapies. Under such circumstances, <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analyses<\/a> become quickly outdated.<\/em><\/div>\n<div class=\"textbox shaded\"><em>&#8220;Cold&#8221; topic: The evidence surrounding the cardiovascular risk associated with rosiglitazone has changed minimally for over a decade. <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">Meta-analyses<\/a> from 2007 and 2010 (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Nissen SE et al. 2007, 2010<\/a>) demonstrated increases in the risk of myocardial infarction with rosiglitazone. Both reviews had large patient sample sizes (27,847 and 35, 531 respectively), a factor which weighed in favor of their persisting relevance. As such, the evidence on this topic has remained largely unchanged since those reviews.<\/em><\/div>\n<h1><span class=\"TextRun SCXW81926157 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW81926157 BCX9\">Grey literature:<\/span><\/span><span class=\"TextRun SCXW81926157 BCX9\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW81926157 BCX9\"> Was a sufficient effort made to find unpublished studies (or unreported results of published studies)?<\/span><\/span><\/h1>\n<div style=\"font-weight: 400\">\n<p>A thorough search of unpublished literature aims to minimize the effects of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a>.<\/p>\n<div class=\"textbox shaded\">\n<p><em>E.g. In a <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analysis<\/a> (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Siu JT et al.<\/a>) of N-acetylcysteine for non-acetaminophen-related acute liver failure the authors searched all of:<\/em><\/p>\n<ul>\n<li><em>The following databases (without language restrictions): Cochrane Hepato\u2010Biliary Group Controlled Trials Register, the Cochrane Central Register of Controlled Trials, MEDLINE Ovid, Embase Ovid, LILACS, Science Citation Index Expanded, and Conference Proceedings Citation Index \u2013 Science <\/em><\/li>\n<li><em>The reference lists of all included studies and relevant papers<\/em><\/li>\n<li><em>The following online clinical trial registries: ClinicalTrial.gov, European Medicines Agency, World Health Organization International Clinical Trial Registry Platform, the Food and Drug Administration, and pharmaceutical company sources for ongoing or unpublished trials<\/em><\/li>\n<\/ul>\n<p><em>The authors of relevant papers were also contacted to inquire regarding any further published or unpublished work.<\/em><\/p>\n<\/div>\n<h2><strong>Why is publication bias so concerning?<\/strong><\/h2>\n<div style=\"font-weight: 400\">\n<p>Studies with statistically significant results (\u201cpositive\u201d studies) are twice as likely to get published, and will typically get published faster (by a median of 1.3 years in one study) compared to trials with statistically non-significant results (&#8220;neutral&#8221; studies) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Hopewell S et al.<\/a>, <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Ioannidis JP<\/a>).<\/p>\n<ul>\n<li>Published trials have a 15% larger estimate of effect compared to unpublished trials (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">McAuley L et al.<\/a>)<\/li>\n<li>Although more common with industry-funded trials, government-funded studies are still prone to publication bias (32% vs. 18% unpublished 5 years after completion) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Jones CW et al.<\/a>)<\/li>\n<li>In one study, 90-98% of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analyses<\/a> with very large effects observed in early trials became substantially smaller once subsequent studies became available (e.g. median <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_103\">odds ratio<\/a> decreased from ~11 to ~4 after more trials were added to the first trial) (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Pereira TV et al.<\/a>)<\/li>\n<li>In one study of 42 <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analyses<\/a>, in 93% of cases the addition of unpublished FDA outcome data changed the efficacy summary estimate (either increased or decreased) compared to the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analysis<\/a> based purely on published outcome data (<a style=\"font-size: 14pt\" href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\">Hart B et al.<\/a><span style=\"font-size: 14pt\">)<\/span><\/li>\n<\/ul>\n<\/div>\n<div style=\"font-weight: 400\">\n<p><span style=\"text-decoration: underline\">Bottom line:<\/span> <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">Meta-analyses<\/a> of only published trials will overestimate the effects of drugs and other interventions, especially when <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analyses<\/a> are conducted &#8220;earlier on&#8221; (before the neutral trials get published). Consequently, there is likely a greater risk of <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">publication bias<\/a> in <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1101\">meta-analyses<\/a> based on a few small studies.<\/p>\n<div class=\"textbox shaded\">\n<p><em>E.g. A review (<a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turner EH et al.<\/a>) of antidepressants found that 94% of published trials demonstrated a statistically significant difference with respect to the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1517\">primary outcome<\/a>. However, when combined with unpublished FDA review data, only 51% of total trials demonstrated a statistically significant difference with respect to the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_1517\">primary outcome<\/a>. Including only published studies increased the <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_119\">relative effect<\/a> size by 32%.<\/em><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1733\" src=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-content\/uploads\/sites\/1246\/2021\/05\/GIF-at-new-speed.gif\" alt=\"\" width=\"720\" height=\"540\" \/><br \/>\n<em>Gif 1. <a class=\"glossary-term\" aria-haspopup=\"dialog\" aria-describedby=\"definition\" href=\"#term_93_102\">Publication bias<\/a> among antidepressant trials as reported by <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/references\/\" target=\"_blank\" rel=\"noopener\">Turner EH et al.<\/a> GIF created by Turner EH.<\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"glossary\"><span class=\"screen-reader-text\" id=\"definition\">definition<\/span><template id=\"term_93_1099\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_1099\"><div tabindex=\"-1\"><p>A review that systematically identifies all potentially relevant studies on a research question. The aggregate of studies is then evaluated with respect to factors such as risk of bias of individual studies or heterogeneity among results. The qualitative combination of results is a systematic review.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_102\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_102\"><div tabindex=\"-1\"><p>Refers to a systematic tendency for results to be published based upon the direction or statistical significance of the results. This results in bias when aggregating evidence if methods are more likely to include published literature than unpublished literature.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_604\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_604\"><div tabindex=\"-1\"><p>A tendency for smaller published studies to demonstrate a larger effect size than larger published studies. One possible cause is publication bias. However, other possible causes include systematic differences between smaller and larger studies (e.g. stricter enrolment criteria, adherence and\/or follow-up in smaller studies, more pragmatic design in larger studies).<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_1101\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_1101\"><div tabindex=\"-1\"><p>A meta-analysis is a quantitative combination of the data obtained in a systematic review.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_704\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_704\"><div tabindex=\"-1\"><p>Randomized controlled trials are those in which participants are randomly allocated to two or more groups which are given different treatments.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_103\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_103\"><div tabindex=\"-1\"><p>Odds ratios are the ratio of odds (events divided by non-events) in the intervention group to the odds in the comparator group. For example, if the odds of an event in the treatment group is 0.2 and the odds in the comparator group is 0.1, then the OR is 2 (0.2\/0.1). See <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for a more detailed discussion.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_1517\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_1517\"><div tabindex=\"-1\"><p>A primary outcome is an outcome from which trial design choices are based (e.g. sample size calculations). Primary outcomes are not necessarily the most important outcomes.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><template id=\"term_93_119\"><div class=\"glossary__definition\" role=\"dialog\" data-id=\"term_93_119\"><div tabindex=\"-1\"><p>Calculates the effect of an intervention via a fractional comparison with the comparator group (i.e. intervention group measure \u00f7 comparator group measure). Used for binary outcomes. Relative risk, odds ratio, or hazards ratio are all expressions of relative effect. For example, if the risk of developing neuropathy was 1% in the treatment group and 2% in the comparator group, then the relative risk is 0.5 (1 \u00f7 2). See the Absolute Risk Differences and Relative Measures of Effect discussion <a href=\"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/back-matter\/appendix\/\">here<\/a> for more information.<\/p>\n<\/div><button><span aria-hidden=\"true\">&times;<\/span><span class=\"screen-reader-text\">Close definition<\/span><\/button><\/div><\/template><\/div>","protected":false},"author":1318,"menu_order":1,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-93","chapter","type-chapter","status-publish","hentry"],"part":25,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapters\/93","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/users\/1318"}],"version-history":[{"count":26,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapters\/93\/revisions"}],"predecessor-version":[{"id":1894,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapters\/93\/revisions\/1894"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/parts\/25"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapters\/93\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/media?parent=93"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/pressbooks\/v2\/chapter-type?post=93"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/contributor?post=93"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/rickyturgeon\/wp-json\/wp\/v2\/license?post=93"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}