Practice recommendations
- When consulting narrative review articles, carefully consider the possibility of citation bias and whether recommendations are based on patient-oriented and not disease-oriented outcomes.
- Consult credible evidence-based resources such as Cochrane before making changes in practice suggested by narrative reviews.
ABSTRACT
Background: Citations in scientific articles may tend to favor the views presented. We studied whether there is such reference bias in narrative review articles that discuss interventions against house dust mites for people with asthma.
Design: Systematic review of reviews identified in a Medline search that expressed an opinion about the clinical effects of physical or chemical intervention methods.
Main outcome measure: Positive bias was judged to have occurred if the reference list contained a higher proportion of trial references with significant results than among all trials available to the authors (published 2 years or more prior to the review).
Results: Seventy reviews were included, of which 63 (90%) recommended physical interventions. Forty-six reviews had trial references, 4 of these only to chemical interventions. In the remaining 42 reviews, reference bias was detected (P=2 x 10-8). The most quoted trial had only 7 patients per group, its claimed significant result was probably erroneous, and it did not report a clinical outcome. Intervention recommendations were often based on nonrandomized evidence, and the most quoted nonrandomized controlled study had included only 10 patients per group but claimed very positive results.
Conclusion: The narrative review articles were severely biased, and their positive intervention recommendations are at variance with the systematic Cochrane Review on this topic and a recent very large trial of physical intervention, both of which failed to find an effect.
Trial reports and narrative review articles sometimes favor references supporting the views of the authors, lending credence to a particular treatment1 or hypothesis.2,3 This reference bias may render the conclusions of an article less reliable. Such was the finding in our study of narrative review articles discussing interventions against house dust mites for people with asthma.
Systematic reviews are a more reliable source of information for busy clinicians than narrative reviews, but they can also be problematic. An assessment of papers on asthma, based on a validated tool, showed that 40 of 50 systematic reviews on prevention and treatment had serious or extensive flaws.4 Cochrane systematic reviews on asthma were more rigorous and better reported than systematic reviews published in peer-reviewed, paper-based journals,4 probably because Cochrane Reviews are conducted according to a set of standard methods5 aimed at minimizing bias in the reviews, and because they usually include only randomized trials.
As the index review for our study, we chose the Cochrane Review on house dust mites control measures for asthma,6 to allow us to judge whether the recommendations in narrative reviews reflected fairly the available, reliable scientific evidence.
Methods
Inclusion criteria. Narrative reviews—that is, reviews that did not have a methods section with a search strategy for relevant research papers—were eligible if they expressed an opinion about the clinical effects of chemical or physical interventions aimed at reducing exposure to house dust mite antigens in the homes of mite-sensitive patients with asthma.
Search strategy. A Medline search selected review articles from 1966 to July 2002 using the terms review* and mite* and asthma*, with no language restrictions. Additional review articles could have been found in other databases, but we chose Medline since it is freely available and likely preferred by physicians.
Data collection. Abstracts identified in the Medline search were read by one of us (LMS), who excluded articles that clearly were not reviews or that reviewed aspects other than treatments (eg, causes of asthma only). Both authors read the remaining articles in full, and conducted another round of exclusion.
Data extraction was performed independently and results were compared. Disagreements were resolved by discussion; they were usually due to simple oversight, such as a missing trial reference. Interventions were classified as physical (eg, vacuum cleaning, washing, bed covers, heating, freezing), chemical (eg, acaricides), or combinations. The review-authors’ opinions were classified into 2 groups: positive, recommending 1 or more interventions (in a few cases rather vaguely, such as allergen avoidance), or neutral/negative (eg, more research is needed or the interventions should not be used).
The index review. The Cochrane review on house dust-mite control measures for asthma was first published in 1998 in the Cochrane Library8 and in BMJ.8 The latest update from 2001 included 28 randomized trials, one of which was 3-armed.9 When referring to randomized trials in our study, the sample is the 28 trial reports included in the Cochrane review. The Cochrane review, which included 939 patients in the analysis, failed to find an effect of chemical or physical methods on any of the outcomes (number of patients improved, asthma symptoms, medication usage, peak flow, FEV1, and bronchial hyperreactivity).