REMI 1930E. Meta-analysis with erroneous conclusions: size matters

[Versión en español]

Artículo original: Random error in cardiovascular meta-analyses: how common are false positive and false negative results? AlBalawi Z, McAlister FA, Thorlund K, Wong M, Wetterslev J. Int J Cardiol 2013; 168(2): 1102-1107. [Summary] [Related articles]

    

Introduction: Meta-analysis is a set of statistical tools to provide a quantitative synthesis of the evidence, obtained from an exhaustive and explicit search ("systematic review"). It is recognised that a meta-analysis can not be insightful if evidence is scarce or poor-quality, so its results should not be interpreted in a mechanistic way. Bias in the primary studies can cause systematic errors, so risk of bias must be thoroughly and carefully evaluated. However, an aspect hitherto little considered in meta-analysis methodology is the likelihood to commit random error (type I error, to find statistically significant results by chance, and type II error, not finding significant results by chance) [1-3]. This risk is greater when the number of studies, patients and/or events is scarce or when data are subjected to repeated statistical testing, but the magnitude and implications of this problem are not well understood.

    

Summary: The authors evaluated all the reviews published by the cardiology group of the Cochrane Collaboration in 2012 including one or more meta-analysis of at least 5 randomized clinical trials. Meta-analyses were classified as true positives if their pooled sample size and/or their cumulative Z-curve crossed the O'Brien-Fleming monitoring boundaries for detecting a relative risk reduction (RRR) of at least 25%, and true negatives if their pooled sample size was sufficient to reject a RRR of 25%. Meta-analysis evaluated were 56, 23 with positive results, and 33 with negative results. 17% were considered potentially false positive and 64% potentially false negative results. Overall, 45% of the meta-analysis contained insufficient information to conclusively rule out a different effect of found. In most cases, the authors of the meta-analysis did not recognize this limitation of their results.

      

Comment: The study shows the risk of reaching premature conclusions with meta-analysis, and how even meta-analysis from an organization as rigorous as the Cochrane Collaboration fail to take into account in many cases the degree of uncertainty in the interpretation of their results. A clear example is that of hypothermia in cardiac arrest, whose effectiveness was considered proven prematurely [4-5]. For a correct interpretation of meta-analysis results is necessary to incorporate techniques to quantify the risk of random error.

   
Eduardo Palencia Herrejón
Hospital Universitario Infanta Leonor, Madrid.
© REMI, http://medicina-intensiva.com. January 2014.

       

Links:

1. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. Brok J, Thorlund K, Gluud C, Wetterslev J. J Clin Epidemiol 2008; 61: 763-769. [PubMed]
2. Updating meta-analyses leads to larger type I errors than publication bias. Borm GF, Donders AR. J Clin Epidemiol 2009; 62: 825-830. [PubMed]
3. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. Wetterslev J, Thorlund K, Brok J, Gluud C. J Clin Epidemiol 2008; 61: 64-75. [PubMed]
4. Hypothermia for neuroprotection in adults after cardiopulmonary resuscitation. Arrich J, Holzer M, Havel C, Mullner M, Herkner H. Cochrane Database Syst Rev 2012: CD004128. [PubMed]
5. Hypothermia after cardiac arrest should be further evaluated--a systematic review of randomised trials with meta-analysis and trial sequential analysis. Nielsen N, Friberg H, Gluud C, Herlitz J, Wetterslev J. Int J Cardiol 2011; 151: 333-341. [PubMed]

  PubMed Search:

• Enunciado: Relevance of information size on error risk in meta-analysis
• Sintaxis: "information size" AND "meta-analysis as topic"[mh]
• [Results]