15 Aug Analyzing the Accuracy of Meta-Analysis Studies
MedicalResearch.com Interview with:
Agnes Dechartres, MD, PhD
Centre de Recherche Epidémiologie et Statistique
Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris
Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France
Medical Research: What are the main findings of the study?
Dr. Dechartres: In this study, we aimed to compare treatment effect estimates obtained from the meta-analysis including all trials to several alternative strategies for analysis. These alternative strategies are:
- 1) the single most precise trial;
- 2) a meta-analysis including only the largest trials;
- 3) a “limit meta-analysis” that is a type of meta-analysis model recently developed to take into account small-study-effect and
- 4) a meta-analysis restricted to trials at low risk of biases.
Our results showed that estimation of treatment effect varies depending of the strategy used with a frequently larger treatment effect in the meta-analysis of all trials than in the single most precise trial, the meta-analysis restricted to the largest trials and the limit meta-analysis, especially in case of subjective outcomes.
Medical Research: Were any of the findings unexpected?
Dr. Dechartres: Some previous meta-epidemiological studies found that small and moderate-sized trials showed larger treatment effects than the largest trials within a meta-analysis. Others found larger treatment effects in trials at high or unclear risk of bias than in trials at low risk for sequence generation, allocation concealment and blinding. Despite the results of these studies, sensitivity analyses based on sample size or risk of bias are seldom reported in meta-analysis reports and most published meta-analyses include all trials whatever their risk of bias or sample size. The alternative strategies used in our study were based on the results of these meta-epidemiological studies so finding a larger treatment effect in the meta-analysis of all trials was not really surprising. Nevertheless, these larger effects seem to be more marked for subjective outcome than for objective outcomes. Also, we did not find any differences according to overall risk of bias which suggests that combining domains into an overall risk of bias may be misleading.
Medical Research: What should clinicians and patients take away from your report?
Dr. Dechartres: Systematic reviews and meta-analyses are widely used in clinical research as a way to summarize all available evidence on a particular topic. Meta-analyses are very attractive because they allows for getting a single estimate of treatment effect from all available studies. Nevertheless, the methods used to conduct the meta-analysis can have an important influence on the results and on the conclusions. Meta-analyses should systematically present sensitivity analyses based on sample size and risk of bias to check the robustness of the findings and of the conclusions. Meta-analyses for which results disagree with the results of large well-done trials should be interpreted carefully.
Medical Research: What recommendations do you have for future research as a result of this study?
Dr. Dechartres: We recommend future authors of systematic reviews and meta-analyses to require help from methodologists and to systematically perform sensitivity analyses based on sample size and risk of bias. We suggest the comparison of the meta-analysis result to the result of the single most precise trial or of the meta-analysis restricted to the largest trials. If 10 trials or more are included, a limit meta-analysis may be of interest. We recommend assessing the impact on treatment effect of the key domains of the Risk of Bias Tool developed by the Cochrane Collaboration (sequence generation, allocation concealment and blinding) separately rather than combining them into an overall risk of bias.
Citation:
Last Updated on August 15, 2014 by Marie Benz MD FAAD