9 September 2019: 

Systematic reviews and meta-analyses are increasingly popular since they provide the best and most reliable unbiased analysis of the existing evidence, which is essential for evidence-based clinical practice. Different approaches and techniques are commonly used to generate this evidence. Understanding the benefits and limitations of these techniques is essential when interpreting the results of a meta-analysis. Riley et al. published an important methodological article in the British Medical Journal [1] that precisely explained two important approaches used in meta-analyses: the fixed effects and random effects models, and when each should be used. They also introduced the concept of the prediction interval (PI) which will be increasingly used in meta-analyses studies. They explained the difference between the confidence interval (CI) and the PI and their applications and interpretations in evidence-based clinical practice. They explained why a PI could provide a more complete summary of the clinical implications of the findings of a meta-analysis that is generally provided by a CI. In meta-analyses, fixed effects models assume that the true effect of treatment for each study included in the meta-analysis is the same or fixed. In other words, there is no between-study difference or heterogeneity in the true treatment effect (assessed by a statistical analysis called the I²) [2]. The differences seen between studies in a fixed effect model are due to random or chance variations in sampling, such that if all studies had a large (infinite) sample size, the differences in study estimates would vanish. On the other hand, the random effects models allow for real differences (heterogeneity) in the treatment effect from study to study. So, the observed estimates of treatment effect can vary between studies because of real differences due to both random and non-random variation. Sources of heterogeneity could include differences in study populations (e.g. age of patients), interventions received (e.g. dose of drug), follow-up length etc. as well as sample size [1, 3]. In meta-analyses, when data are pooled and analysed using random effects models, it is standard to report a CI around the effect estimate. However, when heterogeneity is large, some authors have proposed reporting a PI along with a CI to have a better appreciation of the uncertainty around the effect estimate [1, 4]. The PI, used in this way, is a relatively new concept. The CI and PI are not the same thing. A CI is a range of values, which indicates the degree of uncertainty about a population parameter, for example, the mean blood pressure of a population. A PI, on the other hand, is the interval within which you would expect a future mean value of blood pressure to fall in a new research study.

Putting theory into practice

Our recent systematic review with meta-analyses of 46 reports from 36 randomized controlled trials (RCTs) showed that the effect of testosterone therapy improved a range of sexual function parameters in postmenopausal women [5]. For the sexual desire outcome, in 15 studies that included 3,762 postmenopausal women, we used a random effects meta-analysis as discussed above. This approach assumed that there may be both random and non-random variations (heterogeneity) across the studies that may influence the outcome of interest (sexual desire). In addition, we computed the 95% CI and the 95% PI for the effect estimate of testosterone therapy on sexual desire. The random effects meta-analysis showed that testosterone therapy was effective for improving sexual desire in postmenopausal women when using 95% CI (SMD 0·36, 95% CI 0·22 to 0·50). However, the I2 value for the pooled estimate of sexual desire was 69%, which indicated moderately high heterogeneity between the included studies. So, in order to estimate the potential effects of testosterone on sexual desire in future individual studies, the 95% PI was also estimated. We observed that the PI was much wider than the CI and included zero (SMD 0·36, 95% PI -0.12 to 0.84). This means that although on average the effect of testosterone for the completed studies is positive for sexual desire, a future study may not have a treatment effect, which is different from zero (no difference between treatment and placebo). It is critical to realize that CI and PI convey different but complementary information.

Rakib Islam

Research Fellow, Women’s Health Research Program, Monash University, Australia

References

1. Riley RD, Higgins JP and Deeks JJ. Interpretation of random effects meta-analyses. Bmj 2011; 342: d549.
   https://www.ncbi.nlm.nih.gov/pubmed/21310794
2. Higgins JP, Thompson SG, Deeks JJ and Altman DG. Measuring inconsistency in meta-analyses. Bmj 2003; 327(7414): 557-560.
   https://www.ncbi.nlm.nih.gov/pubmed/12958120
3. Egger M, Davey-Smith G and Altman D. Systematic reviews in health care: meta-analysis in context. John Wiley & Sons, 2008.
4. Higgins JP, Thompson SG and Spiegelhalter DJ. A re‐evaluation of random‐effects meta‐analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172(1): 137-159.
   https://www.ncbi.nlm.nih.gov/pubmed/19381330
5. Islam RM, Bell RJ, Green S, Page MJ, Davis SR. Safety and efficacy of testosterone for women: a systematic review and meta-analysis of randomised controlled trial data. The Lancet Diabetes and Endocrinology. 2019 Jul 25.
   https://www.ncbi.nlm.nih.gov/pubmed/31353194