Systematic Reviews with Meta-Analysis

Systematic reviews with meta-analysis are considered the gold standard for conducting reliable and trustworthy synthesis of available evidence in one area of study (Crocetti, 2016). Specifically, a systematic review (or research synthesis) is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review (Higgins & Green, 2011). Meta-analysis refers to the use of statistical techniques to synthesize results across multiple primary studies. Importantly, systematic reviews and meta-analysis can be conducted independently from each other. Indeed, a systematic review may not include a statistical synthesis of the results and a meta-analysis can be applied to data not retrieved by means of a systematic review. A best practice is to combine the advantages of systematic reviews and meta-analysis in order to provide more sophisticated and advanced reviews of a certain field.

A systematic review with meta-analysis can be conducted to summarize and critically evaluate both inconsistent and consistent literature and be performed with a small, medium, or large number of studies. By means of a systematic review with meta-analysis several research questions can be addressed, such as relevant theoretical (e.g., How does adolescent identity develop over time? Which is the association between adolescent personality and well-being? Are there gender differences in adolescent problem behaviors?) and methodological (e.g., which is the overall reliability of a certain instrument; Hale, Crocetti, Raaijmakers, & Meeus, 2011) questions. In addition, another aspect addressed by several systematic reviews with meta-analysis is the efficacy of interventions and treatments (e.g., is a certain psychosocial intervention effective for adolescents? Campbell Collaboration, 2014; Higgins & Green, 2011).

Importantly, systematic reviews with meta-analysis provide a context to test which factors (moderators) can explain differences in the magnitude of the effect being observed. In this way, it is possible to identify factors that might have accounted for inconsistent findings reported in the literature or individuate a number of conditions that might explain an amplification or a reduction in the effect under investigation. Furthermore, in line the statistics reform occurring in the social sciences (Cumming, 2012; Kline, 2013), systematic reviews with meta-analysis are focused on estimating the dimensions of effects under consideration (this information is provided by the effect sizes) and their precisions (this information is provided by the confidence intervals), more than their statistical significance.

Conducting a systematic review with meta-analysis requires following multiple steps, from the definition of the research questions being addressed until the publication of the results (Crocetti, 2016).

The first step for conducting a systematic review with meta-analysis is to define the object of the review and the research question being addressed. The aim of a systematic review with meta-analysis should be rooted in a clear theoretical background. This is a prerequisite for avoiding “fishing” temptations and committing a mistake that can invalidate the entire process – that is, mixing “apples and oranges” (Lipsey & Wilson, 2001).

The second step consists of specifying inclusion and exclusion criteria. These criteria define which studies will be eligible for inclusion in the systematic review with meta-analysis. Eligibility criteria can be grouped into two main classes: eligibility criteria referring to the characteristics of the study (issues concerning the population, the variables, and the study designs of interest) and those referring to the characteristics of the publication (year, language, and type).

The third step consists of searching the literature. In order to conduct a comprehensive search of all available primary studies, a good practice is to employ multiple search strategies (e.g., computerized database search, searching indexes of journals, and searching in reference lists).

The forth step consists of selecting primary studies. This step implies multiple sub-phases that should be all documented in a diagram (i.e., the PRISMA flow diagram; Moher et al., 2009). First of all, duplicates (i.e., the same reference retrieved from multiple search strategies) can be identified and deleted. Second, the remaining references are screened by checking their title and abstract. If they could potentially match the eligibility criteria they are retained, otherwise they are excluded. Third, the retained references are assessed in the full-text. Articles included in the systematic review can be further included in the meta-analysis if they report data required for statistical computations. To facilitate navigation through these phases, the researcher can benefit from using a reference manager (e.g., Endnote) to save search and selection results.

The fifth step implies coding primary studies, to extract relevant information. This step can be conducted by means of a coding protocol, detailing which data should be extracted from each study and how they should be coded. Data coded from each primary study can be grouped into three categories: (a) characteristics of the study (e.g., age of the sample, type of design, measures being used); (b) characteristics of the publication (e.g., year, language, type); and (c) data for effect size computations.

The sixth step requires conducting statistical analyses. Specifically, the following analyses are required: computing an effect size for each study together with a measure of its precision; assign a weight to each study; obtain the overall effect size; evaluating heterogeneity across studies; and testing for factors that can explain this heterogeneity. The last sub-phase consists of moderator analyses (subgroup analyses and meta-regressions) through which factors that are assumed to affect the magnitude of the effect sizes can be tested. Additionally analyses include sensitivity analyses (to assess which is the impact that each study has on the final result of the meta-analysis) that are particularly important to check the robustness of overall results and the impact of potential study outliers. Finally, it is important to evaluate publication bias that refers to the situation that occurs when published studies (those that can be easily retrieved) differ systematically from unpublished studies (grey literature; Rothstein, Sutton, & Borenstein, 2005). This evaluation can be conducted by means of multiple approaches: funnel plot, Egger’s linear regression method, Begg and Mazumdar’s rank correlation method, Duval and Tweedie’s Trim and Fill method, Rosenthal’s Fail-safe N. All these statistical analyses can be easily conducted by means of software specific for meta-analysis (e.g., ProMeta 2.0, for handling complex meta-analytic databases).

The final step is publishing a high-quality article. In publishing a systematic review with meta-analysis, the researcher should be as detailed as possible. In this respect, the author is strongly supported by following available guidelines, which provide useful tools for preparing high-quality reports of systematic review with meta-analysis. Most important guidelines include PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Liberati et al. 2009; Moher et al., 2009); MARS (Meta-analysis reporting standards; American Psychological Association, 2010); and MOOSE (Meta-analysis of Observational Studies in Epidemiology; Stroup et al., 2000).

For a more detailed explanation and for all the references see:

Crocetti, E. (2016). Systematic reviews with meta-analysis: Why, when, and how? Emerging Adulthood, 4 (1), 3-18.

Crocetti, E. (2015). Rassegne sistematiche, sintesi della ricerca e meta-analisi [Systematic reviews, research synthesis, and meta-analysis]. North Charleston, SC, USA: CreateSpace. ISBN: 9781516898336.