The V Lab
AudienceLearners in medicine, psychology, public health, and evidence synthesis
Study timeAbout 180–240 minutes
PrerequisitesEffect sizes, confidence intervals, regression, and basic R

About the examples and software The medical and psychology studies in this tutorial are simulated study-level aggregate data; they do not correspond to real papers, patients, or participants. This page develops the core calculations in base R so that the models remain transparent. A formal review should use validated specialist software, lock its versions and settings, and have a second analyst verify the work.

How to use this tutorial

This tutorial follows the sequence of a real evidence synthesis:

Answerable question → preregistered protocol → systematic search and screening → risk of bias → comparable effect measures → synthesis model → heterogeneity → robustness and missing evidence → certainty and reporting

The code covers statistical synthesis; it cannot replace a systematic search, duplicate screening, data verification, or risk-of-bias assessment. Read the concepts first, then work through the medical and psychology examples, and finally use the checklist to design your own analysis.

Learning objectives

After completing this tutorial, you should be able to:

  • distinguish systematic reviews, meta-analyses, and narrative reviews;
  • use PICO/PICOS and a target estimand to define a question suitable for synthesis;
  • choose RR, OR, or RD for binary outcomes and MD or Hedges’ gg for continuous outcomes;
  • explain generic inverse-variance weighting, common-effect models, and random-effects models;
  • calculate and interpret QQ, I2I^2, τ2\tau^2, a CI for the mean effect, and a prediction interval;
  • distinguish the roles of DL, REML, Paule–Mandel, and modified Knapp–Hartung;
  • construct and interpret forest plots and funnel plots;
  • conduct leave-one-out, effect-scale, and model-specification sensitivity analyses;
  • use subgroup analysis and meta-regression cautiously;
  • recognize dependence created by multiple outcomes, multiple time points, multi-arm studies, and shared controls;
  • discuss publication bias, small-study effects, certainty of evidence, and PRISMA reporting requirements; and
  • write results paragraphs for medical and psychology meta-analyses without overgeneralizing.

1 The place of meta-analysis within a complete systematic review

1.1 Three concepts that are not interchangeable

Concept Core task Is statistical synthesis required?
Systematic review Search for, screen, appraise, and synthesize evidence using prespecified, transparent, and reproducible methods No
Meta-analysis Quantitatively combine study results under a defensible common question and statistical model Yes
Narrative review Organize literature in prose; methodological transparency can vary greatly No

A meta-analysis can be calculated correctly yet answer the wrong question. For example, if studies of different diseases, different versions of an intervention, different follow-up periods, or scales measuring different constructs are combined, even a precise mean may have no clear meaning. Conversely, when clinical or methodological differences make pooling unreasonable, a high-quality systematic review can appropriately omit meta-analysis.

Garbage in, precisely estimated garbage out Meta-analysis does not “average” systematically biased studies into an unbiased truth. When biases tend in the same direction, adding studies may simply make the wrong conclusion more precise. Study quality, comparability, and missing evidence must inform interpretation rather than appearing only as a one-sentence limitation at the end of the discussion.

1.2 Define the PICO and estimand before every synthesis

Every forest plot should be backed by a clearly specified synthesis question:

  • P (Population): Which patients or participants, at what level of severity, and in which settings?
  • I (Intervention/Exposure): Which components, dose, provider, and duration?
  • C (Comparator): Placebo, usual care, wait-list, or active control?
  • O (Outcome): Which definition, scale, direction, and measurement time?
  • S (Study design): Randomized trials, cohort studies, case-control studies, or another design?
  • Estimand: Intention-to-treat or per-protocol; final values or change scores; RR, MD, or another scale?

The medical example targets the mean RR for 30-day infection risk with a new preventive program versus standard management across a set of exchangeable settings represented by 12 simulated parallel-group randomized trials. The psychology example targets the mean Hedges’ gg for depressive symptoms at the end of treatment across 14 simulated CBT trials. All scales are first aligned so that higher scores indicate worse symptoms; negative values therefore favor CBT.

1.3 Protocols, searches, screening, and duplicate reports

Before examining study results, the protocol should at minimum specify the databases and grey-literature sources, complete search strategies, search dates, language and publication-status restrictions, duplicate-screening process, approach to resolving conflicts, primary outcomes and time points, effect measures, synthesis models, handling of zero events and missing SDs, subgroup analyses, sensitivity analyses, and risk-of-bias tools.

One study may have a registry record, a conference abstract, a primary paper, and several secondary analyses. The unit of a meta-analysis is usually the study or independent comparison, not the number of PDFs. Including the same participants more than once artificially increases precision. Multiple reports should first be linked to the same study and then extracted according to prespecified rules.

PRISMA 2020 is a reporting guideline for systematic reviews; it is not the same as preregistration. Depending on the topic, a protocol may be registered with PROSPERO, OSF, or another suitable platform. Meta-analyses of observational epidemiologic studies can also consult MOOSE.

2 Deriving comparable effect measures from aggregate data

2.1 Binary outcomes: RR, OR, and RD

Let a,ba,b denote events and non-events in the treatment group, and let c,dc,d denote events and non-events in the control group. The risk ratio is calculated on the log scale:

yi=log⁡(RRi)=log⁡{a/(a+b)c/(c+d)},vi=1a−1a+b+1c−1c+d. y_i=\log(RR_i)=\log\left\{\frac{a/(a+b)}{c/(c+d)}\right\},\qquad v_i=\frac1a-\frac1{a+b}+\frac1c-\frac1{c+d}.

The odds ratio is OR=ad/(bc)OR=ad/(bc), and the risk difference is RD=a/(a+b)−c/(c+d)RD=a/(a+b)-c/(c+d). The three measures answer different questions:

Measure Null value Strengths Limitations
RR 1 Relatively intuitive in cohort studies and trials; often fairly stable Baseline risk is still needed to understand the absolute impact
OR 1 Estimable in case-control studies; naturally produced by logistic models Must not be called an RR when the outcome is common
RD 0 Expresses absolute risk change directly Often varies substantially with baseline risk, so between-study heterogeneity may be large

The event direction must be harmonized before analysis. Switching between “infection” and “no infection” transforms the RR into a different quantity that is not simply its reciprocal; outcome coding must not be chosen after seeing which direction is statistically significant.

2.2 Zero events require more than adding 0.5 everywhere

A zero event count in one arm makes the conventional log RR or log OR undefined; studies with zero events in both arms provide no direct information for these relative measures. Adding 0.5 to all four cells can introduce bias with small samples, imbalanced allocation, and rare events. Methods suited to the question—such as Mantel–Haenszel, generalized linear mixed models, beta-binomial models, or other rare-event approaches—should be prespecified and compared.

The tutorial function does not silently modify the data: a zero in any of the four cells requires a prespecified approach. Only when correction = TRUE is set explicitly does it apply a continuity correction to all four cells of that study, while flagging studies with zero events in both arms.

zero_demo <- data.frame(
  study = c("Single-arm zero event", "Zero events in both arms"),
  event_t = c(0, 0), total_t = c(50, 50),
  event_c = c(3, 0), total_c = c(50, 60)
)
zero_effect <- effect_log_rr(
  zero_demo$event_t, zero_demo$total_t,
  zero_demo$event_c, zero_demo$total_c,
  correction = TRUE
)
knitr::kable(
  cbind(zero_demo, zero_effect),
  digits = 3,
  caption = "Teaching example with an explicit 0.5 correction; log RR remains missing for the double-zero study"
)
Teaching example with an explicit 0.5 correction; log RR remains missing for the double-zero study
study event_t total_t event_c total_c yi vi se corrected double_zero
Single-arm zero event 0 50 3 50 -1.95 2.25 1.5 TRUE FALSE
Zero events in both arms 0 50 0 60 NA NA NA TRUE TRUE

2.3 Continuous outcomes: MD and standardized mean difference

When studies use the same scale and units, the mean difference is the easiest measure to interpret:

MDi=X‾Ti−X‾Ci,vi=sTi2nTi+sCi2nCi. MD_i=\bar X_{Ti}-\bar X_{Ci},\qquad v_i=\frac{s_{Ti}^2}{n_{Ti}}+\frac{s_{Ci}^2}{n_{Ci}}.

When different scales measure the same construct, effects can be standardized using the pooled within-group SD:

sp=(nT−1)sT2+(nC−1)sC2nT+nC−2,d=X‾T−X‾Csp,g=J(df)d. s_p=\sqrt{\frac{(n_T-1)s_T^2+(n_C-1)s_C^2}{n_T+n_C-2}},\quad d=\frac{\bar X_T-\bar X_C}{s_p},\quad g=J(df)d.

J(df)J(df) corrects the small-sample bias in Cohen’s dd. This tutorial uses a common large-sample approximation for the variance; specialist software may offer different variance estimators, and the selected option must be documented in the protocol.

The SMD is not a “universal clinical unit.” It is affected by within-study individual heterogeneity, scale reliability, eligibility criteria, and the way the SD is calculated. Scales should be combined only if they truly measure the same construct; sharing a label such as “mental health score” is not enough.

Auditing effect direction in psychology data The extraction form must record what a high score means on each scale and whether it has been reverse-coded. If a high score on one scale indicates improvement while a high score on another indicates worse symptoms, one effect direction must be reversed before synthesis. The protocol should define either “negative values favor the intervention” or the opposite direction, and a second analyst should verify the coding.

3 Generic inverse-variance models

3.1 Each study contributes an effect and a variance

Many meta-analyses can be written in a common form: each study contributes an effect estimate yiy_i and a known or estimated sampling variance viv_i. Under a common-effect model, the weight is wi=1/viw_i=1/v_i:

μ̂=∑iwiyi∑iwi,SE(μ̂)=1∑iwi. \hat\mu=\frac{\sum_i w_i y_i}{\sum_i w_i},\qquad SE(\hat\mu)=\sqrt{\frac1{\sum_i w_i}}.

Weights reflect precision, not study-quality scores. A large study at high risk of bias may still receive a large inverse-variance weight, so risk of bias cannot be addressed by simply multiplying by a subjective “quality weight.”

3.2 Common-effect and random-effects models answer different questions

Model Working assumption Meaning of the summary measure
Common-effect (traditional fixed-effect) All studies estimate one true effect; differences arise from sampling error That common effect
Random-effects Study-specific true effects arise from a distribution; observed differences reflect both true variation and sampling error The mean effect in that distribution

Random-effects weights are wi*=1/(vi+τ2)w_i^*=1/(v_i+\tau^2), where τ2\tau^2 is the between-study variance in true effects. A random-effects model does not “solve” heterogeneity; it uses a distributional model to describe unexplained variation. Model selection should follow from the question and anticipated differences, not from testing QQ first and switching automatically.

Random effects are not inherently more conservative When heterogeneity is present, a random-effects model gives relatively more weight to small studies. If small studies report larger effects because of selective publication or methodological problems, the random-effects mean may move even farther from the null. Both common-effect and random-effects models depend on assumptions about study comparability and bias.

3.3 QQ, I2I^2, H2H^2, and τ2\tau^2

  • Cochran’s QQ: Assesses whether the observed weighted differences are compatible with sampling error alone; it has low power with few studies and is sensitive to trivial differences with many studies.
  • I2I^2: max⁡{0,(Q−df)/Q}×100%\max\{0,(Q-df)/Q\}\times100\%, the relative proportion of observed inconsistency attributable to between-study variation—not the “percentage of studies with heterogeneity.”
  • H2H^2: This tutorial uses the traditional Q/dfQ/df; a value of 1 means that observed inconsistency equals the degrees of freedom, although sampling can produce a value below 1. Some software may report alternative definitions consistent with τ2\tau^2, so implementations must be checked.
  • τ2\tau^2: The model-estimated variance of true effects; its units are the square of the effect scale and, for RR or OR, it is on the log scale.

I2I^2 depends on study precision: the same τ2\tau^2 can produce a higher I2I^2 when studies are more precise. Report τ2\tau^2, a prediction interval, the study contexts, and the effect scale together rather than replacing judgment with fixed 25%/50%/75% labels.

3.4 DL, REML, and Paule–Mandel

  • DerSimonian–Laird (DL): A simple method-of-moments estimator; it may underestimate uncertainty when there are few studies or substantial heterogeneity.
  • REML: Estimates τ2\tau^2 through restricted likelihood, generally has good statistical properties, and is a common default.
  • Paule–Mandel (PM): A method-of-moments estimator that makes the random-effects weighted residual QQ close to its degrees of freedom and is also often robust.

The τ2\tau^2 estimator and the method used for the CI around the mean effect are two separate choices. HKSJ/Knapp–Hartung adjusts uncertainty in the mean effect; it does not estimate τ2\tau^2.

3.5 Modified Knapp–Hartung and prediction intervals

For random-effects means, this tutorial uses modified Knapp–Hartung (mKH): it replaces the normal critical value with a t critical value and constrains the residual scale factor to be at least 1, preventing the KH variance estimate from falling below the unadjusted model variance. mKH can be conservative, especially with very few studies; the protocol should identify the exact implementation.

A CI for the mean effect describes where the mean lies. A prediction interval attempts to describe where the true effect might lie in a new study exchangeable with the included studies:

μ̂±tk−2,0.975τ̂2+Var̂(μ̂). \hat\mu\ \pm\ t_{k-2,0.975}\sqrt{\hat\tau^2+\widehat{Var}(\hat\mu)}.

In this tutorial, Var̂(μ̂)\widehat{Var}(\hat\mu) in the formula is the mKH-adjusted variance of the mean. This is one commonly used Riley/HTS-type approximation, not a unique standard. With few studies, a non-normal effect distribution, or funnel-plot asymmetry, the prediction interval may be unstable; nor is it an interval for an “observed estimate” from a future study.

4 Medical example: 12 randomized trials with a binary outcome

4.1 Question, data structure, and effect direction

The simulated question is: compared with standard care, does a new prevention strategy reduce the 30-day risk of infection? Each row represents an independent parallel-group RCT, and every event is defined as an infection; therefore, RR<1RR<1 favors the new strategy.

stopifnot(
  !anyDuplicated(medical_data$study),
  all(medical_data$event_t <= medical_data$total_t),
  all(medical_data$event_c <= medical_data$total_c),
  all(medical_data$total_t > 0),
  all(medical_data$total_c > 0)
)

medical_display <- within(medical_data, {
  risk_t <- event_t / total_t
  risk_c <- event_c / total_c
})

knitr::kable(
  medical_display[c("study", "year", "event_t", "total_t", "event_c", "total_c",
                    "risk_t", "risk_c")],
  digits = 3,
  col.names = c("Study", "Year", "New-strategy events", "New-strategy total",
                "Standard-care events", "Standard-care total", "New-strategy risk", "Standard-care risk"),
  caption = "Simulated 30-day infection data from the medical trials"
)
Simulated 30-day infection data from the medical trials
Study Year New-strategy events New-strategy total Standard-care events Standard-care total New-strategy risk Standard-care risk
Med-01 2012 12 120 20 118 0.100 0.169
Med-02 2013 14 90 15 92 0.156 0.163
Med-03 2014 10 180 30 175 0.056 0.171
Med-04 2015 31 240 29 238 0.129 0.122
Med-05 2016 20 150 25 148 0.133 0.169
Med-06 2017 27 320 43 315 0.084 0.137
Med-07 2018 9 110 16 112 0.082 0.143
Med-08 2019 29 210 24 205 0.138 0.117
Med-09 2020 31 400 48 395 0.078 0.122
Med-10 2021 11 170 25 168 0.065 0.149
Med-11 2022 20 130 19 132 0.154 0.144
Med-12 2024 28 260 40 255 0.108 0.157

These are teaching data only. Extraction from real studies should also record the unit of randomization, outcome definition, loss to follow-up, ITT denominator, cluster or multi-arm structure, adjusted estimates, risk of bias, and funding source.

4.2 Calculating the log RR for each study

medical_es <- effect_log_rr(
  medical_data$event_t, medical_data$total_t,
  medical_data$event_c, medical_data$total_c
)
medical_meta <- cbind(medical_data, medical_es)
medical_meta$rr <- exp(medical_meta$yi)
medical_meta$rr_lower <- exp(medical_meta$yi - qnorm(0.975) * medical_meta$se)
medical_meta$rr_upper <- exp(medical_meta$yi + qnorm(0.975) * medical_meta$se)

knitr::kable(
  medical_meta[c("study", "rr", "rr_lower", "rr_upper", "se")],
  digits = 3,
  col.names = c("Study", "RR", "95% CI lower", "95% CI upper", "SE of log RR"),
  caption = "Risk ratios for the individual medical trials"
)
Risk ratios for the individual medical trials
Study RR 95% CI lower 95% CI upper SE of log RR
Med-01 0.590 0.302 1.152 0.341
Med-02 0.954 0.489 1.861 0.341
Med-03 0.324 0.163 0.643 0.349
Med-04 1.060 0.660 1.702 0.242
Med-05 0.789 0.459 1.358 0.277
Med-06 0.618 0.392 0.975 0.232
Med-07 0.573 0.264 1.241 0.394
Med-08 1.180 0.712 1.955 0.258
Med-09 0.638 0.415 0.980 0.219
Med-10 0.435 0.221 0.855 0.345
Med-11 1.069 0.599 1.908 0.296
Med-12 0.687 0.437 1.078 0.230

Each interval expresses the sampling uncertainty in that study. Smaller studies usually have wider intervals, but a larger sample does not guarantee less bias.

4.3 Common-effect and random-effects results

medical_common <- fit_meta(medical_meta$yi, medical_meta$vi, method = "common")
medical_dl <- fit_meta(medical_meta$yi, medical_meta$vi, method = "DL")
medical_pm <- fit_meta(medical_meta$yi, medical_meta$vi, method = "PM")
medical_reml <- fit_meta(medical_meta$yi, medical_meta$vi, method = "REML")

fit_row_ratio <- function(fit, label) {
  data.frame(
    Model = label,
    RR = exp(fit$estimate),
    CI_lower = exp(fit$lower),
    CI_upper = exp(fit$upper),
    PI_lower = if (is.finite(fit$pred_lower)) exp(fit$pred_lower) else NA_real_,
    PI_upper = if (is.finite(fit$pred_upper)) exp(fit$pred_upper) else NA_real_,
    tau_squared = fit$tau2,
    check.names = FALSE
  )
}

medical_model_table <- rbind(
  fit_row_ratio(medical_common, "Common effect + Wald"),
  fit_row_ratio(medical_dl, "Random effects DL + mKH"),
  fit_row_ratio(medical_pm, "Random effects PM + mKH"),
  fit_row_ratio(medical_reml, "Random effects REML + mKH")
)

knitr::kable(
  medical_model_table,
  digits = 3,
  caption = "Medical-example summaries under different models and between-study variance estimators"
)
Medical-example summaries under different models and between-study variance estimators
Model RR CI_lower CI_upper PI_lower PI_upper tau_squared
Common effect + Wald 0.729 0.623 0.854 NA NA 0.000
Random effects DL + mKH 0.720 0.571 0.909 0.421 1.23 0.047
Random effects PM + mKH 0.720 0.570 0.909 0.409 1.27 0.053
Random effects REML + mKH 0.721 0.572 0.910 0.429 1.21 0.043

The REML random-effects mean RR is approximately 0.72, with an mKH 95% CI from 0.57 to 0.91. This indicates a lower mean relative risk across the exchangeable settings represented by the model; it is not a promise that every hospital will see the same proportional reduction.

4.4 Heterogeneity and the prediction interval

medical_heterogeneity <- data.frame(
  Metric = c("Cochran Q", "Q degrees of freedom", "Q-test p-value", "I-squared (%)",
             "H-squared", "REML tau-squared", "REML tau"),
  Value = c(
    medical_reml$Q, medical_reml$Q_df, medical_reml$Q_p,
    medical_reml$I2, medical_reml$H2,
    medical_reml$tau2, sqrt(medical_reml$tau2)
  )
)
knitr::kable(
  medical_heterogeneity,
  digits = 3,
  caption = "Heterogeneity statistics for the medical example (tau is on the log-RR scale)"
)
Heterogeneity statistics for the medical example (tau is on the log-RR scale)
Metric Value
Cochran Q 17.595
Q degrees of freedom 11.000
Q-test p-value 0.091
I-squared (%) 37.481
H-squared 1.600
REML tau-squared 0.043
REML tau 0.208

I2I^2 is approximately 37.5%, but it should not simply be labeled “moderate heterogeneity.” The REML 95% prediction interval is more directly informative: RR 0.43 to 1.21, which crosses 1. An average effect incompatible with 1 does not guarantee that the true effect in a similar new setting will be beneficial.

The mean-effect CI and prediction interval do not conflict: the former quantifies the precision with which the mean log RR is estimated; the latter also incorporates genuine between-study variation and addresses how different a potential new setting might be.

4.5 Forest plot

forest_meta(
  medical_meta$yi,
  medical_meta$vi,
  labels = paste(medical_meta$study, medical_meta$year),
  fit = medical_reml,
  ratio = TRUE,
  xlab = "Risk ratio, RR (logarithmic spacing)",
  pooled_label = "REML mean RR",
  prediction_label = "95% prediction interval"
)
Forest plot of risk ratios from twelve medical trials, including individual confidence intervals, a random-effects summary diamond, and a prediction interval.

Forest plot of RRs from the simulated medical trials on a logarithmic scale. Square size reflects random-effects weight; the diamond is the REML mean effect, and the red line is the prediction interval. Axis labels show RRs.

Use the forest plot first to examine direction, precision, outlying results, and visible heterogeneity, and then read the summary diamond. Square size is neither the sample size itself nor a rating of study quality.

4.6 Returning from a relative effect to the absolute scale

The practical meaning of an RR depends on baseline risk. The following calculation applies the mean RR to the median control-group risk among the included trials as one illustrative scenario:

reference_risk <- median(medical_meta$event_c / medical_meta$total_c)
pooled_rr <- exp(medical_reml$estimate)
treated_risk <- reference_risk * pooled_rr
illustrative_rd <- treated_risk - reference_risk

absolute_translation <- data.frame(
  Reference_control_risk = reference_risk,
  Risk_after_applying_mean_RR = treated_risk,
  Risk_difference_per_1000 = 1000 * illustrative_rd,
  check.names = FALSE
)
knitr::kable(
  absolute_translation,
  digits = 3,
  caption = "Translating the mean RR to an illustrative reference risk"
)
Translating the mean RR to an illustrative reference risk
Reference_control_risk Risk_after_applying_mean_RR Risk_difference_per_1000
0.146 0.106 -40.8

This is not a direct meta-analysis of RDs, nor does it propagate all uncertainty in the reference risk, RR, and heterogeneity. Formal decisions should use a credible baseline risk for the target population and display several plausible scenarios.

4.7 Model and leave-one-out sensitivity analyses

Differences among DL, PM, and REML remind us that τ2\tau^2 is not a known constant when the number of studies is limited. Each study should also be omitted in turn while re-estimating τ2\tau^2:

medical_loo <- leave_one_out(
  medical_meta$yi, medical_meta$vi,
  labels = medical_meta$study,
  method = "REML"
)
medical_loo$rr <- exp(medical_loo$estimate)
medical_loo$rr_lower <- exp(medical_loo$lower)
medical_loo$rr_upper <- exp(medical_loo$upper)

knitr::kable(
  medical_loo[c("omitted", "rr", "rr_lower", "rr_upper", "tau2", "I2")],
  digits = 3,
  col.names = c("Omitted study", "Mean RR", "CI lower", "CI upper", "tau-squared", "I-squared (%)"),
  caption = "Leave-one-out REML analysis for the medical example"
)
Leave-one-out REML analysis for the medical example
Omitted study Mean RR CI lower CI upper tau-squared I-squared (%)
Med-01 0.730 0.567 0.941 0.051 41.8
Med-02 0.706 0.550 0.907 0.050 41.0
Med-03 0.763 0.623 0.934 0.014 16.0
Med-04 0.691 0.544 0.879 0.034 32.9
Med-05 0.713 0.550 0.923 0.058 42.9
Med-06 0.732 0.565 0.949 0.055 41.2
Med-07 0.730 0.567 0.938 0.049 41.9
Med-08 0.688 0.548 0.864 0.022 27.3
Med-09 0.730 0.562 0.948 0.058 41.7
Med-10 0.747 0.591 0.945 0.035 34.3
Med-11 0.697 0.547 0.889 0.042 36.7
Med-12 0.723 0.556 0.939 0.061 42.9

Influence analysis is a diagnostic tool, not an algorithm for deleting studies. If one study changes the conclusion, examine its population, design, data, and risk of bias, and report results both with and without it. Deleting it solely because it “makes the result nonsignificant” is outcome-driven analysis.

5 Psychology example: 14 CBT trials using different depression scales

5.1 Why use Hedges’ gg

The simulated studies use the PHQ-9, BDI-II, HADS-D, and CES-D. Because these scales have different units and ranges, the raw MD cannot be pooled across all 14 studies. We assume that, for this question, the scales measure a sufficiently similar construct of depressive symptoms and have been aligned so that higher scores are worse; a negative gg favors CBT.

stopifnot(
  !anyDuplicated(psychology_data$study),
  all(psychology_data$n_t > 1), all(psychology_data$n_c > 1),
  all(psychology_data$sd_t > 0), all(psychology_data$sd_c > 0)
)

knitr::kable(
  psychology_data,
  digits = 2,
  col.names = c("Study", "Scale", "CBT n", "CBT mean", "CBT SD",
                "Control n", "Control mean", "Control SD", "Treatment sessions", "Format"),
  caption = "End-of-treatment depression-scale data from the simulated psychology trials"
)
End-of-treatment depression-scale data from the simulated psychology trials
Study Scale CBT n CBT mean CBT SD Control n Control mean Control SD Treatment sessions Format
Psy-01 PHQ-9 70 11.5 5.1 68 14.5 5.0 10 Individual
Psy-02 BDI-II 55 25.4 8.2 57 27.0 8.0 6 Group
Psy-03 HADS-D 90 10.4 4.0 88 12.0 4.2 8 Individual
Psy-04 CES-D 120 25.5 9.5 118 25.0 9.2 4 Group
Psy-05 PHQ-9 48 11.0 4.8 50 15.0 5.1 8 Individual
Psy-06 BDI-II 75 23.7 7.7 74 26.0 7.9 10 Group
Psy-07 HADS-D 130 9.5 3.8 128 11.5 4.0 6 Individual
Psy-08 CES-D 62 27.0 10.2 65 28.0 9.8 4 Group
Psy-09 PHQ-9 85 10.4 5.3 82 14.0 5.0 12 Individual
Psy-10 BDI-II 110 22.5 8.0 108 24.5 8.3 6 Group
Psy-11 HADS-D 58 10.7 4.1 60 12.5 4.0 8 Individual
Psy-12 CES-D 140 27.9 9.4 136 27.0 9.6 10 Group
Psy-13 PHQ-9 95 12.7 4.9 92 15.5 5.2 10 Individual
Psy-14 BDI-II 72 22.7 7.9 70 25.5 8.1 8 Group

A real review must also verify the diagnostic criteria, type of control, therapist training, feasibility of blinding, adherence, scale version, and follow-up time point. A wait-list control and an active psychological control generally cannot be treated unconditionally as the same comparator.

5.2 Calculating and pooling Hedges’ gg

psych_es <- effect_hedges_g(
  psychology_data$n_t, psychology_data$mean_t, psychology_data$sd_t,
  psychology_data$n_c, psychology_data$mean_c, psychology_data$sd_c
)
psych_meta <- cbind(psychology_data, psych_es)
psych_meta$g_lower <- psych_meta$yi - qnorm(0.975) * psych_meta$se
psych_meta$g_upper <- psych_meta$yi + qnorm(0.975) * psych_meta$se

psych_reml <- fit_meta(psych_meta$yi, psych_meta$vi, method = "REML")
psych_common <- fit_meta(psych_meta$yi, psych_meta$vi, method = "common")

knitr::kable(
  psych_meta[c("study", "scale", "yi", "g_lower", "g_upper")],
  digits = 3,
  col.names = c("Study", "Scale", "Hedges g", "95% CI lower", "95% CI upper"),
  caption = "Standardized mean difference for each psychology trial"
)
Standardized mean difference for each psychology trial
Study Scale Hedges g 95% CI lower 95% CI upper
Psy-01 PHQ-9 -0.591 -0.932 -0.250
Psy-02 BDI-II -0.196 -0.568 0.175
Psy-03 HADS-D -0.389 -0.685 -0.092
Psy-04 CES-D 0.053 -0.201 0.307
Psy-05 PHQ-9 -0.801 -1.212 -0.389
Psy-06 BDI-II -0.293 -0.616 0.029
Psy-07 HADS-D -0.511 -0.759 -0.263
Psy-08 CES-D -0.099 -0.448 0.249
Psy-09 PHQ-9 -0.695 -1.008 -0.383
Psy-10 BDI-II -0.245 -0.511 0.022
Psy-11 HADS-D -0.442 -0.807 -0.076
Psy-12 CES-D 0.094 -0.142 0.331
Psy-13 PHQ-9 -0.552 -0.844 -0.260
Psy-14 BDI-II -0.348 -0.680 -0.017 
psych_results <- rbind(
  data.frame(
    Model = "Common effect + Wald", g = psych_common$estimate,
    CI_lower = psych_common$lower, CI_upper = psych_common$upper,
    PI_lower = NA, PI_upper = NA, tau_squared = 0
  ),
  data.frame(
    Model = "Random effects REML + mKH", g = psych_reml$estimate,
    CI_lower = psych_reml$lower, CI_upper = psych_reml$upper,
    PI_lower = psych_reml$pred_lower, PI_upper = psych_reml$pred_upper,
    tau_squared = psych_reml$tau2
  )
)

knitr::kable(
  psych_results,
  digits = 3,
  caption = "Common-effect and random-effects results for the psychology example"
)
Common-effect and random-effects results for the psychology example
Model g CI_lower CI_upper PI_lower PI_upper tau_squared
Common effect + Wald -0.317 -0.398 -0.236 NA NA 0.000
Random effects REML + mKH -0.345 -0.501 -0.188 -0.848 0.158 0.048

The random-effects mean gg is -0.34 (mKH 95% CI -0.50 to -0.19). On average, the negative direction favors CBT, but this does not indicate how many PHQ-9 points the score decreased, and clinical importance should not be judged solely from fixed rules of thumb such as 0.2/0.5/0.8.

5.3 Heterogeneity and the prediction interval in the psychology example

psych_heterogeneity <- data.frame(
  Q = psych_reml$Q,
  Q_df = psych_reml$Q_df,
  Q_p = psych_reml$Q_p,
  I2_percent = psych_reml$I2,
  tau2 = psych_reml$tau2,
  tau = sqrt(psych_reml$tau2),
  prediction_lower = psych_reml$pred_lower,
  prediction_upper = psych_reml$pred_upper,
  check.names = FALSE
)
knitr::kable(
  psych_heterogeneity,
  digits = 3,
  col.names = c("Q", "df", "Q p-value", "I-squared (%)", "tau-squared",
                "tau", "PI lower", "PI upper"),
  caption = "Heterogeneity and prediction interval for the psychology example"
)
Heterogeneity and prediction interval for the psychology example
Q df Q p-value I-squared (%) tau-squared tau PI lower PI upper
41 13 0 68.3 0.048 0.219 -0.848 0.158

The prediction interval is approximately -0.85 to 0.16 and crosses 0. Even when the mean-effect CI does not cross 0, the true effect in some similar settings may still be close to null or point in the opposite direction. Possible explanations include control intensity, treatment format, sample severity, measurement scale, therapist, and risk of bias, but these data cannot identify the explanation automatically.

forest_meta(
  psych_meta$yi,
  psych_meta$vi,
  labels = paste(psych_meta$study, psych_meta$scale),
  fit = psych_reml,
  ratio = FALSE,
  xlab = "Hedges' g (negative values favor CBT)",
  pooled_label = "REML mean g",
  prediction_label = "95% prediction interval"
)
Forest plot of standardized mean differences from fourteen psychology trials, including individual confidence intervals, a summary effect, and a prediction interval that crosses zero.

Forest plot of Hedges’ g from the simulated psychology trials of CBT. Negative values favor CBT; the diamond is the REML mean effect, and the red line is the prediction interval.

5.4 Prefer the original units when studies use the same scale

The four PHQ-9 studies can be pooled using the MD because their scale and direction are consistent:

phq <- subset(psychology_data, scale == "PHQ-9")
phq_es <- effect_md(phq$n_t, phq$mean_t, phq$sd_t,
                    phq$n_c, phq$mean_c, phq$sd_c)
phq_fit <- fit_meta(phq_es$yi, phq_es$vi, method = "REML")

data.frame(
  Number_of_studies = phq_fit$k,
  Mean_MD = phq_fit$estimate,
  CI_lower = phq_fit$lower,
  CI_upper = phq_fit$upper,
  tau_squared = phq_fit$tau2,
  check.names = FALSE
) |>
  knitr::kable(
    digits = 3,
    caption = "Random-effects mean difference for PHQ-9 studies only (CBT - control, in PHQ-9 points)"
  )
Random-effects mean difference for PHQ-9 studies only (CBT - control, in PHQ-9 points)
Number_of_studies Mean_MD CI_lower CI_upper tau_squared
4 -3.27 -4.6 -1.95 0

The MD is closer to a clinical interpretation, but here it represents only the PHQ-9 subset. It cannot be treated as the same estimand as gg across all scales. Post-treatment-score SMDs and change-score SMDs should not be mixed casually either, unless a prespecified method and its correlation assumptions support doing so.

6 Exploring Heterogeneity: Subgroups and Meta-Regression

6.1 Subgroup Differences Must Be Tested Directly

The following analysis compares individual CBT with group CBT. Pooling each subgroup separately is descriptive only; the subgroup difference should be tested directly through a meta-regression that includes a group indicator, rather than by comparing “significant in one subgroup” with “not significant in the other.”

format_levels <- levels(psych_meta$format)
psych_subgroup <- do.call(rbind, lapply(format_levels, function(level) {
  take <- psych_meta$format == level
  fit <- fit_meta(psych_meta$yi[take], psych_meta$vi[take], method = "REML")
  data.frame(
    Format = level, Studies = sum(take), Mean_g = fit$estimate,
    CI_lower = fit$lower, CI_upper = fit$upper, Tau_squared = fit$tau2
  )
}))

format_dummy <- as.numeric(psych_meta$format == "Individual")
X_format <- cbind(`(Intercept)` = 1, Individual_vs_Group = format_dummy)
format_model <- fit_meta(
  psych_meta$yi, psych_meta$vi,
  method = "REML", X = X_format
)

knitr::kable(
  psych_subgroup,
  digits = 3,
  caption = "Descriptive Synthesis Stratified by CBT Format"
)
Descriptive Synthesis Stratified by CBT Format
Format Studies Mean_g CI_lower CI_upper Tau_squared
Group 7 -0.126 -0.301 0.049 0.013
Individual 7 -0.551 -0.698 -0.403 0.000 
knitr::kable(
  format_model$coefficients,
  digits = 3,
  caption = "Random-Effects Meta-Regression with Group CBT as the Reference"
)
Random-Effects Meta-Regression with Group CBT as the Reference
term estimate se lower upper statistic p
(Intercept) -0.118 0.062 -0.252 0.016 -1.92 0.079
Individual_vs_Group -0.435 0.089 -0.630 -0.240 -4.86 0.000

The Individual_vs_Group coefficient and its p-value provide the direct test of the difference between the two subgroup mean effects. If a moderator has three or more levels, an omnibus F or Wald test of all dummy variables should be used rather than selecting one coefficient. Even when this study-level comparison is precise, it may be confounded by population, control intensity, number of treatment sessions, or risk of bias. It therefore cannot be interpreted as the causal effect of switching the same patient from group CBT to individual CBT.

6.2 Continuous Moderator: Number of Treatment Sessions

sessions_centered <- psych_meta$sessions - mean(psych_meta$sessions)
X_sessions <- cbind(`(Intercept)` = 1, sessions_centered = sessions_centered)
sessions_model <- fit_meta(
  psych_meta$yi, psych_meta$vi,
  method = "REML", X = X_sessions
)

knitr::kable(
  sessions_model$coefficients,
  digits = 3,
  caption = "Exploratory Random-Effects Meta-Regression of the Number of Treatment Sessions"
)
Exploratory Random-Effects Meta-Regression of the Number of Treatment Sessions
term estimate se lower upper statistic p
(Intercept) -0.344 0.068 -0.493 -0.195 -5.04 0.000
sessions_centered -0.051 0.029 -0.114 0.012 -1.75 0.105

The slope describes the study-level association between one additional treatment session and mean gg; it is not an individual-level dose–response effect. Fourteen studies support only very limited moderator exploration. “Approximately 10 studies per coefficient” is, at most, a cautionary lower bound, not a sufficient guarantee. Trying many moderators creates chance findings.

7 Small-Study Effects and Missing Evidence

7.1 What a Funnel Plot Can Suggest

A funnel plot displays effect estimates against their standard errors. If sampling error is the only source of variation, the model is appropriate, and effects are unrelated to study size, more precise studies will generally cluster near the top while smaller studies will be more dispersed near the bottom.

funnel_meta(
  psych_meta$yi,
  psych_meta$vi,
  fit = psych_reml,
  xlab = "Hedges' g"
)
Funnel plot of effect sizes against standard errors for the psychology trials, with a center line and triangular pseudo-confidence limits.

Funnel plot of the simulated psychology trials. Dashed lines show the REML mean effect and its pseudo-95% limits; asymmetry is not equivalent to publication bias.

Funnel-plot asymmetry may arise from publication or selective reporting, but it may also reflect genuine heterogeneity, poorer conduct in small studies, a relationship between the effect scale and baseline risk, outlying studies, or chance. Symmetry likewise does not prove that no results are missing.

7.2 Egger Regression Is Only a Diagnostic

psych_egger <- egger_test(psych_meta$yi, psych_meta$vi)
knitr::kable(
  psych_egger,
  digits = 3,
  caption = "Egger Regression Intercept Test for the Psychology Example"
)
Egger Regression Intercept Test for the Psychology Example
intercept se t df p
-5.26 2.69 -1.95 12 0.075

The Egger test generally should not be emphasized when there are fewer than approximately 10 independent studies. Even with more studies, it is a test for small-study effects, not a “test for publication bias.” Its interpretation is more complicated in the presence of heterogeneity. Trim-and-fill is also not a reliable repair that recovers the true missing studies; at most, it should be used as a sensitivity analysis under strong assumptions.

7.3 Missing Evidence Must Be Evaluated from the Research Process

More informative approaches include searching registries and protocols; comparing planned with reported outcomes; contacting authors; including grey literature; checking time points; examining whether results were selected according to statistical significance; and conducting sensitivity analyses under plausible selection mechanisms. The trustworthiness of the evidence cannot be decided from a single funnel plot.

8 Dependent Effects and Complex Study Structures

8.1 Why the “Number of Effect Rows” Is Not the Number of Independent Studies

Psychology studies often report multiple scales, follow-up times, intervention groups, and subgroups. Medical trials may share one control group across several comparisons. Treating these correlated effects as independent studies double-counts participants and underestimates standard errors.

Structure Source of dependence Reasonable approach
Multiple outcomes from the same study The same participants complete multiple scales Prespecify one primary outcome; or use multivariate or multilevel meta-analysis
Multiple time points Repeated measurements from the same participants Prespecify a primary time point; or explicitly model within-person dependence over time
Multi-arm trial with a shared control Control participants contribute to multiple comparisons Combine relevant intervention arms, split the control group, or use a multivariate method
Cluster-randomized trial Participants within the same cluster are correlated Use an already adjusted effect/SE, or adjust the effective sample size using the ICC
Multiple articles from the same cohort Participants are duplicated Link reports and deduplicate by study ID

Formal analyses can use multivariate or multilevel meta-analysis, or cluster-robust variance estimation by study with a small-sample correction. The required extension packages are not installed in this project, so this page does not demonstrate these complex models using an incorrect independence approximation.

8.2 Adjusted Estimates from Observational Studies

Observational medical studies often report ORs, HRs, or RRs based on different covariate sets. Before pooling, prespecify the preferred adjustment set and confirm that each estimate addresses approximately the same conditional or causal question. Mixing crude estimates with overadjusted estimates produces a difficult-to-interpret average; meta-analysis does not eliminate residual confounding.

9 Risk of Bias, Sensitivity Analysis, and Certainty of Evidence

9.1 Risk of Bias Is Not a Total Score

Randomized trials may be evaluated across domains such as the randomization process, deviations from intended interventions, missing outcome data, outcome measurement, and selective reporting. Observational studies also require careful consideration of confounding and selection. Simply adding domains into a “quality score” conceals potentially fatal problems and is not suitable as a mechanical adjustment to inverse-variance weights.

Risk-of-bias judgments should instead inform interpretation of the direction and magnitude of bias; prespecified sensitivity analyses that exclude studies at high risk; comparisons with evidence at low risk; explanations of heterogeneity; and assessments of certainty of evidence. Exclusion rules must be determined before examining the pooled results.

9.2 A Minimum Sensitivity-Analysis Matrix

Decision Primary analysis Reasonable sensitivity analyses
Synthesis model REML + mKH Common effect; PM/DL; other robust methods in specialist software
Effect scale RR for the medical example; gg for the psychology example RD/OR; MD for a common scale; direction checks
Zero events Prespecified rare-event method Continuity-correction rules; handling of double-zero studies
Risk of bias All eligible studies Exclude studies at high risk in prespecified critical domains
Influential studies All studies Leave-one-out analysis; justified scenario analyses after data checking
Missing SD/correlation Prespecified imputed value Several plausible correlations or SD assumptions
Dependent effects Prespecified single outcome/time point Multivariate, multilevel, or study-clustered robust methods

Sensitivity analyses should reveal which defensible decisions affect the conclusion; they should not be used to search for the smallest p-value.

9.3 Certainty of Evidence

Statistical precision is only one component of certainty. Risk of bias, inconsistency, indirectness, imprecision, and publication bias must also be considered. A meta-analysis with a large sample and a narrow CI may still provide low-certainty evidence if its studies are indirect or at high risk of bias. Frameworks such as GRADE require domain-specific judgments and cannot be generated automatically from I2I^2 or a p-value.

10 Reporting Results and a Reproducible Workflow

10.1 Preregistered Analysis Checklist

Before extracting results, specify:

  • the PICO/PICOS, target population, and estimand for each synthesis;
  • the primary outcome, scale direction, time point, and effect measure;
  • rules for deduplicating multiple reports and selecting among multiple scales, time points, or intervention arms;
  • whether adjusted or unadjusted estimates will be used and the priority order for adjustment sets;
  • the common-effect or random-effects target and the rationale for that choice;
  • the τ2\tau^2 estimator, method for the mean-effect CI, and prediction-interval convention;
  • handling of zero events, missing SDs, change scores, clustering, and crossover designs;
  • the subgroup, moderator, interaction-testing, and multiplicity plan;
  • methods for risk of bias, sensitivity analysis, missing evidence, and certainty of evidence; and
  • software, package versions, function settings, random seeds, and the independent verification process.

10.2 Medical Example: Reporting Template

Twelve simulated parallel-group randomized trials were included, comprising 4733 participants. With infection defined as the event, a REML random-effects model with modified Knapp–Hartung inference yielded an average RR of 0.72 (95% CI 0.57–0.91). The between-study variance was τ2=0.043\tau^2=0.043 on the log RR scale, with I2=37.5%I^2=37.5\%; the 95% prediction interval was 0.43–1.21. The average result suggests a reduction in relative risk, but the prediction interval crosses 1; therefore, the direction of effect should not be assumed to be consistent across all similar settings. All data are simulated, and no real risk-of-bias or certainty-of-evidence assessment was conducted.

10.3 Psychology Example: Reporting Template

Fourteen simulated CBT trials were included, comprising 2406 participants. After harmonizing scale direction, a REML random-effects model with modified Knapp–Hartung inference yielded a mean post-treatment Hedges’ g=−0.34g=-0.34 (95% CI -0.50 to -0.19; negative values favor CBT). The between-study variance was τ2=0.048\tau^2=0.048, with I2=68.3%I^2=68.3\%, and the 95% prediction interval was -0.85 to 0.16. The mean symptom difference favors CBT, but effects vary substantially across studies, the prediction interval includes the null value, and the SMD does not correspond to the clinical units of any single scale.

10.4 PRISMA and Reproducibility

Reports should use the applicable PRISMA 2020 checklist and flow diagram and specify the studies included in each synthesis, model, effect measure, τ2\tau^2 estimator, CI method, heterogeneity statistics, software commands, and sensitivity analyses. Reviews of observational studies may also consult MOOSE.

Retain search results, deduplication records, screening decisions, exclusion reasons, the data dictionary, raw extractions, cleaning logs, analysis datasets, scripts, and the software environment. Ideally, a second analyst should independently reproduce the primary results from the raw extraction table.

11 Quick Guide to Common Errors

Incorrect practice Why it is problematic Better approach
Calling a review systematic merely because “a meta-analysis was performed” Statistical synthesis is not equivalent to systematic methods Report the complete search, screening, risk-of-bias assessment, and protocol
Selecting fixed or random effects according to the p-value for QQ Test power depends on the number of studies, and the two models address different questions Prespecify the model based on the estimand and clinical differences
Writing that “60% of studies are heterogeneous” when I2=60%I^2=60\% I2I^2 is not a proportion of studies Interpret τ2\tau^2, the PI, effect scale, and setting together
Saying random effects have “controlled for heterogeneity” Unexplained differences still remain Explore causes and report a prediction interval
Treating the OR as an RR They can differ substantially when the outcome is common Select the measure according to the design and name it accurately
Automatically adding 0.5 to every zero cell This may introduce bias, and double-zero studies provide different information Prespecify a rare-event method and perform sensitivity analyses
Directly pooling MDs from different scales Their units are not comparable Use the SMD for the same construct, or analyze scales separately
Forgetting to reverse scale direction Beneficial and harmful effects can cancel one another Retain a direction variable and verify it independently
Treating every scale and time point as an independent study Participants are counted repeatedly and the SE is too small Prespecify selection rules or use a dependent-effects model
Claiming a subgroup difference because one subgroup is significant and the other is not A difference in significance is not a test of the difference in effects Directly test the subgroup-by-effect interaction
Interpreting a meta-regression slope as an individual-level causal effect Study-level ecological bias and confounding Report it as an exploratory study-level association
Declaring publication bias whenever the funnel plot is asymmetric Asymmetry has multiple possible causes Consider registrations, protocols, heterogeneity, and selection mechanisms together
Deleting a study because it changes the result This is outcome-driven and changes the target Check the study, report transparently, and retain it in the primary analysis
Reporting only the pooled p-value This omits magnitude, precision, and transportability Report the effect, CI, τ2\tau^2, PI, and risk of bias

12 Exercises and Answers

12.1 Exercise 1: Should These Studies Be Pooled?

Five studies all report “anxiety,” but they respectively examine short-term preoperative anxiety, generalized anxiety disorder, and test anxiety; the interventions and time points also differ. Can they be pooled directly simply because all results can be converted to an SMD?

Show answer No. Mathematical standardization does not make the clinical questions exchangeable. Return to the prespecified PICO, construct, and time window. If necessary, conduct separate syntheses or do not pool statistically, and explain the differences.

12.2 Exercise 2: Common Effect or Random Effects?

The QQ test gives p=0.18p=0.18, so the investigator selects a common-effect model. Is this rationale sufficient?

Show answer No. With few studies, the QQ test has limited power to detect heterogeneity, and model choice should not be controlled by a single test. First clarify whether the target is one common true effect or the mean of a distribution of true effects across a set of settings, while also considering clinical and methodological differences.

12.3 Exercise 3: Mean-Effect CI versus Prediction Interval

The 95% CI for the random-effects mean RR is 0.70–0.88, whereas the prediction interval is 0.50–1.25. How should these results be interpreted?

Show answer The mean true effect is estimated relatively precisely and suggests benefit, but true effects vary across studies; the true RR in a similar new setting could cross 1. The mean-effect CI cannot support a claim that every setting will benefit.

12.4 Exercise 4: Direction of the SMD

In three depression studies, higher scores indicate worse symptoms on two scales but better outcomes on one scale. What happens if gg is calculated directly as “intervention minus control”?

Show answer The third study’s effect will point in the opposite direction from the others, and the pooled effects may cancel incorrectly. Direction must be harmonized before examining the results—for example, by defining negative values as symptom improvement for every study—and the reversal rule must be documented.

12.5 Exercise 5: Double-Zero Events

A small safety trial reports no deaths in either group. What information does it provide to a conventional log RR synthesis?

Show answer A conventional log RR cannot be estimated from this table. A double-zero study does not directly identify the relative risk, but it does indicate that the event was rare in this sample. Whether and how to retain this information depends on the effect measure and rare-event model; silently adding 0.5 and treating it as an ordinary study is not appropriate.

12.6 Exercise 6: Subgroup Significance

The pooled result is p=0.01p=0.01 in adult studies and p=0.20p=0.20 in adolescent studies. Can we conclude that age group modifies the treatment effect?

Show answer No. The difference between the two group effects must be tested directly. Even if the interaction is significant, it remains a study-level comparison and must be interpreted in light of prespecification, the number of studies, confounding, and multiplicity.

12.7 Exercise 7: Funnel Plot

Small studies appear to be missing from the lower-right portion of a funnel plot. Does this prove publication bias?

Show answer No. Asymmetry may result from selective publication, but it may also arise from genuine heterogeneity, conduct quality, the effect scale, outlying studies, or chance. Interpret it together with registries and protocols, searches for unpublished studies, assessments of outcome-reporting bias, and sensitivity analyses.

12.8 Exercise 8: Influential Study

After one large study is excluded, the result changes from nonsignificant to significant. Can that study be deleted?

Show answer Not solely because the result changes. Check eligibility, data, design, risk of bias, and the applicable population. If the study meets the prespecified criteria, retain it in the primary analysis and report the sensitivity result transparently. Any exclusion needs a rationale independent of statistical significance.

13 Quick Reference

13.1 Key Formulas and Null Values

Quantity Calculation or meaning Null value
log RR log⁡[pT/pC]\log[p_T/p_C] 0 (RR = 1)
log OR log⁡[ad/(bc)]\log[ad/(bc)] 0 (OR = 1)
RD pT−pCp_T-p_C 0
MD X‾T−X‾C\bar X_T-\bar X_C 0
Hedges’ gg Small-sample-corrected standardized mean difference 0
Common-effect weight 1/vi1/v_i —
Random-effects weight 1/(vi+τ2)1/(v_i+\tau^2) —
I2I^2 Relative measure of observed inconsistency 0%
τ2\tau^2 Between-study variance of true effects in the model 0

Final Checklist

  • Each synthesis has a clearly defined PICO/PICOS, time point, and estimand.
  • The search, screening, data extraction, deduplication, and reasons for exclusion can be audited.
  • Effect directions, units, adjustment status, and outcome definitions have been independently verified.
  • The study—not the article or effect row—is treated as the independent unit.
  • The model, τ2\tau^2 estimator, CI method, and PI formula were prespecified and documented.
  • Zero events, missing SDs, multi-arm designs, clustering, and repeated outcomes were handled appropriately.
  • Individual-study results, the mean effect, CI, τ2\tau^2, I2I^2, and prediction interval are all reported.
  • Subgroup analyses and meta-regressions test differences directly and avoid overinterpreting them causally.
  • Funnel-plot or Egger results are not treated as proof of publication bias.
  • Leave-one-out and sensitivity analyses have not become outcome-driven deletion rules.
  • Risk of bias, indirectness, imprecision, and missing evidence inform the conclusion.
  • PRISMA items, code, data, versions, and the audit trail are sufficient to support reproducibility.
sessionInfo()
## R version 4.6.1 (2026-06-24)
## Platform: aarch64-apple-darwin23
## Running under: macOS Tahoe 26.5.1
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1
## 
## locale:
## [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
## 
## time zone: America/Edmonton
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] digest_0.6.39   R6_2.6.1        fastmap_1.2.0   xfun_0.60       lattice_0.22-9 
##  [6] cachem_1.1.0    knitr_1.51      htmltools_0.5.9 rmarkdown_2.31  stats4_4.6.1   
## [11] lifecycle_1.0.5 cli_3.6.6       grid_4.6.1      sass_0.4.10     jquerylib_0.1.4
## [16] compiler_4.6.1  tools_4.6.1     nlme_3.1-169    evaluate_1.0.5  bslib_0.12.0   
## [21] yaml_2.3.12     rlang_1.3.0     jsonlite_2.0.0

All studies, participants, and results in this tutorial are simulated solely for teaching meta-analysis. They do not constitute medical, psychotherapeutic, or policy advice. Real-world evidence synthesis should be conducted by a team with subject-matter, information-retrieval, statistical, and methodological expertise according to a preregistered protocol.