Statistical Inference Arena← Mediation leaderboard

Causal Mediation Analysis

How much of a treatment effect passes through the mediators? Given one simulated study at a time, estimate the population natural indirect effect under treatment and return a 95% interval.

NIE = E[Y(1, M(1)) − Y(1, M(0))]

Data and splits

Each table contains 500 independent observations. Give the estimator baseline covariates X, binary treatment A, post-treatment mediator(s) M and outcome Y. Each table contributes one estimate in outcome units. When there are several mediators, the intervention shifts their joint distribution.

Training data

20,000 fitting tables from eight source families, with the population NIE for each table. Use them to train a new method or reproduce the data exposure of the MediPFN Arena run.

Download training stages ↗

Public starter kit

48 training, 16 development and 32 public reference test tables. Includes known effects, a generator, baseline, scorer and a filled example submission.

Download starter ZIP ↓

Official test inputs

2,000 observed tables, variable types, five-fold assignments and a blank predictions CSV. Maintainers hold the population effects for official scoring.

Download official test ZIP ↓
Preview one labeled study

Development study dev_0001 contains 500 observations generated from the He–Song–Xu linear mediation model.

First three observations
X1X2AMY
0.2103103.25592.8887
0.8381003.32083.2593
−0.1772011.20831.8284

The population NIE is 0.0409. The starter's additive OLS estimate is 0.0511, with a 95% bootstrap interval of [0.0212, 0.0831].

Download all 500 observations (CSV)
How were the training and starter tables generated?

The historical fitting set has 8,000 adapter tables, 10,000 error-head tables and 2,000 fresh fine-tuning tables. Within a preset, a base scenario is sampled and its structural parameters are drawn before generating observations. The fine-tuning stage also reuses prior fitting tables. Across the whole fit, replay is counted once. Separate development and calibration tables were used for model selection and interval calibration.

The two-family starter adapts the linear model of He, Song and Xu and the nonlinear model of Shan, Li and Ai. It draws equal numbers of configurations from both families, varies the path coefficients, and draws two independent tables per configuration. Train, development and public test use disjoint configurations. Each split includes negative, zero and positive indirect effects. Equations, parameter distributions and truth calculations ↗

Official test studies include both familiar and held-out designs. The public test archive provides the observed data and fold assignments. The organizers retain the DGP assignment, generating parameters, seeds and true NIE for each official study.

Identification assumptions and variable roles

The data generator defines the true effect by intervening on treatment and the mediators. Methods receive only the observed variables and design information. A standard mediation formula for observational data requires consistency, positivity and assumptions about confounding and counterfactuals across treatment levels. In particular, a treatment-induced confounder of the mediator–outcome relationship can invalidate that formula. Describe the assumptions your method uses in its method card.

Run a baseline

Unzip the starter kit, open its directory and run the commands below. The baseline fits two covariate-adjusted linear regressions, takes the product of the path coefficients, and constructs a 95% interval from 200 row-bootstrap draws. It gives you a file to compare against when integrating your own estimator.

python -m pip install numpy
python baseline.py --tables dev/tables --output predictions.csv
python score.py --truth dev/truth.csv --predictions predictions.csv

After choosing your settings, replace dev with test in both commands to run the public reference test.

Expected results from the starter's linear baseline
Public splitBiasRMSE95% coverageMean interval length
Development · 16 tables−0.31540.664575.00%0.1626
Reference test · 32 tables−0.41440.859675.00%0.3751

The additive baseline omits the interactions in the nonlinear family. The official leaderboard includes a separate random-forest baseline. Run MediPFN and the forest on a worked example ↗

Official evaluation

Fix your method before running the official test. Each of the 1,000 parameter configurations generates two independent studies with the same population NIE. The ZIP contains observed CSVs in tables/ and matching, row-aligned assignments in folds/. Cross-fitted methods should use the supplied five outer folds. Each CSV column is described by role and type in the archive.

Each of the 2,000 studies gets equal weight. For each study, subtract its true effect from its estimate. Bias is the average of these errors. RMSE is the square root of their mean square. Coverage is the share of 95% intervals that contain the true effect. Mean interval length is the average width of those intervals.

The leaderboard also reports within-mechanism sampling variance. For each of the 1,000 mechanisms, we square the difference between its two independent estimates and divide by two, then average these values over mechanisms. The public starter scorer reports bias, RMSE, coverage and mean interval length. If you report runtime, include the mean seconds per study, hardware and what the timing covers. View official results →

Submit a method

Fill the provided predictions_template.csv with one row per official table. Keep the opaque table IDs and the four column names below. The point estimate and both interval endpoints must be finite, with the lower endpoint at or below the upper endpoint.

table_id,estimate,ci_lower,ci_upper

Use the GitHub form to send a versioned code repository or Hugging Face model with run instructions, or a public method card with your official predictions CSV. Describe your estimator, interval construction and training-data exposure. A classical method can specify "no prior training."

Submitting opens a public issue for review. Once maintainers validate all 2,000 predictions and score them against the private effects, they publish the approved method and its aggregate scores on the mediation leaderboard. If you submit code alone, it must produce the official predictions before the method can be ranked.

Back to the mediation leaderboard · Data release v1.0