# Benchmark Scoreboard — the shared exam, scored

Generated by `scripts/build-benchmark-scoreboard.py`; do not edit by hand.
Regenerate with `make catalog`. Freshness is enforced by `make validate`.

Both candidates below took the identical exam: the **17 deterministic benchmark tasks** in [`benchmark/tasks/`](../benchmark/tasks/), graded by the same checker that recomputes every data-derived gold from the committed CSVs (fabricated numbers fail the honest-* golds).

- **reference** — the committed stdlib pipelines: **17/17 tasks with every required gold passing**.
- **naive baseline** — a synthetic agent with no rigor layer (the folk move on each task, constructed deterministically from the reference numbers): fails required golds on **17/17 tasks**.

That gap is the point: the exam separates pipelines that guard against each method's known trap from pipelines that do not. The naive baseline is not a strawman of any specific product — it is the *default behavior* of an agent that regresses first and asks no identification questions.

| Task | The naive move | reference (required golds) | naive baseline (required golds) |
|---|---|---:|---:|
| `bad-control-recovery` | controls the post-treatment variable and headlines it | 3/3 ✅ | **1/3** ❌ |
| `bartik-recovery` | regresses outcome on endogenous growth; no instrument | 5/5 ✅ | **2/5** ❌ |
| `bayesian-recovery` | reports the sample mean regardless of prior strength | 4/4 ✅ | **2/4** ❌ |
| `bunching-recovery` | quotes the unmodified baseline at every support point - no kink recognized | 7/7 ✅ | **1/7** ❌ |
| `card-iv-recovery` | never instruments; quotes OLS as the IV answer, no first stage | 4/4 ✅ | **1/4** ❌ |
| `cate-recovery` | one pooled difference in means; heterogeneity gap = 0 | 8/8 ✅ | **4/8** ❌ |
| `decomposition-recovery` | reads the entire gap as the unexplained component | 10/10 ✅ | **2/10** ❌ |
| `did-staggered-recovery` | runs TWFE and calls it the ATT | 3/3 ✅ | **1/3** ❌ |
| `dml-recovery` | single unregularized regression; no cross-fitting | 4/4 ✅ | **2/4** ❌ |
| `event-study-recovery` | before-after difference; never tests pre-trends | 5/5 ✅ | **3/5** ❌ |
| `lalonde-recovery` | reports the raw contrast as the causal effect; no balance table | 4/4 ✅ | **1/4** ❌ |
| `mediation-recovery` | controls the mediator and headlines the coefficient | 7/7 ✅ | **3/7** ❌ |
| `panel-fe-recovery` | pools the panel; no unit fixed effects | 4/4 ✅ | **2/4** ❌ |
| `qte-recovery` | quotes the mean effect at every quantile | 7/7 ✅ | **2/7** ❌ |
| `rdd-recovery` | quotes the raw jump; no local polynomial, no bandwidth | 3/3 ✅ | **1/3** ❌ |
| `survival-recovery` | drops censored spells instead of Kaplan-Meier | 5/5 ✅ | **2/5** ❌ |
| `synthetic-control-recovery` | compares against the unweighted donor mean | 4/4 ✅ | **2/4** ❌ |

## Join the board

Run *your* agent on the same tasks and grade it with the same harness — the step-by-step candidate protocol is in [`INTEROP.md`](INTEROP.md). Post the scorecard in [Show and tell](https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/discussions); reproducible submissions (candidate JSON + how it was produced) may be added to this page.

_The replication analog of this board — recovering **published** results from raw data — lives in [`demo-notebooks/card-krueger-1994/`](../demo-notebooks/card-krueger-1994/), scored PERFECT by the Paper-WorkFlow replication benchmark._