Evidence scorecard¶
Generated by python scripts/build_scorecard.py from the committed
artifacts of the source tree at package version 1.43.1.
On main that tree can be ahead of the release with the same number.
The page is regenerated before every release; for live numbers call
sp.parity_summary().
No single number shows that one statistics package is better than another. What a careful user can check is whether each estimator reproduces the reference implementation, whether its intervals cover at the nominal rate, how fast it is on the same task, and what has not been checked. This page reports all four, including the failures.
1. Correctness against reference implementations¶
The denominator is the estimators: result classes, plots, exporters,
data loaders and other infrastructure are excluded, because they can
never carry a numerical grade (sp.parity_summary()['denominators']).
| Measure | Estimators | Share |
|---|---|---|
| Registered estimators | 1,009 | 100% |
| With numerical evidence of any kind | 1,009 | 100.0% |
| Same data, matched to an R / Stata / author reference (T2) | 576 | 57.1% |
bit-exact (relative gap <= 1e-6 on estimate and SE) |
501 | 49.7% |
aligned (documented looser tolerance and mechanism) |
75 | 7.4% |
Known truth only (analytical-only) |
379 | 37.6% |
Published numbers reproduced (external-replication) |
54 | 5.4% |
No evidence (unverified, listed in section 4) |
0 | 0.0% |
The Track A harness runs 91 modules on the same CSV bytes in Python and R; 87 of them also run in Stata. Over the 98 estimators whose headline row comes from Track A, the relative gap to the reference (the larger of the R and Stata sides) is as follows (66 of them report a standard error):
| Quantity | Median | 90th percentile | Maximum |
|---|---|---|---|
| Point estimate | 1e-11 | 4e-07 | 6e-03 |
| Standard error | 3e-09 | 3e-03 | 7e-02 |
Rows above 1e-6 each carry a registered mechanism
(tests/r_parity/compare.py::ROW_GAP_NOTES); per-function detail is
sp.parity_status(name) and the full matrix is parity.md.
2. Calibration of confidence intervals¶
Each row fits the estimator to B independent draws of a DGP whose
true effect is known and counts how often the reported 95% interval
covers it. A row passes when its coverage is at least the lower end of
the 99% Wilson band around 0.95 (0.929 for B = 1,000)
and at most 0.02 above its upper end (0.965); slight
over-coverage is conservative, under-coverage is the failure that
matters. This is the rule test_coverage.py applies.
SE/SD near 1 means the reported standard error matches the sampling
spread of the estimate.
11 of 13 rows inside the band. The rows outside it
are explained in
tests/coverage_monte_carlo/FINDINGS.md;
they are reported, not removed.
| Estimator and DGP | B | Coverage | In band | SE/SD |
|---|---|---|---|---|
sp.regress (HC1) on RCT |
1,000 | 0.952 | yes | 1.00 |
sp.regress 2x2 DiD |
1,000 | 0.955 | yes | 1.03 |
sp.ivreg (HC1) on strong-Z IV |
1,000 | 0.962 | yes | 0.98 |
sp.callaway_santanna simple ATT (staggered) |
1,000 | 0.947 | yes | 1.01 |
sp.sun_abraham overall ATT (staggered) |
1,000 | 0.950 | yes | 1.02 |
sp.panel two-way FE |
1,000 | 0.948 | yes | 0.99 |
sp.fast.feols two-way FE (CR1 by unit) |
1,000 | 0.955 | yes | 0.99 |
sp.rdrobust sharp (robust CI) |
1,000 | 0.934 | yes | 0.98 |
sp.sdid placebo (1 treated) |
1,000 | 0.928 | no | 0.99 |
sp.ebalance (CIA, ATT) |
1,000 | 0.945 | yes | 0.97 |
sp.causal_question(design='dml') IRM ATE |
1,000 | 0.968 | yes | 1.04 |
sp.dml(model='plr') theta |
1,000 | 0.883 | no | 0.88 |
sp.causal_question(design='causal_forest') AIPW ATE |
1,000 | 0.959 | yes | 1.04 |
3. Speed against the reference on the same task¶
Both sides read the same input file (SHA-256 checked), run on one
thread, and are timed in the same session; the table shows the
largest size measured. Timings were taken on StatsPAI 1.40.2;
they are re-measured when a paper is re-anchored, not on every
release. Full tables, including the agreement of the two answers:
tests/perf/results/perf_table.md.
| Task | Reference | n | StatsPAI (s) | Reference (s) | Result |
|---|---|---|---|---|---|
| HDFE, two absorbed FE | fixest::feols |
1,000,000 | 0.0315 | 0.0770 | StatsPAI 2.4x faster |
| CS-DiD ATT(g,t) + aggregation | did::att_gt |
125,000 | 0.0345 | 0.0810 | StatsPAI 2.3x faster |
| Synthetic control (ADH) | Synth::synth |
100 | 1.2995 | 4.5940 | StatsPAI 3.5x faster |
| DML PLR, linear learners, 5 folds | DoubleML (Python) |
10,000 | 0.0079 | 0.0087 | tie |
4. Known gaps¶
Estimators with no numerical evidence (0): none.
Families where the fewest estimators are matched to a reference. Many of these are frontier methods (conformal, neural, LLM-assisted) with no established reference implementation, where known-truth recovery and Monte Carlo coverage are the honest ceiling.
| Family | Estimators | Matched to a reference | Share |
|---|---|---|---|
bayes |
14 | 0 | 0.0% |
causal_rl |
6 | 0 | 0.0% |
fairness |
6 | 0 | 0.0% |
neural_causal |
5 | 0 | 0.0% |
dag |
9 | 2 | 22.2% |
causal_discovery |
16 | 4 | 25.0% |
conformal_causal |
12 | 3 | 25.0% |
iv |
8 | 2 | 25.0% |
tmle |
8 | 2 | 25.0% |
nonparametric |
10 | 3 | 30.0% |
metalearners |
13 | 4 | 30.8% |
forest |
27 | 9 | 33.3% |
Agent benchmark. CausalAgentBench v2 (50 tasks, StatsPAI vs the
Python stack vs R, two model providers) is pre-registered in
tests/agent_bench/; its production run waits on the OSF deposit, so
no agent-performance claim is made here yet.
Reproduce¶
python tests/r_parity/compare.py # Track A table (R / Stata goldens committed)
python tests/r_parity/verify_reproduce.py # re-derive the R goldens (needs R)
python tests/stata_parity/verify_reproduce_stata.py # re-derive the Stata goldens (needs Stata)
python tests/coverage_monte_carlo/run_b1000.py # Track B, B = 1000
bash tests/perf/run_when_idle.sh # Track C timings (idle machine)
python scripts/build_scorecard.py # this page