R-parity tolerance audit¶
tests/r_parity/compare.py::TOLERANCES is the single pre-registered
tolerance budget for the StatsPAI ↔ R (and, where materialized,
↔ Stata) parity harness. Every number in that table is a claim: "on
the committed golden fixtures, StatsPAI and the canonical reference
agree to within this relative difference." A reviewer who sees a 500%
entry with no justification is right to discount the whole table, so
this document grades every loose entry, records the gap actually
observed in the committed results/*.json artifacts, and lists —
honestly — the entries we cannot yet justify.
Audited 2026-06-10 at 64 materialized modules; re-audited 2026-09-05 at 87 modules after the JSS parity-closure pass (Sun–Abraham, GLMM, PPML-HDFE, panel SFA, VAR, TWFE event study) and the addition of modules 86 (fect) and 87 (interflex). Re-audited 2026-09-22 at 89 modules (88 rdbwselect and 89 rdms joined the machine tier; 27 glmm_aghq moved to the iterative tier, see the loosening record below). The budget is enforced by
tests/test_parity_harness_contract.py::test_headline_passes_are_inside_registered_r_tolerance
(R side) and
test_stata_headline_over_budget_modules_are_explicitly_registered
(Stata side); the golden JSONs themselves are hash-locked by
TIER_A_FIXTURE_LOCK.json, so neither side of any comparison can drift
silently.
The three tolerance tiers¶
Every entry belongs to one of three regimes. Conflating them in a single column is what makes a tolerance table look arbitrary, so name the regime explicitly:
- Machine precision (
1e-6and below). Same estimand, same convention, closed form or tightly converged optimizer. The residual is floating-point noise (typically1e-15to1e-9; cross-BLAS reassociation in sandwich "meat" sums can reach~1e-8, seeverify_reproduce.py::REPRO_TOL_OVERRIDE). 79 of 89 modules register here on the point estimate. - Convention gap (
1e-4to5e-2). Both implementations are correct, but they compute a documented different quantity: degrees-of-freedom divisors (TvsT−k), small-sample cluster corrections (ssc), expected vs observed information, analytic vs influence-function SEs. The budget bounds the size of the named convention difference, and the mechanism must be stated next to the entry. - Methodological (T3/T4, above
5e-2). The two sides cannot agree deterministically: independent forest RNG (combined Monte Carlo error), bootstrap vs delta-method inference, non-unique SCM donor weights. The budget bounds a documented methodological disagreement, the verdict is graded T3/T4 rather than treated as an ordinary deterministic pass, and the residual-noise source is recorded in the module'sextrablock.
Orthogonal to all three is the reproducibility tolerance
(verify_reproduce.py, 1e-9): same code, same data, same packages
must reproduce the committed golden values nearly bit-exactly. A parity
tolerance never excuses a reproducibility drift.
Grading scheme¶
Each entry at or above 5e-2 (plus the formerly loose entries) gets a
grade:
- A — mechanistic. The two sides compute different, individually documented quantities by construction. The specific method on each side is named (from our source and the R function's documented method).
- B — empirical. Like-for-like comparison with residual numerical or Monte Carlo noise; we report the observed gap and the margin (tolerance ÷ observed gap).
- C — unjustified. Flagged for future work; no mechanism pinned and/or the budget does not actually bound the rows it appears to cover.
Audit rules. Never loosen a value without re-registering it here.
Tighten when the observed gap (recomputed from the committed JSONs,
worst across the R and Stata sides) is more than 5× smaller than the
budget, to ≈3× the observed gap rounded to a clean number, floored at
the harness-wide 1e-6 machine tier.
Scope of the 2026-06 audit round: the 5× rule was first applied to
the legacy loose tier (budgets ≥ 1e-2). A follow-up pass then swept
the mid-tier modules whose margin also exceeded 5×, recomputing the
worst observed rel_se across both reference sides from the
committed JSONs and tightening each to ≈3× that gap (floored at the
1e-6 machine tier):
| module | old rel_se |
obs worst (both sides) | new rel_se |
|---|---|---|---|
42_nbreg |
1e-2 |
1.4e-3 |
5e-3 |
43_heckman |
1e-3 |
8.6e-5 |
5e-4 |
28_frontier |
1e-4 |
1.3e-5 |
5e-5 |
44_mlogit |
1e-4 |
1.2e-5 |
5e-5 |
41_tobit |
1e-3 |
2.0e-6 |
1e-5 |
14_ols_cluster |
1e-3 |
6.1e-9 |
1e-6 |
24_coxph |
1e-3 |
2.6e-15 |
1e-6 |
46_clogit |
1e-3 |
2.7e-9 |
1e-6 |
49_oprobit |
1e-3 |
3.0e-7 |
1e-6 |
45_ologit (margin 5.1×, obs 2.0e-6) sits at the rule boundary and
is left at 1e-5. No value was loosened; the harness contract test
and the offline render both pass at the new budgets.
2026-10: 43_heckman rel_se 5e-4 → 1e-6. The 8.6e-5 gap the
budget covered was not optimiser noise. The two-step variance used the
expected information of the first-step probit, while R sampleSelection
and Stata heckman, twostep use the observed information. With the
observed information the worst gap is 5.6e-11 against R and 4.3e-9
against Stata. This is the decision tree's step 1: a gap that had been
given a tolerance was bisected and turned out to be on our side.
1.32: 40_qreg rel_se 0.10 → 1e-6 (removed from the ≥ 5e-2
table). The old row was graded A as "different sparsity estimators by
construction": StatsPAI's only SE was a Silverman-bandwidth Gaussian-kernel
iid sandwich (7.3% / 3.0% off), and it described quantreg's se="nid" as
Stata qreg's default, which it is not (Stata's default is vce(iid) with
the fitted-quantile sparsity). sp.qreg now offers vce="iid" (Stata's
default, and the new default), "robust" (Stata vce(robust)) and
"nid" (quantreg); the module runs vce="nid" against quantreg se="nid"
(1e-15) and Stata qreg, vce(robust) (2e-8: Stata zeroes fitted
differences below sqrt(eps) where quantreg subtracts sqrt(eps)).
1.32: 41_tobit rel_se 1e-5 → 1e-6. The 2.0e-6 gap above was
the second-difference Hessian and BFGS's gtol=1e-6 stop; sp.tobit now
Newton-polishes on complex-step scores (the shared ML path of truncreg /
biprobit), and the worst joined gap is 3.4e-11 (R) / 1.1e-11 (Stata).
Row-level gap register (2026-10-27)¶
A module budget says how far its rows may drift. It does not say why a row
is not at machine level, and a budget justified by one row can cover
another. The Oaxaca Stata SE sat 1.2% away for months under a 5e-2 budget
written for the R bootstrap, and nobody had asked why Stata's delta method
disagreed with ours. Since 2026-10-27 every joined row of a T2 module whose
gap exceeds 1e-6 (relative, and 1e-8 absolute) must match an entry of
compare.py::ROW_GAP_NOTES. This applies on either reference side, for the
estimate and for the SE. Each entry is one of three kinds:
- convention: the reference computes a documented different quantity.
The entry names
pinned_byrows that run StatsPAI under the reference's convention, and these rows must join at machine level. - reference_numerics: the reference's optimiser, root finder, numerical derivative or approximate covariance stops short. Where a second reference or a converged run exists, it is the pin.
- stochastic_reference: a simulated or resampled quantity. Such a row is a stochastic screen and is never a parity fact.
Three contract tests hold the table to the artifacts. Every row over
machine level must be registered, no entry may be stale, and every pin must
join at machine level. The SE budget tests skip registered rows, so 18
modules' SE budgets could drop to 1e-6.
Building the register found the following:
| Module | Before | Mechanism found | Pin |
|---|---|---|---|
30_oaxaca Stata SE |
1.2% under R's bootstrap budget | classical vs HC1 group regressions, and stochastic vs fixed covariate means | oaxaca, vce(robust) fixed: 2e-14 on the three threefold components |
76_pretrends |
4e-4, attributed to pmvnorm noise; iterative tier |
pretrends fixes the cutoff at 1.96, pmvnorm runs at abseps 1e-3, uniroot at its default tolerance |
threshold=1.96 against an exact integrator and a 1e-14 root: 1.6e-8; machine tier |
48_probit R SE |
0.3% | glm reports expected information |
sampleSelection::probit (observed information): 1e-12 |
42_nbreg R SE |
0.14% | glm.nb holds θ at its estimate |
model_info['se_conditional_on_dispersion']: 3e-13 |
61_betareg R SE |
0.7% | betareg reports expected information |
hessian = TRUE: 1.2e-7 (accuracy of optim's numerical Hessian) |
51_newey Stata SE |
0.5% under a 1e-2 budget |
newey applies N/(N−K) |
hac_small=True / adjust = TRUE: 4e-16 |
67_panel_glm R SE |
1e-5 | fixest stops IWLS at glm.tol = 1e-8 |
glm.tol = 1e-12: 1e-15 (logit), 1e-7 (Poisson) |
28_frontier |
1.3e-5 off R and Stata | StatsPAI: round-off of a fixed-step Hessian | fixed in the package (Richardson Hessian + Newton finish): 1e-8 |
29_panel_sfa Stata |
3e-6 | StatsPAI stopped short; fixed-step Hessian | same fix: 2e-10 / 8e-10 |
07_scm Stata |
4.2e-4, read as solver tolerance | synth builds e(Y_synthetic) from weights rounded to 0.001 |
StatsPAI weights rounded the same way: 5e-16 (*_w3dp) |
12_sdid Stata |
7.2e-8 | sdid stores e(ATT) via strofreal() (7 digits) |
full value read from Mata: 1e-15 |
What the fixtures themselves turned up: %.16g does not round-trip every
double. The Python side used to fit the in-memory frame, while R and Stata
fitted the parsed text. dump_csv now swaps in the correctly rounded parse,
and Stata's import delimited returns exactly those doubles. R's read.csv
(and readr::parse_double) differs in the last bit on 10–36% of values on
the fixtures measured (03, 13, 28, 89). R therefore sees an input
perturbation of about 1e-16 relative. That is the floor below which a py–R
agreement says nothing, and it is far below every budget.
Decision (2026-10-27): leave the R side on read.csv. Every R reader we
tried misparses the same values (read.csv, scan / as.numeric,
data.table::fread; readr slightly more often: 42 / 42 / 42 / 53 of 1,540
values on 43_heckman, 39 / 39 / 39 / 40 of 3,000 on 30_oaxaca, none on
01_ols), so there is no drop-in fix. A correctly rounded R reader (e.g.
through Rmpfr) would change every R golden for a gain at the 1e-16 level,
and any row it could matter for would already show up above 1e-6 in
ROW_GAP_NOTES. Revisit only if a registered row is ever traced to it.
Reproducing the audit¶
Run from the repository root. This recomputes, for every module, the worst joined SE gap across both reference sides and flags sentinel (no-joined-SE-row) budgets and over-budget non-headline rows:
import importlib.util
import sys
from pathlib import Path
spec = importlib.util.spec_from_file_location(
"compare", Path("tests/r_parity/compare.py")
)
compare = importlib.util.module_from_spec(spec)
sys.modules["compare"] = compare
spec.loader.exec_module(compare)
for module, tol in sorted(compare.TOLERANCES.items()):
rows = compare.collect(module)
worst_se = max(
(v for d in rows for v in (d.rel_se, d.rel_se_st) if v is not None),
default=None,
)
budget = tol.get("rel_se", tol.get("abs_se"))
if worst_se is None:
print(f"{module:22s} rel_se budget {budget:8.2g} no joined SE row (sentinel)")
elif worst_se > budget:
print(f"{module:22s} rel_se budget {budget:8.2g} OVER: observed {worst_se:.3g}")
The two enforcement commands (both must pass after any tolerance change):
# from the repository root
python3 tests/r_parity/compare.py # re-render parity tables (idempotent)
python3 -m pytest tests/test_parity_harness_contract.py -q -o addopts=''
Entries at or above 5e-2 — full justification¶
Observed gaps are recomputed from the committed
tests/r_parity/results/<module>_{py,R}.json and
tests/stata_parity/results/<module>_Stata.json artifacts (worst row;
R side / Stata side). "Margin" is tolerance ÷ worst observed gap.
| Module | Quantity | Tolerance | Grade | Observed gap (R / Stata) | Justification |
|---|---|---|---|---|---|
07_scm |
rel_est |
1.0 | A (T4) | headline 0.0239 vs R; donor weights up to 1.78 vs R, 0.0072 vs Stata | Classical SCM on the Basque fixture (abadie2003economic; method abadie2010synthetic). The donor-weight solution is not unique under the ADH nested-V specification: multi-start diagnostics find multiple near-best weight classes, and R Synth::synth and Stata synth land on measurably different local optima. StatsPAI's native solver tracks Stata. Verdict is GAP (T4 reference-disagreement disclosure), not PASS; module 52_scm_unique certifies exact recovery on an identified DGP. |
13_causal_forest |
rel_est |
0.01 | B (T3) | 0.0026 / — | AIPW doubly robust ATE/ATT on both sides (sp.causal_forest vs grf::causal_forest + average_treatment_effect; wager2018estimation, athey2019generalized). The two forests cannot share RNG, so this row is one draw per engine; the T3 grade rests on refitting both engines under 50 seeds on fixed data (tests/reference_parity/test_grf_seed_mc_equivalence.py): their gap is below 0.1 sampling SE (TOST) at 500, 2,000 and 8,000 trees on this design and on a second fixture, and the two engines' seed-to-seed SDs agree within a factor of two (seeds spaced 1e5 apart; grf's consecutive seeds share most of their draws). (Before 1.31 the row was graded by dividing the single-draw gap by the combined sampling SE, which is not the forest's Monte Carlo error.) A multi-seed truth-recovery pytest guard backs it. Widened from 0.005 when the ATT row moved onto grf's own plug-in + Hajek-correction estimator: both sides now run the same estimator, so the residual is pure forest MC, and 0.005 left only 1.07× margin against machine-to-machine drift at the time. Since 1.31 (independent per-group seeds and per-nuisance random streams in the engine) the ATE row sits at 0.03% and the ATT row at 0.26%. Margin 3.9×. |
13_causal_forest |
rel_se |
0.05 | B | 0.0068 / — | The AIPW operator is now pinned exactly — fed grf's own forest outputs, StatsPAI reproduces grf::get_scores elementwise to 2.3e-14 and grf's ATE/ATT estimate and std.err to 1e-15 (tests/reference_parity/test_grf_aipw_operator_parity.py) — so this budget covers forest RNG only, not an unresolved formula difference. Tightened from 0.50 to 0.25 after the ATT convention fix removed the historical 14.6% ATT row, and to 0.05 in 1.29 when the GRF engine brought the worst row to 0.63%. Since 1.31's seeding fix the ATE SE is 0.16% and the ATT SE 0.68% from grf's. Margin 7.3×. |
Only the T3 forest and the T4 SCM remain in this table. On 2026-10-27
05_sunab, 29_panel_sfa, 30_oaxaca and 36_mediation left it: their
SE budgets are now 1e-6, and the rows that are not at machine level are
registered one by one in compare.py::ROW_GAP_NOTES (see "Row-level gap
register" below). 06_rd's 0.10 entry was already stale; its SE budget has
been 1e-6 since the native selector replaced the port. All remaining
entries are at 3e-2 or tighter and are graded inline in compare.py. Modules 26_glmm_logit, 27_glmm_aghq and
47_ppmlhdfe_3fe left this table on 2026-09-05: their 1.9% / 1.9% /
1.8% "convention" gaps were StatsPAI defects (fixed-effect covariance
conditional on the variance components; a small-sample factor matching
neither reference) and are now at 1e-2 / 2e-5 / 1e-6.
Tightenings applied 2026-06-10¶
Rule applied: observed gap (worst over R and Stata sides, all joined
rows — stricter than the headline-only enforcement) more than 5×
smaller than the budget → tighten to ≈3× observed, floored at the
harness machine tier 1e-6. No value was loosened.
| Module | Quantity | Old | New | Observed worst gap | Basis |
|---|---|---|---|---|---|
03_hdfe |
rel_se |
1e-2 | 1e-6 | 8.4e-15 | The "1-df convention gap" comment was stale: with ssc='fixest', IID SEs match fixest::feols/reghdfe at machine level on both sides. |
15_hdfe_cluster |
rel_se |
5e-2 | 1e-6 | 1.25e-11 | Stale "ssc convention": CR1 nested-FE cluster SEs now match on both sides. |
30_oaxaca |
rel_se |
1.0 | 0.05 | 1.25e-2 | Grade-A delta-vs-bootstrap gap; 3× observed ≈ 0.0375, rounded to 0.05. |
11_psm |
rel_se |
5.0 | 1e-6 | sentinel (no joined se field) |
att_psm carries se=None on all three sides by design; see "Sentinel entries". Since 2026-09-22 the point-valued se_teffects_ai row joins Stata under rel_est at 7e-8 (se_method='abadie_imbens_2016', the Abadie-Imbens 2016 estimated-score variance teffects psmatch reports); the default se_abadie_imbens (psmatch2 ai(1) ties, 640.41 vs 621.79) is a documented convention -- sample- vs population-ATT target and no estimated-score term. |
12_sdid |
rel_se |
5e-2 | 1e-6 | sentinel | Point-only ATT row (arkhangelsky2021synthetic); placebo SEs are backend-native diagnostics under distinct names. |
16_bjs |
rel_se |
0.25 | 1e-6 | sentinel | BJS imputation (borusyak2024revisiting); SE rows are side-specific (se_cluster_if / se_didimputation / se_stata_did_imputation). |
07_scm |
rel_se |
1.0 | 1e-6 | sentinel | All SCM rows are point-only. |
18_augsynth |
rel_se |
1.0 | 1e-6 | sentinel | augsynth fixture (benmichael2021augmented) emits no joinable SE. The rel_est budget (2e-5) is now grade A: the 7.9e-6 R residual is augsynth::synth_qp's OSQP stopping tolerance (eps 1e-8); with OSQP at 1e-13 augsynth agrees with StatsPAI's exact simplex QP and the audited Stata/Mata bridge (18_augsynth.do) to 1e-11. |
19_gsynth |
rel_se |
1.0 | 1e-6 | sentinel | gsynth fixture (xu2017generalized) emits no joinable SE. An audited Stata/Mata bridge (19_gsynth.do, gsynth's control-only factor convention with r fixed at the R r.cv) joins the ATT at 2.2e-15; fect_stata method(ife) is a different estimator (IFEct) and is kept as an unjoined diagnostic. |
20_bacon |
rel_se |
1.0 | 1e-6 | sentinel | The Goodman–Bacon decomposition (goodmanbacon2021difference, goodmanbacon2019bacondecomp) has no SEs on any side. |
31_dfl |
rel_se |
1.0 | 1e-6 | sentinel | Point-only decomposition rows. |
39_arima |
rel_se |
1e-2 | 1e-6 | sentinel | No SE row joins on this fixture. |
52_scm_unique |
rel_se |
1.0 | 1e-6 | sentinel | All rows point-only. |
Verification: after these edits,
python3 tests/r_parity/compare.py re-rendered all parity tables
byte-identically (the budget does not enter any rendered artifact for
rel_se-only changes, and no rel_est was touched, so the strictness
tiers and the JSS Appendix B tables are unchanged), and
python3 -m pytest tests/test_parity_harness_contract.py -q -o addopts=''
passes every tolerance-related contract. A row-level checker asserting
the new values against all joined SE rows on both reference sides
(not just headline rows) also passes for every tightened entry.
Closed 2026-09-22 (no tolerance change)¶
73_did2sSE promoted to T2.sp.gardner_did's defaultvce='analytic'now computes the did2s corrected two-stage clustered variance (stage-1 estimation error propagated throughγ = (X10'X10)^{-1} X1'X2), matching Rdid2sat 2.7e-10 and Statadid2sat 1.3e-14 on the static ATT and all 26 event-study horizons at 2.1e-14.rel_seregistered at the 1e-6 machine tier; the pre-1.29 stage-2-only SE (25.9% low) is reachable asvce='stage2'and kept as the unjoinedstatic_ATT_stage2_serow.04_csdidStata SE convention pinned as a row.group_overall_fixedshare=sp.aggte(share_variance=False)joinscsdid'sestat groupGAverage at 6.2e-15;STATA_SE_GAP_NOTESkeeps the 0.27% note for the default row only.18_augsynth/19_gsynthStata bridges materialised (see the sentinel table above);STATA_SKIP_REASONnow lists four modules (13, 77, 79, 80). Stata-side reproducibility: 85/85 modules, zero drift.
Loosening applied 2026-09-22: 27_glmm_aghq point estimate 1e-6 → 5e-6 (grade B)¶
The only budget ever loosened under the audit rule, registered here with the measurement that justifies it.
- The likelihood is shared. The
logLikrow joins Rlme4::glmer(nAGQ = 8) at rel 1e-13, andlme4's deviance function (devFunOnly = TRUE) evaluated at StatsPAI's(θ, β)is within 4.7e-11 of its value atlme4's own optimum. Both sides compute the same 8-point adaptive Gauss–Hermite approximation. - The objective is flat. Along the (intercept, √var) direction a 5e-11 deviance step moves the intercept by 5e-7, i.e. the argmax is numerically undetermined at the 1e-6 level for a 1500-observation log-likelihood evaluated in double precision.
- The reference's own optimisers disagree at that level. On the
committed fixture
lme4withbobyqa(the script's setting,rhoend = 1e-12),nloptwrap(xtol_abs = ftol_abs = 1e-14),optimx/nlminbandNelder_Meadreturn intercepts 1.1e-6, 1.5e-6 and 1.4e-5 apart from one another while all reproduce the deviance to 1e-10. Statamelogit, intpoints(8)(defaultmvaghermite, tighttolerance(1e-10) ltolerance(1e-13)) lands 2e-8 from StatsPAI on the intercept; itslogLikdiffers by 5e-6 because Stata's mean–variance adaptive grid is a different 8-point approximation, and all three sides agree to 3e-8 at 30 points (intmethod(mcaghermite)gives the same 30-point numbers to 1e-15). - Budget. 5e-6 ≈ 3× the widest
lme4-vs-lme4spread that still reproduces the deviance to 1e-10. The module therefore reports in the iterative strictness tier (79 machine / 8 iterative / 1 moderate / 1 T4 as of this change), not the machine tier. The SE budget (2e-5) is unchanged; observed 4.8e-7 (R) / 4.2e-6 (Stata). - Guard.
test_glmm_parity_uses_tight_optimizer_solution_with_se_convention_guardadditionally assertslogLikrel < 1e-9, so a future drift in the quadrature itself cannot hide behind the optimiser-noise budget.
Sentinel entries¶
Several modules are point-only by design: their SE estimators differ
by construction, so the harness stores them as side-specific diagnostic
rows with distinct statistic names that never join, and the headline
row carries se=None. Example — module 11_psm: StatsPAI reports the
Abadie–Imbens (2006, eq. 14) sample-ATT SE (se_abadie_imbens = 640.41),
the estimator Stata psmatch2, ai(J) ties reports; MatchIt::matchit
documents no canonical analytic SE for nearest-neighbor matching with
replacement, so the R fixture records a weighted-lm-on-matched-data
diagnostic (se_matchit_lm); Stata teffects psmatch reports its own
Abadie–Imbens variant (abadie2016matching, se_teffects_ai = 621.79),
3.4% below ours. That gap was localised on 2026-09-22 (previously an
open item under the §5 rule): teffects psmatch reports the Abadie &
Imbens (2016) variance for matching on the estimated propensity score,
σ²_δ − c'V_γ c + d'V_γ d, whose two extra terms (−74 905 + 8 036 on
NSW-DW) are negative here, and whose base term targets the population
ATT (it includes the (τ_i − τ)² heterogeneity term) rather than the
sample ATT of eq. 14. sp.match(se_method='abadie_imbens_2016') rebuilds
the number from StatsPAI's own matched frame, within-arm conditional
variances and logit information matrix to rel 6.4e-8 (the residual is
Stata's ML stopping rule), so the point-valued se_teffects_ai row now
joins Stata under the module's rel_est budget of 1e-6. The default
se_abadie_imbens (psmatch2 ai(1)) is unchanged and documented as the
sample-ATT, known-score convention.
For such modules the rel_se budget is vacuous — no value, loose
or tight, is ever exercised. The previous loose values (up to 5.0 for
11_psm) were leftovers from the original 2026-05-04 harness commit,
before the SE rows were split into side-specific diagnostics, and read
as if we tolerated a 500% SE gap. They are now pinned at the 1e-6
machine floor as sentinels: if a future fixture regeneration ever
makes an SE row join, the contract fails loudly and the maintainer must
consciously register a justified budget instead of inheriting a stale
loose one.
Demonstrating a convention gap: the VAR df divisor¶
Module 33_var's SE budget illustrates why "both correct, different
convention" must be stated mechanistically. StatsPAI's default matches
Stata var (conditional-MLE divisor T); vars::VAR runs
per-equation lm() (divisor T−k). The entire R-side SE gap is the
deterministic ratio sqrt(T/(T−k)):
import numpy as np
import pandas as pd
import statspai as sp
rng = np.random.default_rng(42)
n = 200
y1, y2 = np.zeros(n), np.zeros(n)
for t in range(2, n):
y1[t] = 0.5 * y1[t - 1] - 0.2 * y2[t - 2] + rng.standard_normal()
y2[t] = 0.3 * y2[t - 1] + 0.1 * y1[t - 1] + rng.standard_normal()
df = pd.DataFrame({"y1": y1, "y2": y2})
stata_side = sp.var(df, lags=2, se_df="stata") # divisor T (Stata var)
r_side = sp.var(df, lags=2, se_df="r") # divisor T-k (vars::VAR)
T = stata_side.n_obs
k = 2 * 2 + 1 # 2 lags x 2 variables + constant per equation
ratio = np.asarray(r_side.se["y1"]) / np.asarray(stata_side.se["y1"])
print(f"observed SE ratio = {ratio.flat[0]:.6f} "
f"(identical for every coefficient: {np.allclose(ratio, ratio.flat[0])})")
print(f"sqrt(T / (T - k)) = {np.sqrt(T / (T - k)):.6f}")
Output: observed SE ratio = 1.012871 … — the ratio−1 form (1.287%)
of the max-normalised 0.0127 gap recorded for every 33_var SE row in
parity_table.md (the committed fixture has T=198, k=5,
sqrt(198/193) − 1 = 1.287%; the table normalises by the larger side).
Known weak spots¶
The honest list, re-audited 2026-09-05. Every entry that used to sit here as grade C or "budget bounds nothing" has been closed by locating the mechanism on the StatsPAI side and, where a like-for-like row exists, pinning both references at machine level. The closures are kept below (struck through) because the pattern is the lesson; the open items follow.
- ~~
29_panel_sfa(rel_se1e-3) — grade C.~~ Closed. The Stata reference was a different likelihood (truncated-normal Battese–Coellitiwith freemu); constrained tomu = 0it agrees with StatsPAI to 3e-6 on every SE and 1e-6 on every point row, so the registered 5e-2 now bounds onlyfrontier's approximate covariance (graded B above). - ~~
33_var(rel_se1e-3) — budget mis-keyed.~~ Closed. The Python side emitseq_*(divisorT, Stata) andeq_*__Tk(divisorT-k, Rvars::VAR) rows and each reference side emits only its own convention; every compared SE is at 1e-15 and the budget is 1e-6. - ~~
05_sunab— fixest-side mechanism unpinned.~~ Closed. The 17.1% figure came from the old R script re-aggregating cohort×period cells by hand without their covariance; the nativefixestaggregation is now used. The residual 0.08% at single-cohort times was a StatsPAI degrees-of-freedom defect (nine all-zero cohort×relative-time columns counted in K, time effects not counted), fixed in 1.24.0. What remains is the documented Sun–Abraham Prop. 3 share term (mechanism A), demonstrated by theatt_rel_<e>_fixedsharerows matchingfixestto 8e-12 and the default rows matching Stataeventstudyinteractto 8e-12. - ~~
26_glmm_logit/27_glmm_aghq— SE convention not derived.~~ Closed. Not a convention: the fixed-effect covariance omitted the variance-component uncertainty. The full observed-information Hessian of the marginal log-likelihood (Statavce(oim)) puts AGHQ at 5e-7 (R) / 4e-6 (Stata) and Laplace at 4e-6 (Stata); the remaining 0.27% on the Laplace R side islme4's own optimum sitting 1.5e-4 away in β (itslogLikis 2.8e-7 lower). - ~~
47_ppmlhdfe_3fe— R-side residual unpinned.~~ Closed. StatsPAI applied(N-1)/(N-k)withk= slopes only, matching neither reference.ssc="stata"(N/(N-1)) matchesppmlhdfeto 1e-11 andssc="fixest"(N/(N-K), K counting absorbed FE levels) matchesfixest::fepoisto 1e-8; modules 37 and 47 emit one row per convention. 13_causal_forestrel_se0.05 — T3 by construction. The residual is forest-RNG dispersion intau.hatand cannot reach T2; the binding evidence is the exact AIPW-operator pin plus the Track B coverage sweep, not the relative band (observed worst 0.68% since 1.31).- ~~Headline-only enforcement.~~ Closed.
test_every_r_se_row_is_inside_budgetgates the registeredrel_seon every R-joined SE row of every PASS module (all 87 pass), andtest_every_stata_se_row_is_inside_budget_or_registeredrequires any Stata SE row over budget to carry a mechanism incompare.py::STATA_SE_GAP_NOTES(two entries:04_csdidgroup_overall0.27%, fixed-share aggregation;71_dml_familyPLIV,ivreg'sN/(N-K). The83_lpdidlead-row entry, 3.9e-4 and attributed toreghdfe's K, was removed on 2026-09-07: the gap was a StatsPAI lead-sample rule one period stricter thanlpdid's, and the rows now agree to 2e-16). - ~~
04_csdidgroup_overallStata SE (0.27%) — open.~~ Closed (2026-09-05).csdid'sestat groupGAverage aggregates the per-cohort influence functions with the cohort shares held fixed;did::aggte(type = "group")andsp.aggteadd the share-estimation term (did:::wif). Rebuilding the fixed-share aggregate from StatsPAI's joint cell influence functions reproducescsdid's SE to 1e-14 (the earlier independent-cell attempt overshot because it dropped the cross-cell covariance). Mechanism A, registered inSTATA_SE_GAP_NOTESand pinned bytest_csdid_group_overall_stata_se_is_the_fixed_share_aggregation. - Contract-frozen values.
tests/test_parity_harness_contract.pyasserts exact equality for several budgets (e.g.26_glmm_logit,27_glmm_aghq,36_mediation,38_drdid,10_honest_did,11_psmrel_est). Any future change must touch both files in the same commit, deliberately.
Last audited: 2026-10-27 (row-level register; 89 parity modules). Previously 2026-09-22 (StatsPAI 1.29.0 source snapshot). Re-run
the snippet above and refresh this document whenever a TOLERANCES
entry changes.