Skip to content

Stability and Validation Tiers

StatsPAI now separates API lifecycle from numerical validation evidence. This is the main correction to the older catalogue: stability='stable' no longer means "R/Stata parity-grade" by itself.

Three Fields

Field Scope Meaning
stability whole function API stable, experimental, or deprecated
validation_status evidence for numerical output certified, validated, api_stable, experimental, or deprecated
limitations parameter/variant gaps documented unsupported variants inside an otherwise usable function

Use stability when you care about public API compatibility. Use validation_status when you care about audited numerical evidence.

Current JSS source-snapshot audit counts: 494 certified, 487 validated, 642 api_stable, and 3 experimental registry symbols. The intentionally harsh denominator is that 569 stable auto-registered symbols still lack parity backing; treat them as API-stable, not numerically validated. The audit decomposes that denominator into class-like/function-like and category counts, so breadth remains auditable rather than becoming a hidden validation claim. Within the hand-written stable API surface, the current audit enforces zero unbacked entries: API-only helpers carry unit-contract evidence while remaining api_stable, not numerically validated. Unit and regression tests are API-contract evidence; they do not promote a function to validated without known-truth, reference-parity, external-parity, coverage, or explicit convention evidence. Paper-JSS/replication/results/validation_evidence_audit.{json,md} verifies that all 542 certified/validated symbols have registry-attached evidence notes, that certified symbols carry attached evidence naming an external reference implementation and the tolerance it was held to, and that validated symbols are not backed only by unit/regression tests. Package metadata in the JSS source snapshot is 1.29.0; the source snapshot should still be synchronized with a clean tagged release before final publication. The JSS archive records this boundary in Paper-JSS/replication/results/source_snapshot_manifest.{json,md}, and cd Paper-JSS && make release-audit is the strict gate for a clean tagged final-publication snapshot.

Stability

  • stable: a breaking change to the signature comes only after a release that emits a DeprecationWarning, with a MIGRATION.md entry.
  • experimental: method/API may shift across minor versions.
  • deprecated: scheduled for removal; replacement should be documented in MIGRATION.md.

Frozen core API

The 32 everyday entry points (sp.list_functions(core=True), listed first by sp.help()) carry a stronger promise: code written against them keeps running. Their signatures are recorded in tests/core_api_snapshot.json, and tests/test_core_api_freeze.py (also a pre-push hook) fails on any change that would break an existing call:

  • a parameter disappears (a rename is fine when the old spelling stays accepted through @accepts_aliases);
  • a default changes, or an optional parameter becomes required;
  • positional parameters are reordered, or one becomes keyword-only;
  • a new parameter has no default;
  • a core function is removed.

New optional parameters are not breaks. An intended break goes through a deprecation release first, then python scripts/core_api_snapshot.py --write --ack "fn.param=<MIGRATION.md text>", which refuses unless the quoted text is in MIGRATION.md and keeps the acknowledgement in the snapshot's history.

Every function card says which promise applies, in api_contract:

api_contract Applies to Promise
frozen the core API snapshot-tested, as above
deprecation-cycle other stable functions breaking changes announced a minor release ahead, by policy
unstable experimental functions may change in any minor release
deprecated deprecated functions scheduled for removal

api_contract is about the signature. Whether the numbers are checked against a reference is validation_status, below; the two are independent.

Validation

  • certified: the function was compared against a named external reference implementation on identical inputs and agreed within a pre-registered tolerance — usually tests/r_parity/ + tests/stata_parity/, or a frozen R fixture in tests/reference_parity/. Two rows follow CLAUDE.md §5.1 to a Python reference because that is the implementation the method's own authors maintain: sp.metalearner against econml and sp.dml_sensitivity against DoubleML.

The tier is derived, not hand-assigned. src/statspai/_parity_index.json is built from the committed goldens by scripts/build_parity_index.py, and statspai._parity_taxonomy.validation_tier_for is the single map from a parity grade to a tier, shared by the registry and the index. A test asserts the two cannot disagree for any stable symbol, so sp.parity_status(name) and sp.describe_function(name)["validation_status"] always tell the same story — including in an installed wheel, where the index ships in the package. - validated: known-truth recovery (a closed form or a deterministic DGP with a known parameter) or replication of published numbers, with no same-byte cross-language comparison. Monte Carlo coverage rows and documented convention disclosures are recorded against a function but never earn a tier on their own (statspai._parity_taxonomy.validation_tier_for). - api_stable: stable public API. Unit/regression tests may attach API-contract evidence here, but that evidence is not numerical validation. - experimental: mirrors stability='experimental'. - deprecated: mirrors stability='deprecated'.

Filtering

import statspai as sp

sp.list_functions()                              # all registered functions
sp.list_functions(stability="stable")            # stable API
sp.list_functions(validation_status="certified") # parity-backed functions
sp.agent_cards(validation_status="certified")    # parity-backed agent cards

spec = sp.describe_function("regress")
spec["stability"]          # "stable"
spec["validation_status"]  # "certified"
spec["validation_notes"]   # parity artifact / reference notes

statspai list --stability experimental
statspai list --validation certified
statspai describe rdrobust

sp.help() prints both STABILITY and VALIDATION count blocks. Per-function help shows Stability:, Validation:, Evidence:, and Known limitations when available.

Promotion Path

  1. Promote experimental to stable when the public API is ready for SemVer compatibility.
  2. Promote api_stable to validated when analytic/reference parity tests exist.
  3. Promote validated to certified when the function enters the cross-language or published-reference parity harness.
  4. Remove a limitation only when the unsupported variant lands with its own test.

Current Limitation Hotspots

These are machine-readable through sp.describe_function(name)["limitations"] and should be treated as the priority backlog for production hardening:

  • callaway_santanna: repeated cross-sections (panel=False) support estimator="dr" / "ipw" / "reg" (Sant'Anna-Zhao drdid_rc / std_ipw_did_rc / reg_did_rc), both control groups and the multiplier bootstrap. Point estimates match R did::att_gt(panel=FALSE); the simple-ATT standard errors carry a ~0.15% gap from how the aggregation denominator is formed and are held at 1% (tests/reference_parity/test_cs_rcs_parity.py). clustervars= under RCS requires bstrap=True.
  • rdrobust: observation-level weights are reserved and raise NotImplementedError; exact R parity is attached to bwselect="cct" or common manual bandwidths, while the default mserd selector is a documented convention.
  • rddensity: native default bandwidths, mass-point ECDF handling, and jackknife CJM local-density inference mirror rddensity::rddensity on the JSS parity fixture. Manual side-specific bandwidths are still treated as explicit user controls; backend="r" remains available when direct execution of the R package is required.
  • synth: ADH/Synth parity requires the same special_predictors recipe; SDID/augmented/gsynth rows include documented regularisation or local-optimum convention gaps.
  • causal_forest: the NSW-DW parity row is overlap-diagnostic evidence, not a clean ATT point-estimate parity claim.
  • did_imputation: parity is aggregation-convention sensitive; inspect sp.parity_gap_report() before reporting exact cross-language equality.
  • etwfe: the default panel estimate now reports the R etwfe::emfx(type="simple") / Stata jwdid, estat simple treated-observation-weighted simple ATT for cgroup="notyet"; use cgroup="nevertreated" for the R cgroup="never" estimand, or sp.etwfe_emfx(..., weighting="cohort") for the historical StatsPAI cohort-share summary.
  • etwfe / wooldridge_did: cohort-level ATTs changed in 1.27.0. result.detail, sp.etwfe_emfx(type="group") and every weighting="cohort" aggregation were read off an unsaturated cohort x post regression and were off by up to 37% against R etwfe::emfx(type="group"); they are now aggregated from the saturated cohort x period cells. The sp.etwfe pooled ATT is unchanged. See MIGRATION.md.
  • hal_tmle: variant="projection" is reserved and raises NotImplementedError.
  • network_exposure: only design="bernoulli" is implemented.
  • etwfe: panel=False with cgroup="nevertreated" is not implemented.
  • continuous_did: method="cgs" is an MVP without full CGS parity.
  • did_multiplegt_dyn: experimental. Effects, placebos, switcher counts, Av_tot_eff and the analytic influence-function SEs are pinned to DIDmultiplegtDYN / Stata did_multiplegt_dyn (Track A 78 and the castle-doctrine reference test); controls=, trends_nonparam=, normalized= and continuous= are implemented and pinned to DIDmultiplegtDYN 2.3.4 at machine precision (tests/reference_parity/test_dcdh_options_parity.py). Not implemented: trends_lin (an implementation following the reference's documentation does not reproduce its numbers past horizon 0, so it is withheld rather than shipped wrong — see docs/rfc/multiplegt_dyn.md), predict_het, and the heteroskedastic-weights variant; joint tests come from the cluster bootstrap only.

Auditing

python scripts/stability_audit.py
python scripts/stability_audit.py --unbacked
python scripts/stability_audit.py --check
python scripts/stability_audit.py --json
python Paper-JSS/replication/scripts/validation_evidence_audit.py

scripts/stability_audit.py --check fails if any hand-written stable API entry lacks attached validation or API/unit-contract evidence. Auto-registered entries are reported separately because they represent breadth imported into the registry, not the validated numerical core defended in the JSS paper.

The JSS packager also extracts Python source paths from registry evidence notes. The current submission manifest includes 779 such registry evidence files, and Paper-JSS/replication/scripts/verify_submission_package.py fails if any referenced evidence file is absent from the archive.

Programmatic evidence summaries:

sp.validation_report()
sp.coverage_matrix(level="parity")
sp.parity_gap_report()

sp.parity_gap_report() parses the already-generated 3-way parity table and reports documented convention gaps, missing Stata siblings, priorities, and next actions.

Last updated: JSS source-snapshot validation audit (2026-06-28).