statspai.imputation

imputation

Missing data handling and multiple imputation.

Provides MICE (Multiple Imputation by Chained Equations), EM imputation, and analysis tools for multiply-imputed data.

MICEResult

Bases: ResultProtocolMixin

Results from MICE imputation.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(7)
>>> x1 = rng.normal(size=120)
>>> x2 = 0.5 * x1 + rng.normal(size=120)
>>> y = 1.0 + x1 + x2 + rng.normal(size=120)
>>> df = pd.DataFrame({"y": y, "x1": x1, "x2": x2})
>>> df.loc[rng.choice(120, size=15, replace=False), "x2"] = np.nan
>>> res = sp.mice(df, m=3, method="pmm", seed=0)
>>> type(res).__name__
'MICEResult'
>>> res.n_imputations
3
>>> bool(res.complete(0)["x2"].notna().all())
True

complete

complete(m: int = 0) -> DataFrame

Return the m-th completed dataset (0-indexed).

combine

combine(estimates: List[Dict[str, Any]]) -> Dict[str, Any]

Apply Rubin's rules to combine estimates across imputations.

mi_test

mi_test(pooled: Dict[str, Any], terms: Union[str, Sequence[str]], *, method: str = 'equal_fmi', small: bool = True) -> Dict[str, Any]

Joint Wald test of pooled multiple-imputation coefficients.

Parameters:

Name	Type	Description	Default
pooled	dict	Output of :func:sp.mi_estimate (it carries the within / between covariances ubar_matrix / b_matrix and the complete-data df dfcom).	required
terms	str or list of str	Coefficient names tested jointly equal to zero (var_names).	required
method	('equal_fmi', 'unrestricted')	"equal_fmi" is Stata mi test's default test (equal fractions of missing information across the tested coefficients); "unrestricted" is mi test, ufmitest.	"equal_fmi"
small	bool	Small-sample denominator df, Stata's default: Reiter (2007) for the equal-FMI test and Barnard & Rubin (1999) / Marchenko & Reiter (2009) otherwise. Applied only when the pooled result carries a finite complete-data df (dfcom, e.g. the residual df of sp.regress); without one the large-sample df is used, as Stata does when e(df_r) is not set. small=False is mi test, nosmall.	True

Returns:

Type	Description
dict	F, df1, df2, pvalue, rvi (average relative increase in variance), method, terms, n_imputations, dfcom and df_adjustment ("small" or "large", the df actually used).

Warns:

Type	Description
AssumptionWarning	When Reiter's df is used outside its range (nu_c* <= 4 (1 + a), a tiny complete-data df relative to the missing information): the value is Stata's, but it can fall below 4 and should not be trusted. Use method="unrestricted" or more data.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(7)
>>> df = pd.DataFrame({"x1": rng.normal(size=200), "x2": rng.normal(size=200)})
>>> df["y"] = 1 + df["x1"] + 0.5 * df["x2"] + rng.normal(size=200)
>>> df.loc[rng.choice(200, 30, replace=False), "x2"] = np.nan
>>> pooled = sp.mi_estimate(sp.mice(df, m=5, seed=0), sp.regress,
...                         formula="y ~ x1 + x2")
>>> res = sp.mi_test(pooled, ["x1", "x2"])
>>> res["df1"], res["df_adjustment"]
(2, 'small')

References

[@reiter2007small], [@barnard1999small], [@marchenko2009improved]; Stata 18 [MI] manual, mi estimate methods and formulas.

mi_estimate

mi_estimate(mice_result: MICEResult, estimator: Any, **kwargs: Any) -> MIEstimateResult

Run an estimator on each imputed dataset and combine using Rubin's rules.

Parameters:

Name	Type	Description	Default
mice_result	MICEResult	Result from mice().	required
estimator	callable	Estimation function that returns an EconometricResults object.	required
**kwargs	Any	Arguments passed to the estimator.	{}

Returns:

Type

Description

MIEstimateResult

A dict of the combined estimates (Rubin's rules), keyed as in :meth:MICEResult.combine, plus var_names and n_obs; it also has summary() / tidy() and labelled params / std_errors ... so sp.regtable(result) takes it. The within-imputation covariance is the estimator's full var_cov when it exposes one (data_info['var_cov']), else the diagonal of squared standard errors. The complete-data df for Barnard-Rubin is the df of the fit's t reference distribution, as Stata mi estimate takes it from e(df_r): n - k for sp.regress, G - 1 with clustered standard errors, and infinity (Rubin's large-sample df) for likelihood fits that report z statistics (sp.logit, sp.poisson ...) or fits that record no df. This reproduces R mice::pool and Stata mi estimate (checked against Stata 18 for regress, regress, vce(cluster) and logit).

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(7)
>>> x1 = rng.normal(size=120)
>>> x2 = 0.5 * x1 + rng.normal(size=120)
>>> y = 1.0 + x1 + x2 + rng.normal(size=120)
>>> df = pd.DataFrame({"y": y, "x1": x1, "x2": x2})
>>> df.loc[rng.choice(120, size=15, replace=False), "x2"] = np.nan
>>> mice_res = sp.mice(df, m=3, seed=0)
>>> combined = sp.mi_estimate(mice_res, sp.regress, formula="y ~ x1 + x2")
>>> combined['n_imputations']
3
>>> combined['var_names']
['Intercept', 'x1', 'x2']