Skip to content

statspai.panel

panel

Unified panel regression module for StatsPAI.

Provides a single entry point panel() covering all panel estimators:

Static models — FE, RE, Between, First Difference, Pooled OLS, Two-way FE Correlated RE — Mundlak (1978), Chamberlain (1982) Dynamic panel — Arellano-Bond, Blundell-Bond (System GMM) HDFE absorption — high-dimensional fixed-effects OLS (Stata's reghdfe / R's fixest)

All results return PanelResults with built-in diagnostics:

import numpy as np, pandas as pd import statspai as sp rng = np.random.default_rng(0) ids = np.repeat(np.arange(50), 6) alpha_i = rng.normal(size=50)[ids] x1 = rng.normal(size=300) + 0.5 * alpha_i x2 = rng.normal(size=300) y = 1.0 + 0.5 * x1 - 0.3 * x2 + alpha_i + rng.normal(size=300) df = pd.DataFrame({"id": ids, "t": np.tile(np.arange(6), 50), ... "y": y, "x1": x1, "x2": x2}) result = sp.panel(df, "y ~ x1 + x2", entity='id', time='t') type(result).name 'PanelResults' result.params.round(1).to_dict() {'x1': 0.5, 'x2': -0.3} haus = result.hausman_test() # FE vs RE lm = result.bp_lm_test() # Pooled vs RE ftest = result.f_test_effects() # Joint significance of FE ftest["df1"], ftest["df2"] (49, 248) cmp = result.compare('re') # Side-by-side comparison

References

Wooldridge, J.M. (2010). Econometric Analysis of Cross Section and Panel Data. Mundlak, Y. (1978). "On the Pooling of Time Series and Cross Section Data." Chamberlain, G. (1982). "Multivariate Regression Models for Panel Data." Arellano, M. and Bond, S. (1991). "Some Tests of Specification for Panel Data." Blundell, R. and Bond, S. (1998). "Initial Conditions and Moment Restrictions." Hausman, J.A. (1978). "Specification Tests in Econometrics." Breusch, T.S. and Pagan, A.R. (1980). "The Lagrange Multiplier Test." Pesaran, M.H. (2004). "General Diagnostic Tests for Cross Section Dependence." Correia, S. (2017). "Linear Models with High-Dimensional Fixed Effects."

FEOLSResult dataclass

Bases: ResultProtocolMixin

Result of sp.feols().

Attributes:

Name Type Description
params Series

Coefficient estimates indexed by regressor name.

std_errors Series

Standard errors indexed by regressor name.

vcov ndarray

Variance-covariance matrix of the coefficients, in params order; cov_params() returns it labelled.

tvalues, pvalues Series
conf_int_lower, conf_int_upper Series
residuals ndarray

In-sample residuals (after FE absorption).

fitted_within ndarray

Predicted values from X β (excludes FE contribution).

n_obs int
n_singletons_dropped int
n_fe List[int]

Number of groups per absorbed FE dimension.

dof_fe int

Degrees of freedom consumed by the FEs.

df_resid int
r2_within float
r2, r2_a, r2_a_within float

R², adjusted R² and adjusted within R², as reghdfe's e(r2), e(r2_a), e(r2_a_within): the adjustment charges the absorbed fixed effects, including those nested in a cluster (df_a_nested).

rss, tss float

Residual and total (around the mean) sums of squares.

rmse float

sqrt(rss / (N - K - df_a - df_a_nested)) (e(rmse)).

df_a, df_a_nested int

reghdfe's e(df_a) (absorbed coefficients, excluding effects nested in a cluster) and e(df_a_nested).

df_inference float

Degrees of freedom of the t reference distribution of pvalues / conf_int_* (inf = normal). Clustered: min(G) - 1 capped at the residual df, as reghdfe's e(df_r).

se_type str

'iid' | 'cluster' | 'multiway_cluster' | 'wild_cluster'

cluster_info dict

Metadata (cluster names, counts).

formula str
absorber Absorber

Reusable absorber (includes keep_mask to subset rows).

Examples:

>>> import statspai as sp
>>> from statspai.panel.feols import feols
>>> df = sp.mincer_wage_panel()
>>> res = feols("log_wage ~ education + experience | period",
...             data=df, cluster="period")
>>> type(res).__name__
'FEOLSResult'
>>> res.se_type
'cluster'
>>> bool(res.params["education"] > 0)
True

data_info property

data_info: Dict[str, Any]

Sample sizes and the reference-distribution df (postestimation).

nobs property

nobs: int

Estimation-sample size (e(N)); alias of n_obs.

r2_adj property

r2_adj: float

Adjusted R-squared (e(r2_a)); alias of r2_a.

cov_params

cov_params() -> DataFrame

Coefficient covariance labelled by regressor (e(V)).

vcov stays the bare array other code indexes positionally; this is the same matrix with the regressor names on both axes.

Absorber

Reusable HDFE demean operator.

Build once from a DataFrame's FE columns; reuse demean to sweep any outcome / regressor vector or matrix. Useful when fitting many models that share the same absorbing FEs (e.g. event-study coefficient paths).

Parameters:

Name Type Description Default
fe_data DataFrame, ndarray (n, K), or None

FE columns. Must have no NaN. May be None (or an (n, 0) array) when only varying-slope terms are absorbed, in which case n_obs is inferred from slopes.

required
weights ndarray(n)

Observation weights. If given, weighted means are used.

None
drop_singletons bool

If True, singleton observations (FE groups of size 1) are pruned before building the absorber. keep_mask stores the surviving rows.

True
tol float

Convergence threshold on max |dx| per iteration.

1e-8
maxiter int

Maximum alternating-projection iterations.

10000
accelerate bool

Enable Irons-Tuck Δ² acceleration.

True
solver ('map', 'lsmr', 'lsqr')

Within-transformation backend. "map" uses alternating projections with Irons-Tuck acceleration (default, typically fastest on well-conditioned panels). "lsmr" / "lsqr" delegate to scipy.sparse.linalg.lsmr / lsqr on the sparse FE design matrix — more robust for ill-conditioned or highly nested FE structures. See the migration guide for how this maps to pyreghdfe.

"map"
slopes sequence of SlopeSpec

Varying-slope terms to absorb alongside fe_data. Each counts as one more absorbed dimension in the alternating projections.

None
n_obs int

Row count, required only when fe_data is None and slopes is empty.

None

Attributes:

Name Type Description
keep_mask ndarray of bool

Rows retained after singleton pruning. Callers must apply this mask to y, X, and any weights before passing to demean.

n_kept int

Number of surviving observations.

n_dropped int

Number of singleton observations removed.

n_fe list of int

Number of groups per FE dimension (post-prune).

slope_ops list

Pre-conditioned varying-slope projectors (post-prune). Each exposes n_levels and n_degenerate (levels whose within-level design is rank-deficient and absorbs nothing).

n_slope_levels list of int

Number of levels per varying-slope term (post-prune).

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> n = 300
>>> fe_data = pd.DataFrame({
...     "firm": rng.integers(0, 20, n),
...     "year": rng.integers(0, 5, n),
... })
>>> y = (fe_data["firm"] * 0.5 + fe_data["year"] * 0.3
...      + rng.normal(0, 1, n))
>>> absorber = sp.Absorber(fe_data)
>>> y_within = absorber.demean(y.to_numpy())  # keep_mask applied
>>> absorber.n_fe
[20, 5]
>>> bool(abs(y_within.mean()) < 1e-6)
True

demean

demean(x: ndarray, copy: bool = True, already_masked: bool = False) -> ndarray

Within-transform x by sweeping out all absorbed FEs.

Parameters:

Name Type Description Default
x (ndarray, shape(n) or (n, p))

Variable(s) to residualize. n must equal either the full input size (then keep_mask is applied) or n_kept (when already_masked=True).

required
copy bool

If True, operate on a copy; if False, modify x in place. Callers passing fresh arrays can set False to save memory.

True
already_masked bool

Skip application of keep_mask.

False

Returns:

Type Description
ndarray

Residualized x with shape (n_kept,) or (n_kept, p).

residualize

residualize(x: ndarray, copy: bool = True) -> ndarray

Alias for demean — returns FE-residualized version of x.

SlopeSpec

Bases: NamedTuple

A varying-slope absorbed term.

Represents Stata's absorb(i.g#c.x) / absorb(i.g##c.x) and fixest's | g[[x]] / | g[x].

Attributes:

Name Type Description
group ndarray(n)

Raw group labels (numeric or object). Factorized internally.

x ndarray(n)

Continuous variable whose coefficient varies across levels of group.

with_intercept bool

False → absorb only the G slope columns x · 1[g = j] (Stata #, fixest [[x]]). True → additionally absorb the G level intercepts 1[g = j] (Stata ##, fixest [x]).

name str

Display name, used in diagnostics and warnings.

Examples:

>>> import numpy as np
>>> import statspai as sp
>>> rng = np.random.default_rng(0)
>>> g = np.repeat(np.arange(4), 25)
>>> x = rng.normal(size=100)
>>> spec = sp.SlopeSpec(group=g, x=x, with_intercept=False, name="i.g#c.x")
>>> spec.with_intercept
False
>>> spec.name
'i.g#c.x'

PanelCompareResults

Bases: ResultProtocolMixin

Side-by-side comparison of two panel models.

Produced by :meth:PanelResults.compare. Holds the two fitted models and renders a shared coefficient/SE table plus per-model diagnostics via :meth:summary, or a side-by-side coefficient plot via :meth:plot.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(40):
...     alpha = rng.normal()
...     for t in range(5):
...         x1, x2 = rng.normal(), rng.normal()
...         y = 1.0 + 0.5 * x1 - 0.3 * x2 + alpha + rng.normal(0, 0.5)
...         rows.append({"id": i, "year": t, "y": y, "x1": x1, "x2": x2})
>>> df = pd.DataFrame(rows)
>>> r = sp.panel(df, "y ~ x1 + x2", entity="id", time="year", method="fe")
>>> cmp = r.compare("re")                      # FE vs RE side by side
>>> isinstance(cmp, sp.PanelCompareResults)
True
>>> bool("Panel Comparison" in cmp.summary())
True

plot

plot(variables: Optional[List[str]] = None, **kwargs: Any) -> Any

Side-by-side coefficient comparison plot.

Returns:

Type Description
(fig, ax)

PanelRegression

Deprecated: use panel() directly. Kept for backward compatibility.

Construct with the same keyword arguments as :func:panel, then call :meth:fit; the result is an ordinary :class:PanelResults. New code should call sp.panel(...) instead.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(40):
...     alpha = rng.normal()
...     for t in range(5):
...         x1, x2 = rng.normal(), rng.normal()
...         y = 1.0 + 0.5 * x1 - 0.3 * x2 + alpha + rng.normal(0, 0.5)
...         rows.append({"id": i, "year": t, "y": y, "x1": x1, "x2": x2})
>>> df = pd.DataFrame(rows)
>>> model = sp.PanelRegression(data=df, formula="y ~ x1 + x2",
...                            entity="id", time="year", method="fe")
>>> res = model.fit()
>>> isinstance(res, sp.PanelResults)
True

PanelResults

Bases: EconometricResults

Panel regression results with built-in diagnostics.

Extends EconometricResults with panel-specific tests that can be called directly on the result object (see Examples for a runnable setup)::

result = sp.panel(df, "y ~ x1 + x2", entity='id', time='t')
result.hausman_test()        # FE vs RE
result.bp_lm_test()          # Pooled vs RE (Breusch-Pagan LM)
result.f_test_effects()      # Joint significance of entity FE
result.compare('re')         # Compare with RE side by side

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(40):
...     alpha = rng.normal()
...     for t in range(5):
...         x1, x2 = rng.normal(), rng.normal()
...         y = 1.0 + 0.5 * x1 - 0.3 * x2 + alpha + rng.normal(0, 0.5)
...         rows.append({"id": i, "year": t, "y": y, "x1": x1, "x2": x2})
>>> df = pd.DataFrame(rows)
>>> r = sp.panel(df, "y ~ x1 + x2", entity="id", time="year", method="fe")
>>> isinstance(r, sp.PanelResults)
True
>>> sorted(r.params.index)
['x1', 'x2']
>>> h = r.hausman_test()                       # FE vs RE specification test
>>> set(["statistic", "df", "pvalue"]).issubset(h)
True

hausman_test

hausman_test(alpha: float = 0.05, sigmamore: bool = False) -> Dict[str, Any]

Hausman (1978) specification test: FE vs RE.

Under H0 (RE consistent), both FE and RE are consistent but RE is efficient. Under H1, only FE is consistent.

Parameters:

Name Type Description Default
alpha float
0.05
sigmamore bool

Stata's hausman fe re, sigmamore: base both covariance matrices on the RE disturbance variance. Use it when the classical statistic is negative (recommendation == "inconclusive").

False

Returns:

Type Description
dict

'statistic', 'df', 'pvalue', 'recommendation' ('FE', 'RE' or 'inconclusive'), 'psd_violation', 'sigmamore', 'interpretation'

bp_lm_test

bp_lm_test() -> Dict[str, Any]

Breusch-Pagan (1980) Lagrange Multiplier test for random effects.

Tests H0: Var(alpha_i) = 0 (Pooled OLS is appropriate) vs H1: Var(alpha_i) > 0 (Random Effects needed).

Returns:

Type Description
dict

'statistic', 'df', 'pvalue', 'recommendation', 'interpretation'

f_test_effects

f_test_effects() -> Dict[str, Any]

F-test for joint significance of entity fixed effects.

Tests H0: all alpha_i = 0 (entity effects not needed).

Returns:

Type Description
dict

'statistic', 'df1', 'df2', 'pvalue', 'interpretation'

pesaran_cd_test

pesaran_cd_test() -> Dict[str, Any]

Pesaran (2004) CD test for cross-sectional dependence in residuals.

Returns:

Type Description
dict

'statistic', 'pvalue', 'interpretation'

plot

plot(type: str = 'coef', **kwargs: Any) -> Any

Generate panel-specific plots.

Parameters:

Name Type Description Default
type str

'coef' — Coefficient forest plot (default) 'effects' — Distribution of entity fixed effects 'residuals' — Residual diagnostics (2x2 grid) 'hausman' — Visual FE vs RE comparison

'coef'
**kwargs Any

Passed to the underlying plot function.

{}

Returns:

Type Description
(fig, ax)

plot_effects

plot_effects(**kwargs: Any) -> Any

Shortcut for .plot(type='effects'). Distribution of entity FE.

plot_residuals

plot_residuals(**kwargs: Any) -> Any

Shortcut for .plot(type='residuals'). Residual diagnostics (2x2).

plot_hausman

plot_hausman(**kwargs: Any) -> Any

Shortcut for .plot(type='hausman'). Visual FE vs RE comparison.

compare

compare(method: str, **kwargs: Any) -> PanelCompareResults

Re-estimate with a different method and compare side by side.

Parameters:

Name Type Description Default
method str

Alternative method to compare against.

required

Returns:

Type Description
PanelCompareResults

Side-by-side comparison with diagnostics.

hdfe_feols

hdfe_feols(formula: str, data: DataFrame, *, weights: Optional[Union[str, ndarray]] = None, cluster: Optional[Union[str, List[str]]] = None, se_type: Optional[str] = None, vce: Optional[str] = None, wild: bool = False, wild_n_boot: int = 999, wild_weight_type: str = 'webb', wild_seed: Optional[int] = None, conley_lat: Optional[str] = None, conley_lon: Optional[str] = None, conley_cutoff: Optional[float] = None, conley_time: Optional[str] = None, conley_unit: Optional[str] = None, conley_lag: Optional[int] = None, conley_lag_cross: int = 0, conley_kernel: str = 'uniform', conley_time_kernel: str = 'bartlett', alpha: float = 0.05, drop_singletons: bool = True, tol: Optional[float] = None, maxiter: int = 10000, df_inference: Optional[Union[str, float]] = None, absorb_dof: str = 'reghdfe', weights_as: Optional[str] = None, cluster_df: Optional[str] = None) -> FEOLSResult

reghdfe-style OLS with high-dimensional fixed effects.

Parameters:

Name Type Description Default
formula str

"y ~ x1 + x2 | fe1 + fe2 + fe3". The | fe... part is optional. Both sides accept bare names, c.x / i.f, a:b, a*b, f1^f2 and the varying-slope forms i.f#c.x / i.f##c.x / f[[x]] / f[x] — see the module docstring for the full grammar. Any Python identifier is a column name, so non-ASCII names (工资 ~ 暴露度) work.

A third part makes it 2SLS, Stata's ivreghdfe: "y ~ exog | fe1 + fe2 | endog ~ z1 + z2" ("y ~ 1 | ..." without exogenous regressors). All variables are swept of the fixed effects by the same absorber, then 2SLS runs on the residuals; cluster= (one way), vce='robust' or iid SEs follow ivreg2's small-sample conventions, and result.iv_diagnostics carries the first-stage F tests, Kleibergen-Paap rk LM and Wald F, Cragg-Donald F, Anderson-Rubin test and Hansen J (all as ivreghdfe ..., first).

required
data DataFrame
required
weights str or ndarray

Observation weights. Column name or raw array. Alone they are analytic weights (Stata [aw=]).

None
weights_as ('analytic', 'sampling')

What the weights are. 'analytic' is what weights= alone already means, and saying so silences the warning about dispersed weights under the classical variance. 'sampling' (Stata [pw=]) needs cluster=: this function has no heteroskedasticity-robust variance for a weighted fit, and refuses rather than report the classical one (sp.feols(..., weights_as='sampling') has it). 'frequency' is refused.

'analytic'
cluster str or list

One-way or multi-way cluster column(s).

None
se_type str

A label, not a switch: it never changes the variance, which is chosen by cluster= / wild= / vce=. When given it must agree with what those options select, and a disagreeing value is refused: 'iid' (none of them), 'cluster' (one cluster column, also with vce='CR2' / 'CR3' / 'jackknife'), 'multiway_cluster' (several), 'wild_cluster' (wild=True / vce='wild'), 'hc_robust' (vce='robust' / 'hc1' / 'hc0'), 'conley' (vce='conley'); the spelling passed to vce= is accepted too. Leave it out and set the variance with vce= / cluster=; result.se_type reports what was computed.

None
vce str

Canonical SE-menu keyword (matches sp.regress / sp.feols):

  • "robust" / "hc1" — heteroskedasticity-robust on the FE-absorbed design with reghdfe's small-sample factor N/(N-k-df_a); matches Stata reghdfe ..., vce(robust).
  • "hc0" — no small-sample factor.
  • "CR2" / "CR3" / "jackknife" — Pustejovsky-Tipton (2018) bias-reduced cluster-robust on the within design (requires cluster=, one-way); matches R clubSandwich::vcovCR(plm).
  • "conley" — Conley spatial HAC on the within design (requires conley_lat=/conley_lon=/conley_cutoff=; Stata acreg planar distance convention).
  • "wild" — shorthand for wild=True (requires cluster=).
None
wild bool

If True (and cluster is given), return wild-cluster-bootstrap p-values / CIs alongside classical cluster SE. Applied variable- by-variable. Only supported with a single cluster column.

This is boottest's WCR bootstrap with the null imposed and the fixed effects re-absorbed in every draw (exactly, through 2G sweeps of cluster-indicator columns done once). Running the bootstrap on already-demeaned data -- the quick "FWL" shortcut -- is valid only when every absorbed effect is nested in the cluster; otherwise its bootstrap t distribution is too narrow (see panel/_wild_fe.py; on 1.5 million rows with four FE groups and 100 occupation clusters the shortcut gave p = 0.116 against 0.176). The cost is dominated by those 2G sweeps plus the confidence-set inversion, not by the draws themselves (on a 230,000-row subsample 99, 999 and 9,999 draws took 31, 29 and 35 seconds).

False
wild_n_boot int

Bootstrap replications.

999
wild_weight_type ('rademacher', 'webb', 'mammen')
'rademacher'
wild_seed int
None
conley_lat str

Coordinate columns (decimal degrees) for vce="conley".

None
conley_lon str

Coordinate columns (decimal degrees) for vce="conley".

None
conley_cutoff float

Conley distance cutoff in km for vce="conley".

None
conley_time optional

Panel Conley: spatial + time HAC as Stata acreg ..., id() time() lag() with pfe1() pfe2(). Pairs within a unit take the time kernel alone up to conley_lag periods; pairs of different units take the spatial kernel times the time kernel up to conley_lag_cross periods (acreg's lagdist(), default 0: contemporaneous only). All three must be given together. Without them the spatial kernel runs over every pair of rows, which on a panel treats a unit's own rows in different periods -- and every cross-period pair of neighbours -- as correlated; that warns.

None
conley_unit optional

Panel Conley: spatial + time HAC as Stata acreg ..., id() time() lag() with pfe1() pfe2(). Pairs within a unit take the time kernel alone up to conley_lag periods; pairs of different units take the spatial kernel times the time kernel up to conley_lag_cross periods (acreg's lagdist(), default 0: contemporaneous only). All three must be given together. Without them the spatial kernel runs over every pair of rows, which on a panel treats a unit's own rows in different periods -- and every cross-period pair of neighbours -- as correlated; that warns.

None
conley_lag optional

Panel Conley: spatial + time HAC as Stata acreg ..., id() time() lag() with pfe1() pfe2(). Pairs within a unit take the time kernel alone up to conley_lag periods; pairs of different units take the spatial kernel times the time kernel up to conley_lag_cross periods (acreg's lagdist(), default 0: contemporaneous only). All three must be given together. Without them the spatial kernel runs over every pair of rows, which on a panel treats a unit's own rows in different periods -- and every cross-period pair of neighbours -- as correlated; that warns.

None
conley_lag_cross int

Time bandwidth for pairs of different units (lagdist()).

0
conley_kernel ('uniform', 'bartlett')

Spatial kernel (acreg's bartlett option selects Bartlett).

'uniform'
conley_time_kernel ('bartlett', 'uniform')

Time kernel of the panel variant (acreg's hac is Bartlett, 1 - |dt| / (lag + 1); without hac it is uniform).

'bartlett'
alpha float
0.05
drop_singletons bool
True
tol float

Convergence tolerance of the absorber (largest change of a sweep, relative to the variable's scale). Default 1e-8, or 1e-12 when a varying slope (i.f#c.x / f[x]) is absorbed: slope sweeps converge slowly, and at 1e-8 the SEs of a four-FE model with a quarter-specific slope were still 1e-5 away from reghdfe's (which converges at its own 1e-8), against 4e-8 at 1e-12. reghdfe's collinearity screen is derived from 1e-8 either way.

None
maxiter int

Iteration cap of the absorber.

10000
df_inference (None, 'resid', 'normal')

Reference distribution of the p-values and confidence intervals. None (default) follows reghdfe's e(df_r): t(G - 1) under clustering (G = the smallest cluster count, capped at the residual df), t(N - K - df_a) otherwise. 'resid' uses the residual df also under clustering (StatsPAI <= 1.32, close to the normal with many observations); 'normal' the standard normal; a number that many degrees of freedom. The standard errors are the same in every case.

None
absorb_dof ('reghdfe', 'areg')

What the absorbed effects cost a clustered variance. 'reghdfe' does not charge an effect nested in the cluster variable. 'areg' charges every absorbed level, as the dummy-variable regression and Stata's areg do; with drop_singletons=False the slopes and standard errors are then those of sp.regress('y ~ x + C(g)') without building the dummies. OLS with one-way clustering only.

'reghdfe'
cluster_df ('min', 'per_term')

Small-sample factor of a multi-way cluster variance (the inclusion-exclusion sum V_a + V_b - V_ab). None (the default) is 'min': every term is scaled by G_min/(G_min - 1), the smallest cluster count, which is what Stata reghdfe and R fixest (and so sp.feols(vcov={'CRV1': 'a + b'})) report. 'per_term' scales each term by its own G/(G - 1), as R sandwich::vcovCL does (the default before 1.41). The two differ by a few percent with few clusters in one dimension; one-way clustering is the same either way. The choice is recorded in cluster_info['cluster_df']. Explicit values apply to OLS with fixed effects only; without absorbed effects or with instruments the multi-way factor is per term.

.. versionchanged:: 1.41 The default for multi-way clustering is 'min' (was 'per_term'; see MIGRATION.md).

'min'

Returns:

Type Description
FEOLSResult

Examples:

Two-way fixed effects (firm and year) with cluster-robust SE. This function is exported at top level as :func:statspai.hdfe_ols.

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> firm_fe = rng.normal(0, 1, 40)
>>> year_fe = rng.normal(0, 0.5, 6)
>>> rows = []
>>> for i in range(40):
...     for t in range(6):
...         educ = rng.normal(12, 2)
...         exper = rng.normal(10, 3)
...         lwage = (1.0 + 0.08 * educ + 0.02 * exper
...                  + firm_fe[i] + year_fe[t] + rng.normal(0, 0.3))
...         rows.append((i, t, lwage, educ, exper))
>>> df = pd.DataFrame(rows, columns=['firm', 'year', 'lwage',
...                                  'educ', 'exper'])
>>> res = sp.hdfe_ols("lwage ~ educ + exper | firm + year", data=df,
...                   cluster='firm')
>>> sorted(res.coef.index.tolist())
['educ', 'exper']

absorb_ols

absorb_ols(y: ndarray, X: ndarray, fe: Union[DataFrame, ndarray, None], weights: Optional[ndarray] = None, cluster: Optional[Union[ndarray, List[ndarray]]] = None, drop_singletons: bool = True, tol: float = 1e-08, maxiter: int = 10000, return_absorber: bool = False, solver: str = 'map', slopes: Optional[Sequence[SlopeSpec]] = None, cluster_df: Optional[str] = None, collinear_tol: Optional[float] = None, absorb_dof: str = 'reghdfe') -> dict

OLS with absorbed high-dimensional fixed effects (reghdfe-style).

Solves y = X β + Σ_k α_{g_k} + ε by sweeping out the FEs from both y and X (Frisch-Waugh-Lovell) and running OLS on residuals.

Parameters:

Name Type Description Default
y (ndarray, shape(n))
required
X (ndarray, shape(n, p))

Regressors excluding the absorbed FEs and the constant (the constant is absorbed by any FE dimension).

required
fe DataFrame or ndarray(n, K)

Fixed-effect columns.

required
weights ndarray(n)

Observation weights.

None
cluster ndarray or list of ndarrays

One-way or multi-way cluster variables for robust SEs. If provided, returns cluster-robust SEs (one-way: Liang-Zeger sandwich; multi-way: inclusion-exclusion Cameron-Gelbach-Miller).

None
drop_singletons bool
True
tol (float, int)

Demean convergence controls.

1e-08
maxiter (float, int)

Demean convergence controls.

1e-08
return_absorber bool

If True, also return the Absorber object for reuse.

False
solver ('map', 'lsmr', 'lsqr')

Within-transformation backend. See :class:Absorber.

"map"
collinear_tol float

The tol that reghdfe's collinearity screen is derived from (min(1e-6, collinear_tol / 10)); defaults to tol. Lets a caller tighten the sweep without moving the collinearity threshold.

None
absorb_dof ('reghdfe', 'areg')

Degrees of freedom the absorbed effects cost a clustered variance. "reghdfe" does not charge an effect nested in the cluster variable (firm effects under firm clustering); "areg" charges every absorbed level, which is the dummy-variable regression and Stata's areg. The two agree without clustering and whenever no effect is nested in a cluster.

"reghdfe"
cluster_df ('per_term', 'min')

Small-sample factor of the multi-way (inclusion-exclusion) cluster variance. "per_term" scales each term by its own G_S/(G_S − 1) (R sandwich::vcovCL, :func:multiway_cluster_vcov); "min" scales every term by G_min/(G_min − 1) with G_min the smallest one-way cluster count -- Stata reghdfe's convention (and fixest's cluster.df = "min"). Identical for one-way clustering. None (the default) is "per_term" for now; with multi-way clustering it emits a FutureWarning, because the default becomes "min" in 1.42 (sp.hdfe_ols made that switch in 1.41). Pass the value explicitly to keep a result and silence the warning.

.. deprecated:: 1.41 The implicit "per_term" default for multi-way clustering.

"per_term"

Returns:

Type Description
dict with keys:

coef (p,), se (p,), vcov (p,p), resid (n_kept,), n (n_kept), df_resid, dof_fe, r2_within, rss / tss_within (weighted when weights is given), n_singletons_dropped, converged, iters, absorber (if requested)

Examples:

Two-way (firm + year) fixed effects with firm-clustered SEs:

>>> import numpy as np
>>> import pandas as pd
>>> import statspai as sp
>>> rng = np.random.default_rng(0)
>>> n = 500
>>> firm = rng.integers(0, 30, n)
>>> year = rng.integers(0, 10, n)
>>> X = rng.normal(size=(n, 2))
>>> y = (X @ np.array([1.5, -0.7]) + rng.normal(size=30)[firm]
...      + rng.normal(size=10)[year] + rng.normal(0, 0.5, n))
>>> fe = pd.DataFrame({"firm": firm, "year": year})
>>> out = sp.absorb_ols(y, X, fe, cluster=firm)
>>> print(np.round(out["coef"], 2))   # truth [1.5, -0.7]
[ 1.53 -0.71]
>>> print(out["n"], out["converged"])
500 True

demean

demean(x: ndarray, fe: Union[DataFrame, ndarray, None], weights: Optional[ndarray] = None, drop_singletons: bool = True, tol: float = 1e-08, maxiter: int = 10000, solver: str = 'map', slopes: Optional[Sequence[SlopeSpec]] = None) -> Tuple[ndarray, ndarray]

Return the within-transformed x and the singleton keep mask.

Convenience wrapper around :class:Absorber. See Absorber for the solver kwarg semantics.

Examples:

Sweep firm and year fixed effects out of a column:

>>> import numpy as np
>>> import pandas as pd
>>> import statspai as sp
>>> rng = np.random.default_rng(0)
>>> n = 200
>>> fe = pd.DataFrame({"firm": rng.integers(0, 20, n),
...                    "year": rng.integers(0, 5, n)})
>>> x = rng.normal(size=n) + 2.0 * fe["firm"].to_numpy()
>>> xw, keep = sp.demean(x, fe)
>>> print(xw.shape, int(keep.sum()))  # no singletons dropped
(200,) 200
>>> print(abs(xw.mean()) < 1e-8)      # FE means swept out
True

panel_logit

panel_logit(data: DataFrame, y: str, x: List[str], id: str = 'id', time: str = 'time', method: str = 'fe', n_quadrature: int = 12, robust: str = 'nonrobust', cluster: Optional[str] = None, maxiter: int = 200, tol: float = 1e-08, alpha: float = 0.05) -> EconometricResults

Panel logit model.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
y str

Binary dependent variable (0/1).

required
x list of str

Regressors.

required
id str

Unit and time identifier columns.

'id'
time str

Unit and time identifier columns.

'id'
method str

'fe' (conditional FE logit), 're' (random effects), 'cre' (Mundlak).

'fe'
n_quadrature int

Gauss-Hermite quadrature points (RE/CRE only).

12
robust str

'nonrobust' or 'robust'.

'nonrobust'
cluster str or None

Column for cluster-robust SEs.

None
maxiter int

Maximum optimizer iterations.

200
tol float

Gradient tolerance.

1e-08
alpha float

Significance level for confidence intervals.

0.05

Returns:

Type Description
EconometricResults

Examples:

>>> import statspai as sp
>>> import numpy as np
>>> import pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(60):
...     a = rng.normal()  # unit effect
...     for t in range(5):
...         xit = rng.normal()
...         p = 1.0 / (1.0 + np.exp(-(0.8 * xit + a)))
...         rows.append({"id": i, "time": t,
...                      "y": int(rng.uniform() < p), "x": xit})
>>> df = pd.DataFrame(rows)
>>> res = sp.panel_logit(df, y="y", x=["x"], id="id", time="time", method="fe")
>>> bool("x" in res.params.index)
True

panel_probit

panel_probit(data: DataFrame, y: str, x: List[str], id: str = 'id', time: str = 'time', method: str = 're', n_quadrature: int = 12, robust: str = 'nonrobust', cluster: Optional[str] = None, maxiter: int = 200, tol: float = 1e-08, alpha: float = 0.05) -> EconometricResults

Panel probit model.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
y str

Binary dependent variable (0/1).

required
x list of str

Regressors.

required
id str

Unit and time identifier columns.

'id'
time str

Unit and time identifier columns.

'id'
method str

're' (random effects) or 'cre' (Mundlak). FE probit not supported (incidental parameters problem).

're'
n_quadrature int

Gauss-Hermite quadrature points.

12
robust str

'nonrobust' or 'robust'.

'nonrobust'
cluster str or None

Column for cluster-robust SEs.

None
maxiter int

Maximum optimizer iterations.

200
tol float

Gradient tolerance.

1e-08
alpha float

Significance level for confidence intervals.

0.05

Returns:

Type Description
EconometricResults

Examples:

>>> import statspai as sp
>>> import numpy as np
>>> import pandas as pd
>>> from scipy.stats import norm
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(50):
...     a = rng.normal()  # unit effect
...     for t in range(4):
...         xit = rng.normal()
...         p = norm.cdf(0.7 * xit + a)
...         rows.append({"id": i, "time": t,
...                      "y": int(rng.uniform() < p), "x": xit})
>>> df = pd.DataFrame(rows)
>>> res = sp.panel_probit(df, y="y", x=["x"], id="id", time="time",
...                       method="re", n_quadrature=8)
>>> bool("x" in res.params.index)
True

plot_within_between

plot_within_between(data: DataFrame, variables: List[str], entity: str, ax: Any = None, figsize: Tuple[float, float] = (8, 6), color: str = '#2C3E50', title: Optional[str] = None) -> Tuple[Any, Any]

Bar chart comparing within vs between variation for each variable.

Helps assess which estimator is appropriate: - High between / low within → BE may be efficient - High within / low between → FE captures the action - Similar → both capture similar information

Parameters:

Name Type Description Default
data DataFrame
required
variables list of str

Variables to decompose.

required
entity str

Entity identifier column.

required
ax matplotlib Axes
None
figsize tuple
(8, 6)
color str
'#2C3E50'
title str
None

Returns:

Type Description
(fig, ax)

balance_panel

balance_panel(data: DataFrame, entity: str, time: str) -> DataFrame

Balance a panel by keeping only units observed in every time period.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
entity str

Entity (unit) identifier column.

required
time str

Time period column.

required

Returns:

Type Description
DataFrame

Balanced panel (same column order, sorted by entity then time).

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(5):
...     for t in range(4):
...         if i == 3 and t == 2:
...             continue  # drop one obs -> unbalanced
...         rows.append({"id": i, "year": t, "y": rng.normal()})
>>> df = pd.DataFrame(rows)
>>> df.groupby("id")["year"].count().tolist()
[4, 4, 4, 3, 4]
>>> balanced = sp.balance_panel(df, entity="id", time="year")
>>> int(balanced.groupby("id")["year"].count().nunique())  # all same count
1
>>> balanced["id"].nunique()  # the short unit is dropped
4

panel_compare

panel_compare(data: DataFrame, formula: str, entity: str, time: str, methods: Optional[List[str]] = None, robust: str = 'nonrobust', cluster: Optional[str] = None, **kwargs: Any) -> DataFrame

Estimate the same model with multiple methods and compare.

Parameters:

Name Type Description Default
data DataFrame
required
formula str
required
entity str
required
time str
required
methods list of str

Methods to compare. Default: ['pooled', 'fe', 're', 'twoway', 'mundlak'].

None
robust str

Passed to each panel() call.

'nonrobust'
cluster str

Passed to each panel() call.

'nonrobust'

Returns:

Type Description
DataFrame

Comparison table with coefficients, SEs, and diagnostics.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for i in range(40):
...     alpha = rng.normal()
...     for t in range(5):
...         edu, exp = rng.normal(), rng.normal()
...         wage = 1.0 + 0.5 * edu - 0.3 * exp + alpha + rng.normal(0, 0.5)
...         rows.append({"id": i, "year": t, "wage": wage,
...                      "edu": edu, "exp": exp})
>>> df = pd.DataFrame(rows)
>>> comparison = sp.panel_compare(
...     df, "wage ~ edu + exp", entity="id", time="year",
...     methods=["pooled", "fe", "re"],
... )
>>> isinstance(comparison, pd.DataFrame)
True
>>> bool("edu" in comparison.index)  # one row per coefficient
True

panel

panel(data: Any = None, formula: Optional[str] = None, entity: Optional[str] = None, time: Optional[str] = None, *, method: str = 'fe', **kwargs: Any) -> Union[PanelResults, FEOLSResult]

Unified panel-regression dispatcher.

Parameters:

Name Type Description Default
data DataFrame
None
formula str

Patsy-style outcome ~ regressors specification.

None
entity str

Cross-section identifier column.

None
time str

Time identifier column.

None
method str

Estimator family:

  • Static: fe / fixed, re / random, be / between, fd / first_difference, pooled / pooled_ols, twoway / two_way.
  • Correlated random effects: mundlak, chamberlain.
  • Dynamic GMM: ab / arellano_bond / gmm, system / blundell_bond / system_gmm.
  • HDFE absorption: hdfe / feols / reghdfe (high-dimensional fixed-effects OLS).
``'fe'``
**kwargs Any

Forwarded to the chosen estimator. Classical methods accept robust / cluster / weights / alpha / balance / lags / gmm_lags, plus (for method='fe') the extended SE menu vce='CR2'/'CR3'/'jackknife'/'conley'/'wild', two-way cluster=['a', 'b'], conley_lat/lon/cutoff and wild_reps / seed. HDFE accepts cluster / vce / se_type / wild / alpha etc.

{}

Returns:

Type Description
Result object whose type depends on ``method``.

Examples:

>>> import statspai as sp
>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> n_id, n_t = 80, 8
>>> ids = np.repeat(np.arange(n_id), n_t)
>>> years = np.tile(np.arange(2000, 2000 + n_t), n_id)
>>> exp = rng.normal(10, 3, n_id * n_t)
>>> edu = rng.normal(12, 2, n_id * n_t)
>>> alpha_i = np.repeat(rng.normal(0, 1, n_id), n_t)
>>> wage = 1.0 + 0.05 * exp + 0.08 * edu + alpha_i + rng.normal(0, 0.5, n_id * n_t)
>>> df = pd.DataFrame({'id': ids, 'year': years, 'wage': wage,
...                    'exp': exp, 'edu': edu})
>>> # Default: within (FE) estimator
>>> r = sp.panel(df, "wage ~ exp + edu", entity='id', time='year')
>>> # Random effects
>>> r = sp.panel(df, "wage ~ exp + edu", entity='id', time='year',
...              method='re')
>>> # Friendly alias (case insensitive)
>>> r = sp.panel(df, "wage ~ exp", entity='id', time='year',
...              method='Fixed')
>>> # Arellano-Bond dynamic panel
>>> df['wage_lag'] = df.groupby('id')['wage'].shift(1)
>>> r = sp.panel(df, "wage ~ wage_lag + edu", entity='id', time='year',
...              method='gmm', lags=1)
>>> # HDFE absorption (multiple FEs in formula)
>>> r = sp.panel(df, "wage ~ exp | id + year", method='hdfe',
...              cluster='id')

xtoverid

xtoverid(data: DataFrame, y: str, x: Sequence[str], *, id: str, cluster: Optional[str] = None, alpha: float = 0.05) -> Dict[str, Any]

Robust test of random effects against fixed effects.

Random effects assumes the regressors are uncorrelated with the unit effect; those are over-identifying restrictions that fixed effects does not impose. The random-effects equation is re-estimated with the regressors in deviation-from-panel-mean form added, and the test is the Wald statistic that the added terms are zero, with a cluster-robust variance. Unlike the classical Hausman test it remains valid under heteroskedasticity and serial correlation, and it is never negative.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
y str

Outcome.

required
x sequence of str

Regressors. Time-invariant ones are kept in the model but cannot be tested (their deviation is zero) and do not count as restrictions.

required
id str

Panel identifier.

required
cluster str

Cluster variable. Default: the panel identifier.

None
alpha float

Level used in the interpretation.

0.05

Returns:

Type Description
dict

statistic (chi-squared), df, pvalue, tested (the regressors that vary within panels) and interpretation.

Notes

The cluster-robust variance carries the factor G/(G-1) * (N-1)/(N-K) and the statistic is referred to the chi-squared distribution, which reproduces the Stata command xtoverid after xtreg, re vce(cluster). The same hypothesis is tested by the Mundlak regression, sp.panel(method='mundlak').

Examples:

>>> import statspai as sp
>>> df = sp.dgp_panel(n_units=80, n_periods=6, seed=1)
>>> out = sp.xtoverid(df, "y", ["x"], id="unit")
>>> int(out["df"])
1
References

arellano1993testing; wooldridge2010econometric

xtserial

xtserial(data: DataFrame, y: str, x: Sequence[str], *, id: str, time: str, alpha: float = 0.05) -> Dict[str, Any]

Wooldridge test for first-order serial correlation in panel errors.

The model is estimated in first differences, which removes the unit effect. If the level errors are serially uncorrelated, the differenced errors have first-order autocorrelation -0.5; the test regresses the differenced residuals on their lag and tests that coefficient against -0.5 with a panel-clustered variance.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
y str

Outcome.

required
x sequence of str

Regressors (time-varying; a time-invariant one differences to zero).

required
id str

Panel and time identifiers. time must count periods (integers): a difference is taken only between consecutive periods.

required
time str

Panel and time identifiers. time must count periods (integers): a difference is taken only between consecutive periods.

required
alpha float

Level used in the interpretation.

0.05

Returns:

Type Description
dict

statistic (F), df (1, G - 1), pvalue, rho (the coefficient of the residual on its lag), n_obs and params / std_errors of the first-difference regression.

Notes

Rejection means the usual fixed- or random-effects standard errors are wrong (clustering on the panel repairs them) and that a dynamic model may be called for. The test needs at least three consecutive periods per panel and works with unbalanced panels and gaps.

Examples:

>>> import numpy as np, pandas as pd
>>> import statspai as sp
>>> rng = np.random.default_rng(0)
>>> rows = []
>>> for firm in range(80):
...     e = 0.0
...     for year in range(8):
...         e = 0.7 * e + rng.normal()  # AR(1) errors
...         x = rng.normal()
...         rows.append((firm, year, 1 + x + e, x))
>>> df = pd.DataFrame(rows, columns=["firm", "year", "y", "x"])
>>> out = sp.xtserial(df, "y", ["x"], id="firm", time="year")
>>> out["df"], bool(out["pvalue"] < 0.01)
((1, 79), True)
References

wooldridge2010econometric; drukker2003testing

xtsum

xtsum(data: DataFrame, variables: Optional[Sequence[str]] = None, *, id: str) -> DataFrame

Overall, between and within summary statistics of panel variables.

Parameters:

Name Type Description Default
data DataFrame

Panel data in long format.

required
variables sequence of str

Variables to describe. Default: every numeric column except id.

None
id str

Panel identifier.

required

Returns:

Type Description
DataFrame

Indexed by (variable, component) with component in overall / between / within and columns mean, sd, min, max, obs. obs is the number of observations (overall), of panels (between) and the average number of periods per panel (within).

Notes

The between statistics describe the panel means xbar_i. The within statistics describe x_it - xbar_i + xbar, the deviation from the panel mean with the grand mean added back so that it is on the scale of x; its standard deviation uses the divisor N - 1. This is Stata's xtsum.

A variable with almost no within variation (a small within sd) is barely identified in a fixed-effects regression, whatever its overall variation.

Examples:

>>> import statspai as sp
>>> df = sp.dgp_panel(n_units=30, n_periods=6, seed=0)
>>> table = sp.xtsum(df, ["y", "x"], id="unit")
>>> list(table.columns)
['mean', 'sd', 'min', 'max', 'obs']
>>> float(table.loc[("y", "between"), "obs"])
30.0