Skip to content

sp.paper() — Data → Publication-Draft Pipeline

One call from a DataFrame + a natural-language question to a publication-quality draft (markdown, LaTeX, or Word) with EDA, identification check, recommended estimator, fitted result, and robustness suite — pre-assembled.

When to use

  • You have data and a research question; you want a draft you can edit, not a slide deck.
  • You're orchestrating an autonomous agent that needs to produce empirical reports without holding the analyst's hand through each stage.
  • You want a single artifact that captures why the estimator was chosen (identification + assumptions), not just the point estimate.

If you only need the fitted model → use the matched estimator directly (sp.regress, sp.did, sp.iv, ...). If you want the full workflow without rendering → use sp.causal().

Quick start

import statspai as sp
import pandas as pd

df = pd.read_stata("nlsw88.dta")

draft = sp.paper(
    df,
    "effect of training on wage",
    treatment="training",
    y="wage",
    covariates=["education", "experience"],
)

print(draft.to_markdown())   # human readable
draft.write("nlsw88_draft.tex")  # LaTeX
draft.write("nlsw88_draft.docx") # Word (requires python-docx)

What you get

A PaperDraft with seven sections:

Section Contents
Question The natural-language question + parsed outcome / treatment / design.
Data Sample size, missingness top-5, outcome moments, treatment distribution, mean covariates by arm, std-diff.
Identification Verdict from sp.check_identification plus per-finding severity / fix.
Estimator Recommended method + sp.<fn>() call + rationale + key assumptions.
Results Estimate, SE, 95% CI, p-value, n_obs (or coefficient table for regressions).
Robustness Method-specific suite (e.g. parallel trends + e-value for DiD; first-stage F + AR-CI for IV).
References BibTeX-style entries pulled from result.cite().

Plus an extra Pipeline notes section if any stage failed (so the draft never silently hides errors).

The question parser

sp.paper() runs a lightweight parser to extract column hints from the natural-language question — but explicit kwargs always win.

Pattern Example Hints
"effect of X on Y" "effect of training on wage" treatment=training, y=wage
"Y ~ X" "wage ~ training" y=wage, treatment=training
"difference-in-differences" / "did" "DiD effect of ..." design='did'
"regression discontinuity" / "rd" "RD with running variable score" design='rd', running_var='score'
"instrument(ing)? " "using distance as an instrument" design='iv', instrument='distance'
"discontinuity at " / "threshold " "threshold at 0.5" cutoff=0.5, design='rd'

The parser never overrides explicit args — it only fills in what you didn't pass. To bypass it entirely, pass everything explicitly.

draft = sp.paper(
    df,
    question="(any string)",  # parser ignored when args complete
    y="wage", treatment="training", design="did",
    time="year", id="worker_id",
)

Rendering formats

Markdown

The default. Easy to render in Jupyter, GitHub, or as a starting point for a PR description.

draft.to_markdown()

LaTeX

Wraps the same content in a complete article document with a proper thebibliography. Inline markdown markers (**bold**, ` code, bullet lists, code fences) are translated to LaTeX equivalents.

tex = draft.to_tex()
draft.write("paper.tex")

Word

Requires pip install python-docx. Falls back to a markdown file with a notice header if the dependency is missing — never crashes silently.

draft.to_docx("paper.docx")

Combining with the rest of StatsPAI

With LLM-DAG

# Step 1: build a validated DAG (P1-A)
dag = sp.llm_dag_constrained(df, variables=[...],
                              oracle=...).to_dag()

# Step 2: feed it into the paper pipeline
draft = sp.paper(df, "effect of X on Y", dag=dag)

With agent tool-use

sp.paper is registered with full agent metadata (assumptions, pre_conditions, failure_modes, alternatives):

schema = sp.function_schema('paper')
# → JSON-schema usable as an OpenAI / Anthropic tool definition

The PaperDraft.to_dict() method returns a JSON-serializable payload suitable for round-tripping through agent tool calls.

For Agents

  • Pre-conditions: data must contain the outcome (either passed via y= or parsed from "effect of X on Y"). The pipeline never silently invents a column.
  • Recovery: failures in any single stage are captured in a Pipeline notes section rather than raising; agents should check len(draft.workflow.diagnostics.findings) and the presence of "Pipeline notes" before accepting the draft as complete.
  • Determinism: with the same data and arguments, output is bit-for-bit reproducible (no LLM is invoked unless the user passes dag= from a prior LLM-DAG run).
  • Cost: O(1) workflow runs (one diagnose + one estimate + one robustness). For sensitivity sweeps, use sp.spec_curve separately.

Caveats

  1. Heuristic parser — if your question phrasing is unusual, pass y and treatment explicitly. The parser is conservative on purpose: it would rather miss a hint than mis-fill one.
  2. Robustness suite is design-specific — DiD gets parallel-trends + e-value; IV gets first-stage F. Custom robustness checks should be added to the underlying CausalWorkflow (sp.causal(...)).
  3. Citations are pulled from each result's cite() method when available; not every estimator implements one yet.
  4. No LLM is called by default — sp.paper is purely deterministic unless you wire LLM-DAG via dag=.

References

  • CausalWorkflow (sp.causal(...)): the underlying orchestrator.
  • sp.check_identification: identification-stage diagnostics.
  • sp.recommend: estimator-selection rule engine.
  • sp.evalue_from_result, sp.honest_did, sp.anderson_rubin_ci, sp.rdsensitivity: per-design robustness primitives.

For Agents

Pre-conditions - data must contain the outcome column (y or parsed) - If treatment given, it must be a column - A fitted StatsPAI result or CausalQuestion is available. - Citations and identifying assumptions have been attached or can be inferred.

Identifying assumptions - Question parser is heuristic — explicit kwargs always win - Underlying sp.causal() determines design when not specified - Generated prose is a draft; authors remain responsible for causal claims.

Failure modes → recovery

Symptom Exception Remedy Try next
ValueError 'Could not determine the outcome y' ValueError Pass y=... explicitly or include 'effect of X on Y' in the question text
Pipeline notes section appears in draft (none — informational) One pipeline stage failed; inspect draft.workflow.diagnostics and pipeline_errors
Missing citations, assumptions, or validation notes in generated text AssumptionWarning Call audit_result() or attach citations before rendering the paper section. sp.audit_result

Alternatives (ranked) - sp.causal - sp.recommend - sp.replication_pack - sp.paper_tables - sp.modelsummary

Typical minimum N: 1