Conformity¶

Split conformal prediction for sklearn-style regression and probabilistic classification estimators. Python 3.9+, NumPy 2+, and scikit-learn 1.6.1+.

pip install conformity-calib

Regression¶

Use a sklearn pipeline with a conformal regressor, constructor-level automatic calibration, and explicit interval requests:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from conformity import ConformalRegressor

X, y = make_regression(n_samples=1000, noise=20, random_state=42)
X_new = X[:5]  # Replace with new observations in practice.

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("conformal", ConformalRegressor(
        Ridge(),
        auto_calibrate=True,
        tts_kwargs={"random_state": 42},
    )),
])

# During development: score point predictions.
results = cross_validate(pipe, X, y, cv=5, scoring="r2")

# Fit the final pipeline; calibration happens internally.
pipe.fit(X, y)
points = pipe.predict(X_new)
# Later: request intervals for your own testing.
points, intervals = pipe.predict(X_new, return_interval=True)

This is the default usage pattern in these docs. Pipeline declares each step with an explicit name. Automatic calibration reserves 20% of each fit’s training observations by default. Point predictions work with ordinary sklearn scorers; request intervals explicitly with return_interval=True (optionally pass alpha, which defaults to 0.05). cross_validate fits clones, so fit the original pipeline before predicting.

The outer scaler fits on all observations passed to each pipeline fit, including the final wrapper's internal calibration features. This workflow is supported, but the usual split-conformal coverage proof does not apply to that placement of learned preprocessing. For that guarantee, put learned preprocessing inside the wrapped estimator, as described in THEORY.md.

Model-first configuration and sklearn pipelines¶

Pass a model, pipeline, or GridSearchCV as the estimator. By default, ordinary fit(X, y) splits, fits, and calibrates. Constructor tts_kwargs configures split options; explicit fit-time options override them.

With sklearn metadata routing disabled (the default), the outer pipeline forwards prediction keywords to the final wrapper. return_interval=False forces points; True returns points and intervals; omission returns points. A standard pipeline does not forward custom calibrate, predict_interval, or evaluate methods: transform inputs with pipe[:-1].transform(...) before calling these methods on pipe[-1].

For manual calibration, set auto_calibrate=False, hold out calibration observations before fitting any pipeline steps, then call pipe[-1].calibrate(pipe[:-1].transform(X_calib), y_calib). Calibration replaces previous scores and warns on replacement. Refitting or calling set_params invalidates calibration; set_params also invalidates the fitted model. A supplied fitted estimator is cloned and refitted; there is no prefit mode.

Automatic calibration with cross_validate¶

Use cross_validate(pipe, X, y, cv=5, scoring="r2") as in the regression example. Each clone splits its fold's training observations into model-training and calibration partitions. Fold test observations are reserved for scoring. No separate calibrate call or fit-time calibration flag is needed.

Set return_estimator=True to inspect the fitted fold pipelines; each final wrapper has its own n_calibration_ and calibration scores. A custom scorer receives (fitted_pipeline, X_test, y_test) and returns a scalar. For example:

def coverage(pipe, X_test, y_test):
    return pipe[-1].evaluate(
        pipe[:-1].transform(X_test), y_test, alpha=0.1
    )["coverage"]

results = cross_validate(
    pipe, X, y, cv=5,
    scoring={"r2": "r2", "coverage": coverage},
    return_estimator=True,
    error_score="raise",
)

Cross-validation evaluates separate split-conformal models; it does not combine their scores into a CV+ predictor. Empirical coverage can differ from the nominal level. If selecting a model using these scores, reserve fresh calibration observations for its final coverage guarantee. For classification's marginal coverage interpretation, use fold splitting independent of outcomes, such as shuffled KFold; sklearn's default classifier folds are stratified and require care about sampling assumptions.

Classification¶

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from conformity import ConformalClassifier

X, y = load_iris(return_X_y=True)
X_new = X[:5]  # Replace with new observations in practice.
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("conformal", ConformalClassifier(
        LogisticRegression(max_iter=1000),
        method="aps",  # or "lac" (default)
        auto_calibrate=True,
        tts_kwargs={"random_state": 42},
    )),
])
results = cross_validate(pipe, X, y, cv=5, scoring="accuracy")
pipe.fit(X, y)
labels = pipe.predict(X_new)
sets, probabilities = pipe.predict(X_new, return_set=True)
sets = pipe[-1].predict_set(pipe[:-1].transform(X_new))  # boolean (n, n_classes)

Labels can be strings or noncontiguous numbers. Set columns and probability columns follow classes_. Calibration labels absent from the fitted estimator are rejected. The estimator must implement predict_proba and expose classes_. For the coverage theorem, classes_ must include every possible future label; unseen classes cannot be covered even by an all-class set. Probabilities must be finite, between zero and one, and sum to one per row.

• LAC: score a candidate class by 1 - probability.
• APS: score it by cumulative descending probability up to and including that class. This implementation is deterministic and conservative; ties follow stable classes_ order. It does not implement randomized APS or RAPS.

Empty sets are permitted. P-values count ties conservatively: (1 + count(calibration_score >= candidate_score)) / (n + 1).

sklearn integration¶

predict and score use point predictions for sklearn scorers, model selection, cross-validation, and ensembles. Request intervals with return_interval=True or boolean sets with return_set=True. The explicit predict_interval / predict_set methods require calibration. Alpha controls conformal output and has no effect on point predictions.

from sklearn.model_selection import GridSearchCV
search = GridSearchCV(
    ConformalRegressor(Ridge()),
    {"estimator__alpha": [0.1, 1.0, 10.0]},
    scoring="neg_mean_absolute_error",
)
search.fit(X, y)
search.best_estimator_.calibrate(X_calib, y_calib)  # fresh, held-out observations

Both wrappers support cloning, nested estimator parameters, sparse CSR/CSC matrices, feature-count validation, and numeric DataFrames with feature-name validation. DataFrames are preserved for column-selecting pipelines. Sparse inputs still require support in the wrapped estimator. Multi-output regression, multilabel classification, precomputed kernels, metadata routing, and arbitrary structured/raw-text inputs are outside the current API.

Controls and metrics¶

Constructor options are estimator, auto_calibrate, and tts_kwargs; classifiers additionally expose method. tts_kwargs accepts typed train_test_split options. Automatic calibration defaults to test_size=0.2, train_size=None, random_state=None, shuffle=True, and stratify=None. Set random_state inside tts_kwargs to reproduce a split. A fit-time tts_kwargs mapping replaces the constructor mapping. Stratifying calibration by outcome can change the exchangeability assumptions; use it deliberately rather than relying on an automatic classification default.

fit(..., sample_weight=weights, **fit_params) forwards estimator fitting options. Sample weights are validated and split alongside training observations. Other fit parameters, except step-qualified sample weights, are passed unchanged: provide metadata aligned to the training partition, or split manually. Pipelines require step-qualified parameters such as ridge__sample_weight; these weight vectors are validated and automatically sliced too. Pass them with their estimator-relative step names. The unqualified sample_weight is forwarded as supplied; it is not automatically routed to a pipeline step. Calibration is unweighted.

evaluate(..., sample_weight=...) returns:

All conformal metrics are also available as standalone package exports and accept optional evaluation weights:

• prediction_interval_coverage, prediction_interval_width, interval_score
• prediction_set_coverage, prediction_set_size, prediction_set_empty_rate, prediction_set_singleton_rate

Boolean set coverage requires classes=pipe[-1].classes_. Metrics accept lists and arrays, reject malformed/empty inputs, and validate weights. Weights change the reported empirical metric, not the conformal guarantee. Interval score combines width and missed-bound penalties; lower is better.

Statistical contract¶

With n held-out calibration scores, the threshold is the ceil((n + 1) * (1 - alpha))-th order statistic, including an infinite sentinel at rank n + 1. There is no interpolation or clipping. If the requested rank exceeds n, regression intervals are unbounded and classification sets contain every fitted class; a warning explains why. Prediction includes candidates whose scores equal the threshold.

Under exchangeability of calibration and future observations conditional on the training information, with the score function fixed without calibration data, split conformal gives marginal coverage of at least 1 - alpha. It does not promise coverage for every individual, subgroup, or realized test batch. Reusing training data as calibration, tuning on calibration outcomes, time dependence, and distribution shift can invalidate this guarantee. Keep an independent test set for evaluation. Extremely small alpha requires large calibration sets to produce finite thresholds.

These conventions follow the split conformal regression literature and the sklearn estimator interface. Sorted calibration scores are cached once; threshold lookup is constant-time, and p-values use binary search rather than sorting on every prediction.

Development¶

uv sync --group dev
uv run pytest
uv run ruff check src tests

See DOCUMENTATION.md for the API and THEORY.md for score definitions, the rank proof, assumptions, pipeline placement, and limitations.

Releases¶

Set project.version in pyproject.toml, run uv lock, then commit and push your changes. Use a new version for every release.

Run Create GitHub Release from the GitHub Actions extension in VS Code, selecting the branch containing the release commit. You can also run it from VS Code's terminal with GitHub CLI:

gh workflow run create-release.yml --ref main

The workflow must first be committed to the repository's default branch to be available for manual runs. It reads the version from pyproject.toml, builds and checks the wheel and source archive, creates a matching tag (for example, v0.1.6) at the selected commit, and publishes a GitHub release with the .whl and .tar.gz files attached and generated release notes. It refuses to reuse an existing tag.

After creating the release, it calls python-publish.yml to publish those same distribution files to PyPI. GitHub suppresses release-event workflows for releases created with GITHUB_TOKEN, so the explicit reusable-workflow call performs the handoff without another token. The existing release: published trigger still supports releases created manually through GitHub; those runs build distributions before publishing. Tests and lint run separately on pushes and pull requests.

Configure a PYPI_API_TOKEN repository or pypi environment secret, or configure PyPI Trusted Publishing for .github/workflows/python-publish.yml and the pypi environment and leave the secret unset. Any protection rules on the pypi environment apply before publishing.