API reference¶

The README contains workflows. THEORY.md specifies the statistical assumptions, scores, rank proof and limits.

Default workflow¶

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from conformity import ConformalRegressor

X, y = make_regression(n_samples=1000, noise=20, random_state=42)
X_new = X[:5]  # Replace with new observations in practice.

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("conformal", ConformalRegressor(
        Ridge(),
        auto_calibrate=True,
        tts_kwargs={"random_state": 42},
    )),
])

results = cross_validate(pipe, X, y, cv=5, scoring="r2")

pipe.fit(X, y)
points = pipe.predict(X_new)
points, intervals = pipe.predict(X_new, return_interval=True)

This is the default usage pattern in these docs. Pipeline is imported directly from sklearn.pipeline and takes a list of named steps. Point predictions work with ordinary sklearn scorers; request intervals explicitly with return_interval=True (optionally pass alpha, which defaults to 0.05). cross_validate fits clones, so fit the original pipeline before predicting.

The outer scaler fits on all observations passed to each pipeline fit, including the final wrapper's internal calibration features. This workflow is supported, but the usual split-conformal coverage proof does not apply to that placement of learned preprocessing. For that guarantee, put learned preprocessing inside the wrapped estimator, as described in THEORY.md.

Lifecycle¶

1. Construct ConformalRegressor(estimator, *, auto_calibrate=True, tts_kwargs=None) or ConformalClassifier(estimator, *, method="lac", auto_calibrate=True, tts_kwargs=None).
2. fit(X, y, auto_calibrate=None, tts_kwargs=None, *, sample_weight=None, **fit_params) validates inputs, discards stale fitted/calibration state, clones the estimator, and fits it. With automatic calibration, splitting happens before the wrapped estimator is fitted. tts_kwargs accepts typed train_test_split options; defaults are test_size=0.2, train_size=None, random_state=None, shuffle=True, and stratify=None. Set random_state inside tts_kwargs for reproducible splits. A fit-time mapping replaces the constructor mapping. Unqualified and step-qualified *__sample_weight vectors are validated and sliced to the training partition. Other metadata is forwarded unchanged; unqualified weights are not automatically routed into a pipeline.
3. calibrate(X, y) validates the feature schema and targets and replaces calibration scores. Invalid calibration inputs leave previous valid scores intact. Calibration data must be independent of fitting and tuning.
4. predict_point(X) and score(X, y, sample_weight=None) need fitting only. Classification also exposes predict_proba(X).
5. predict_interval(X, alpha=0.05) or predict_set(X, alpha=0.05) needs calibration. Alpha must be a finite scalar strictly between zero and one. Boolean sets use classes_ order.
6. predict(X, alpha=0.05) returns ordinary predictions. Use return_interval=True for regression or return_set=True for classification to request a conformal tuple explicitly. Alpha does not affect point predictions.
7. evaluate(X, y, alpha=0.05, *, sample_weight=None) returns a dictionary of point and conformal diagnostics.

fit and calibrate return self. Calling set_params, including nested estimator parameters, invalidates both the fitted estimator and calibration. Refitting invalidates calibration even if the refit fails. Recalibration warns before replacing existing scores. Changing a classifier's method directly requires recalibration; use set_params for normal sklearn configuration.

Fitted attributes¶

Prediction records the most recent alpha/threshold for introspection. Concurrent prediction calls can overwrite these diagnostics; use the returned arrays as the result of each call. Do not concurrently mutate, fit, or recalibrate one instance. Clone separate instances for independent fitting. The wrapped estimator must itself be safe for concurrent inference if inference is shared.

Metrics¶

Coverage includes endpoints. Set metrics accept boolean membership arrays; coverage requires classes in column order. All diagnostics aggregate over samples and accept keyword-only sample_weight. Weights must be finite, nonnegative, correctly shaped, and have positive total weight.

Interval inputs must have nonempty shape (n_samples, 2) with ordered bounds and no NaN. Outward infinite bounds are accepted and have infinite width/interval score. Zero-weight observations are excluded before weighted aggregation, so an unbounded zero-weight interval does not create 0 * infinity NaNs.

interval_score is width plus 2 / alpha times the distance below the lower bound or above the upper bound. Use the same alpha as the interval construction.

Scope¶

This release implements unweighted split conformal absolute-residual regression and LAC/deterministic APS classification. It does not implement CV+, jackknife+, conformalized quantile regression, conditional/Mondrian calibration, online/time-series guarantees, weighted covariate-shift calibration, or randomized scores. Adding these needs separate calibration protocols and dedicated statistical validation.

Sklearn integration uses mixins before BaseEstimator, estimator cloning, validation, and fitted attributes. Point predictions follow the standard estimator contract. The default workflow uses the wrapper as the final pipeline step. For the usual split-conformal coverage guarantee, put learned preprocessing inside the supplied estimator so calibration observations are excluded before it fits.

Constructor auto_calibrate (default True) and tts_kwargs configure ordinary fit calls. Explicit fit-time calibration flags override the constructor. Regression predict(..., return_interval=True/False) and classification predict(..., return_set=True/False) request conformal output per call and support prediction through a standard Pipeline with metadata routing disabled.

Constructor parameters¶

Both wrappers require estimator as their only positional parameter. All other parameters are keyword-only. The estimator must be sklearn-cloneable and expose fit and predict; classifiers additionally require predict_proba and fitted classes_. The supplied instance is never fitted directly.

At fit time, auto_calibrate=None uses the constructor value. Explicit False disables automatic calibration for that fit. tts_kwargs=None uses constructor split options when automatic calibration is enabled; an explicit dictionary replaces those options. Supplying fit-time split options while automatic calibration is disabled raises ValueError.

Prediction methods and shapes¶

Here m is the number of query rows and c is the number of fitted classes.

Regression return_interval and classification return_set accept booleans and default to False. True returns (points, bounds) for regression or (boolean_sets, probabilities) for classification. Set columns follow classes_. All conformal methods require calibration; point methods require fitting only.

evaluate returns regression keys r2, mae, mse, coverage, mean_width, interval_score, or classification keys accuracy, log_loss, coverage, mean_size, empty_rate, singleton_rate. Evaluation data should be independent of training, calibration, and model selection when assessing generalization.

Verification¶

The suite checks sklearn estimator compliance, cloning, model selection, internal search isolation, outer pipeline forwarding for both tasks, training-weight alignment, score ties, p-values, and the infinite sentinel. Exact finite-population tests enumerate every possible future index, verifying the coverage lower bound for regression, LAC and deterministic APS, with ties and without relying on simulation tolerances. Tests verify implementation under specified assumptions; they cannot establish exchangeability of a user's data.

Cross-validation lifecycle¶

The default workflow scores point predictions during development and requests internally calibrated intervals later:

from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from conformity import ConformalRegressor

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("conformal", ConformalRegressor(
        Ridge(), auto_calibrate=True, tts_kwargs={"random_state": 42}
    )),
])
results = cross_validate(pipe, X, y, cv=5, scoring="r2")
pipe.fit(X, y)
points = pipe.predict(X_new)
points, intervals = pipe.predict(X_new, return_interval=True, alpha=0.1)

Automatic calibration reserves 20% of each fit’s training observations by default. The preceding scaler fits before the internal split, so this layout supports interval testing but does not establish the usual split-conformal coverage guarantee.

cross_validate clones the outer pipeline and wrapper for each fold. Set constructor auto_calibrate=True to make its ordinary fit(X_fold_train, y_fold_train) perform a fresh internal calibration split. return_estimator=True returns the fitted pipeline clones, each with its own final wrapper's estimator_, calibration scores and n_calibration_. The original pipeline remains unfitted. error_score="raise" surfaces fitting or scoring errors directly. See the complete cross-validation example.

A custom scorer has signature (fitted_estimator, X_test, y_test) and returns a scalar. With the default pipeline layout, fitted_estimator[-1].evaluate(fitted_estimator[:-1].transform(X_test), y_test, alpha=0.1)["coverage"] scores fold coverage. Each calibration partition comes only from the fold's training data. For the usual coverage proof, learned preprocessing belongs inside the wrapped estimator, so the split excludes calibration observations before any learned transformations fit. A wrapper that is the final outer step cannot move the split ahead of preceding steps.

Type annotations and code documentation¶

Every Python module, class, function, and method in src and tests has a docstring, and every function parameter and return value is annotated (implicit self is excluded). Ruff enforces this with the ANN rules and docstring presence rules. Run ruff check src tests and pytest from the development environment before submitting changes.

Array inputs use NumPy ArrayLike; feature matrices also accept SciPy sparse matrices and arrays. DataFrames implement the array conversion protocol and retain their columns at runtime. Numeric predictions use float64 arrays, boolean prediction sets use boolean arrays, and class labels preserve their original dtype. Array dimensions and finite-value constraints remain runtime checks. predict has a union return type because per-call flags determine whether it returns points or a conformal tuple. Use predict_point, predict_interval, or predict_set for a fixed return type.

Self describes chainable fitting and calibration methods. Any is restricted to arbitrary label dtypes and estimator-specific metadata such as fit keywords and split overrides. The packaged py.typed marker makes these inline annotations available to downstream type checkers. Ruff checks annotation presence; it does not perform static type checking.