ConformalClassifier¶

Reference¶

class conformity.ConformalClassifier(
    estimator,
    *,
    method='lac',
    auto_calibrate=True,
    tts_kwargs=None,
)

Split conformal classification using LAC or deterministic APS scores.

Parameters¶

estimator BaseEstimator: Cloneable sklearn classifier implementing predict_proba.
method {'lac', 'aps'}, default='lac': Use lac (1 - probability) or aps (cumulative ranked probability).
auto_calibrate bool or None, default=True: Reserve held-out calibration data during fit; defaults to True.
tts_kwargs TrainTestSplitKwargs or None, default=None: train_test_split options for automatic calibration. By default, test_size=0.2, shuffle=True and stratify=None.

Attributes¶

method_ {"lac", "aps"}: Score method recorded by successful calibration; changing it requires recalibration. Shared fitted attributes are documented on the base class.

Attributes¶

estimator_ BaseEstimator: Fitted clone; created by fit and removed by parameter changes.
n_features_in_ int: Number of validated fitting features.
feature_names_in_ ndarray: Input names, present when fitting named string columns.
classes_ ndarray: Classifier labels in probability and membership-column order.
calibration_scores_ ndarray: Immutable held-out scores; created by successful calibration.
sorted_calibration_scores_ ndarray: Immutable sorted scores cached for rank queries.
n_calibration_ int: Number of held-out calibration observations.
alpha_used_ float: Miscoverage probability from the most recent threshold query.
quantile_level_ float: Finite-sample rank divided by calibration count; may exceed one.
q_level_ float: Most recent threshold, including infinity when the rank exceeds n.
is_calibrated_ bool: Whether held-out scores exist. Refitting clears calibration state.

Examples¶

>>> from sklearn.datasets import load_iris
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.pipeline import Pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> from conformity import ConformalClassifier
>>> X, y = load_iris(return_X_y=True)
>>> X_dev, X_test, y_dev, y_test = train_test_split(
...     X, y, test_size=0.2, random_state=0
... )
>>> X_train, X_calib, y_train, y_calib = train_test_split(
...     X_dev, y_dev, test_size=0.25, random_state=1
... )
>>> model = Pipeline([
...     ("scaler", StandardScaler()),
...     ("conformal", ConformalClassifier(
...         LogisticRegression(max_iter=500), method="lac", auto_calibrate=False
...     )),
... ])
>>> _ = model.fit(X_train, y_train)
>>> _ = model[-1].calibrate(model[:-1].transform(X_calib), y_calib)
>>> mask, probabilities = model.predict(X_test, alpha=0.1, return_set=True)
>>> mask.shape, probabilities.shape
((30, 3), (30, 3))
>>> model.classes_.tolist()
[0, 1, 2]
>>> mask.dtype.name
'bool'

Methods¶

Method	Purpose
calibrate	Replace held-out scores and return this predictor for method chaining.
evaluate	Return accuracy, log loss, coverage and prediction set diagnostics.
fit	Fit a fresh clone; optionally reserve an independent calibration split.
get_metadata_routing	Get metadata routing of this object.
get_params	Get parameters for this estimator.
predict	Return point labels, or (boolean membership sets, probabilities).
predict_p_values	Return conservative conformal p-values, including calibration ties.
predict_point	Return the wrapped estimator's labels without calibration.
predict_proba	Return validated probabilities in classes_ order.
predict_set	Return a boolean membership matrix of shape (n_samples, n_classes).
score	Return point accuracy using point predictions.
set_fit_request	Request metadata passed to the `fit` method.
set_params	Set sklearn parameters and invalidate fitted and calibrated state.
set_predict_request	Request metadata passed to the `predict` method.
set_score_request	Request metadata passed to the `score` method.
### calibrate

[source]

calibrate(X, y)

Replace held-out scores and return this predictor for method chaining.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
y array-like of shape (n_samples,): One target per sample; classification labels follow the fitted classes.

Returns¶

self object: Predictor with replacement calibration scores.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.

evaluate¶

[source]

evaluate(
    X,
    y,
    alpha=0.05,
    *,
    sample_weight=None,
)

Return accuracy, log loss, coverage and prediction set diagnostics.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
y array-like of shape (n_samples,): One target per sample; classification labels follow the fitted classes.
alpha float, default=0.05: Miscoverage probability strictly between zero and one.
sample_weight array-like of shape (n_samples,) or None, default=None: Optional finite, nonnegative weights; fitting or evaluation only, never calibration.

Returns¶

report dict of str to float: Keys are accuracy, log_loss, coverage, mean_size, empty_rate and singleton_rate.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.
RuntimeError: If conformal output is requested before calibration.

fit¶

[source]

fit(
    X,
    y,
    auto_calibrate=None,
    tts_kwargs=None,
    *,
    sample_weight=None,
    **fit_params,
)

Fit a fresh clone; optionally reserve an independent calibration split.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
y array-like of shape (n_samples,): One target per sample; classification labels follow the fitted classes.
auto_calibrate bool or None, default=None: Reserve an independent calibration split when fitting.
tts_kwargs TrainTestSplitKwargs or None, default=None: train_test_split options for this fit; requires automatic calibration. If provided, this mapping replaces the constructor mapping. Missing options use defaults: test_size=0.2, train_size=None, random_state=None, shuffle=True and stratify=None. Set random_state here for reproducible automatic splits.
sample_weight array-like of shape (n_samples,) or None, default=None: Optional finite, nonnegative weights; fitting or evaluation only, never calibration.
**fit_params dict: Estimator-specific fit metadata forwarded to the cloned estimator.

Returns¶

self object: Fitted predictor; calibrated when automatic calibration is enabled.

Raises¶

TypeError: If the estimator cannot be cloned or lacks the required prediction methods.
ValueError: If input shapes, values, weights or configuration are invalid.

get_metadata_routing¶

[inherited API]

get_metadata_routing()

Get metadata routing of this object.

Returns¶

routing MetadataRequest: A :class:`~sklearn.utils.metadata_routing.MetadataRequest` encapsulating routing information.

get_params¶

[inherited API]

get_params(deep=True)

Get parameters for this estimator.

Parameters¶

deep bool, default=True, default=True: If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns¶

params dict: Parameter names mapped to their values.

predict¶

[source]

predict(
    X,
    alpha=0.05,
    *,
    return_set=False,
)

Return point labels, or (boolean membership sets, probabilities).

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
alpha float, default=0.05: Miscoverage probability strictly between zero and one.
return_set bool, default=False: Request conformal output alongside ordinary predictions.

Returns¶

predictions ndarray or tuple of ndarray: Point labels, or (membership, probabilities) when return_set=True.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.
RuntimeError: If conformal output is requested before calibration.

predict_p_values¶

[source]

predict_p_values(X)

Return conservative conformal p-values, including calibration ties.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.

Returns¶

p_values ndarray of shape (n_samples, n_classes): Conservative conformal p-values in classes_ order, including ties.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.
RuntimeError: If conformal output is requested before calibration.

predict_point¶

[source]

predict_point(X)

Return the wrapped estimator's labels without calibration.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.

Returns¶

predictions ndarray of shape (n_samples,): Point predictions from the fitted estimator.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.

predict_proba¶

[source]

predict_proba(X)

Return validated probabilities in classes_ order.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.

Returns¶

probabilities ndarray of shape (n_samples, n_classes): Finite class probabilities in fitted classes_ order.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.

predict_set¶

[source]

predict_set(X, alpha=0.05)

Return a boolean membership matrix of shape (n_samples, n_classes).

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
alpha float, default=0.05: Miscoverage probability strictly between zero and one.

Returns¶

membership ndarray of bool, shape (n_samples, n_classes): Included labels in classes_ order; empty and all-class sets are permitted.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.
RuntimeError: If conformal output is requested before calibration.

score¶

[source]

score(
    X,
    y,
    sample_weight=None,
)

Return point accuracy using point predictions.

Parameters¶

X {array-like, sparse matrix, DataFrame} of shape (n_samples, n_features): Feature matrix of shape (n_samples, n_features); dense, sparse, or DataFrame.
y array-like of shape (n_samples,): One target per sample; classification labels follow the fitted classes.
sample_weight array-like of shape (n_samples,) or None, default=None: Optional finite, nonnegative weights; fitting or evaluation only, never calibration.

Returns¶

score float: Point-prediction R² for regression or accuracy for classification.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.
NotFittedError: If fit has not established a fitted estimator.

set_fit_request¶

[inherited API]

set_fit_request(
    *,
    auto_calibrate='$UNCHANGED$',
    sample_weight='$UNCHANGED$',
    tts_kwargs='$UNCHANGED$',
)

Request metadata passed to the fit method.

Parameters¶

auto_calibrate str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for auto_calibrate parameter in fit.
sample_weight str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for sample_weight parameter in fit.
tts_kwargs str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for tts_kwargs parameter in fit.

Returns¶

self object: The updated object.

set_params¶

[source]

set_params(**params)

Set sklearn parameters and invalidate fitted and calibrated state.

Parameters¶

**params dict: Sklearn parameters, including nested estimator parameters.

Returns¶

self object: Predictor with updated parameters and invalidated fitted state.

Raises¶

ValueError: If input shapes, values, weights or configuration are invalid.

set_predict_request¶

[inherited API]

set_predict_request(
    *,
    alpha='$UNCHANGED$',
    return_set='$UNCHANGED$',
)

Request metadata passed to the predict method.

Parameters¶

alpha str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for alpha parameter in predict.
return_set str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for return_set parameter in predict.

Returns¶

self object: The updated object.

set_score_request¶

[inherited API]

set_score_request(*, sample_weight='$UNCHANGED$')

Request metadata passed to the score method.

Parameters¶

sample_weight str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED, default='$UNCHANGED$': Metadata routing for sample_weight parameter in score.

Returns¶

self object: The updated object.

is_calibrated_¶

Report whether calibration scores are available.

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search