polars-learn
Skip to content
polars-learnSchemaValidator

SchemaValidator

Reference

[source]
class polars_learn.SchemaValidator(
    mode='strict',
    *,
    allow_extra_columns=False,
    enforce_column_order=True,
    extra_columns='preserve',
    convert_pandas=False,
    warn_on_conversion=True,
    verbosity=0,
)

Learn a schema and validate subsequent Polars DataFrames or LazyFrames.

Use this transformer when training data defines your model's input contract. fit records every training column's name, order, and Polars dtype; transform checks new data against that learned schema and handles permitted casts, extra columns, and ordering according to your options. Strict mode rejects dtype changes, semi-strict allows defined lossless widening, and permissive attempts Polars casts that may lose information. Missing required columns always fail. This validates structure and dtype compatibility, not business meaning, nullability, or feature distributions. Choose SchemaSpec when the required schema is already known.

Polars is the execution engine. Native DataFrames and LazyFrames are accepted; pandas input requires convert_pandas=True and is converted behind the scenes. Output is native Polars by default, including an eager Polars frame for converted pandas input. Use set_output(transform="pandas") for compatibility output; install polars-learn[pandas] for pandas and PyArrow. Lazy output defers row execution, while eager Polars or pandas output collects during transform. Prefer native Polars input/output for speed and to avoid conversion overhead.

Parameters

mode {"strict", "semi-strict", "permissive"}, default="strict"

Validation policy. Strict mode requires identical dtypes. Semi-strict allows only the documented lossless widening policy. Permissive mode attempts Polars casts with strict=True. Successful casts can be lossy; failure occurs immediately for eager input or at lazy execution.

allow_extra_columns bool, default=False

Allow columns not present during fitting. Missing required columns always raise, regardless of the validation mode.

enforce_column_order bool, default=True

Require the relative order of required columns to match training. If false, output uses fitted column order followed by preserved extras.

extra_columns {"preserve", "drop"}, default="preserve"

Treatment of permitted extras. This option does not itself permit extra columns; allow_extra_columns must also be true.

convert_pandas bool, default=False

Allow pandas DataFrames and convert them to Polars. Requires pandas; some dtypes also require PyArrow.

verbosity {0, 1, 2}, default=0

Print progress at 1 and schema/operation details at 2; 0 is silent.

warn_on_conversion bool, default=True

Emit PandasConversionWarning when converting pandas input.

Attributes

schema_ dict of str to polars.DataType

Learned required columns and dtypes, in training order.

n_features_in_ int

Number of training columns.

feature_names_in_ ndarray of shape (n_features_in_,), dtype=object

Required input names in training order.

output_features_ ndarray of shape (n_features_in_,), dtype=object

Fitted required names. Inference-only extras are not included.

Raises

TypeError

If common options have invalid types.

ValueError

If mode, extra_columns, or verbosity is invalid.

See also

FeatureEngineer: Engineer features after validating raw inputs.

Notes

Semi-strict allows integer widening when the source dtype's entire range fits inside the target dtype, Float32 to Float64, and integer to Float64 only when every non-null value round-trips exactly. It rejects string to numeric conversion, narrowing, and Null dtype to another dtype. Null values in compatible columns are preserved; nullability constraints are not learned. Fit always learns a fresh schema; transform always validates against it. Input data and schemas owned by other transformers are never mutated. Lazy inputs are validated using metadata without collecting rows. Cast failures and value-dependent integer-to-float checks run at collection. Explicit eager output wraps Polars cast errors in SchemaValidationError; collecting a returned LazyFrame can raise native Polars exceptions. Default output preserves input kind. set_output can force "polars" (eager) or "polars-lazy" output, or pandas output for compatibility. With enforced order and preserved extras, output keeps input order; disabling order enforcement puts required columns first.

Examples

>>> import polars as pl
>>> from polars_learn import SchemaValidator
>>> train = pl.DataFrame({"x": pl.Series([1], dtype=pl.Int64)})
>>> test = pl.DataFrame({"x": pl.Series([2], dtype=pl.Int32)})
>>> validator = SchemaValidator(mode="semi-strict").fit(train)
>>> validator.transform(test).schema["x"]
Int64

Methods

Method Purpose
fit(X, y=None) Learn column names, order, and dtypes without changing the input.
transform(X) Validate the learned schema and normalise permitted differences.
fit_transform(X, y=None, **fit_params) Fit to training data and return its transformed representation.
get_feature_names_out(input_features=None) Return the fitted output column names.
get_params(deep=True) Get estimator constructor parameters.
set_params(**params) Set constructor parameters using scikit-learn conventions.
set_output(*, transform=None) Configure native Polars output or pandas compatibility output.

fit

[source]
fit(X, y=None)

Learn column names, order, and dtypes without changing the input.

Call on training data to establish the schema used by later transforms. Reads column metadata and records every input column as required, in its training order. It does not cast values, learn nullability, or evaluate lazy rows. A successful refit replaces the previous learned contract; y is ignored because validation does not learn from the target.

Parameters

X polars.DataFrame, polars.LazyFrame or pandas.DataFrame

Training data whose schema becomes the required schema. All input columns become required. Pandas requires convert_pandas=True. Lazy input uses metadata without evaluating rows.

y object, default=None

Ignored. Present for scikit-learn estimator compatibility.

Returns

self SchemaValidator

Fitted validator with a newly learned schema.

Raises

TypeError

If the input type is unsupported or pandas conversion is disabled.

ValueError

If validation configuration is invalid.

transform

[source]
transform(X)

Validate the learned schema and normalise permitted differences.

Call after fitting to validate new observations before feature engineering or prediction. Checks all required columns against schema_, applies the chosen dtype policy, and handles permitted extras and ordering. The fitted schema remains unchanged. Structural errors fail during this call; lazy value-dependent cast checks run when the resulting query is collected. The original input is never modified.

Output follows set_output: native Polars by default, eager Polars for "polars", a LazyFrame for "polars-lazy", or a pandas DataFrame for "pandas". Eager Polars and pandas output collect lazy rows. Pandas output uses Polars conversion defaults and may change dtype/null representation; missing pandas or PyArrow raises ImportError at conversion. Pandas input requires convert_pandas=True independently of the output choice.

Parameters

X polars.DataFrame, polars.LazyFrame or pandas.DataFrame

Input containing all fitted required columns. Additional columns must be explicitly permitted. Pandas requires conversion enabled.

Returns

X_validated polars.DataFrame, polars.LazyFrame or pandas.DataFrame

Validated input with any configured casts, ordering, and extra column policy applied. Default preserves input kind; set_output can force the return container. The original input is never mutated.

Raises

sklearn.exceptions.NotFittedError

If fit has not completed successfully.

TypeError

If input or options are unsupported, or pandas conversion is disabled.

ValueError

If validation configuration is invalid.

SchemaValidationError

If required columns are missing, extras or order are disallowed, dtypes are incompatible, or a cast fails. Multiple mismatches are reported together, with eager cast exceptions preserved as the cause.

Notes

Lazy schema and policy failures occur during this method. Permissive cast failures occur at lazy execution and can be native Polars errors. Lossless integer-to-float checks run at execution and raise SchemaValidationError on failure. Explicit eager output wraps Polars collection errors in SchemaValidationError with their causes.

fit_transform

[source]
fit_transform(
    X,
    y=None,
    **fit_params,
)

Fit to training data and return its transformed representation.

Convenience method for preparing training data: calls fit(X, y) and then transform(X) on the same input. This updates fitted state and returns the prepared frame. The target is ignored and extra fit parameters are unsupported. See fit for the class-specific learned contract and transform for validation and output behavior.

Output follows set_output: native Polars by default, eager Polars for "polars", a LazyFrame for "polars-lazy", or a pandas DataFrame for "pandas". Eager Polars and pandas output collect lazy rows. Pandas output uses Polars conversion defaults and may change dtype/null representation; missing pandas or PyArrow raises ImportError at conversion. Pandas input requires convert_pandas=True independently of the output choice.

Parameters

X polars.DataFrame, polars.LazyFrame or pandas.DataFrame

Training data. Pandas input requires convert_pandas=True. Lazy input is checked through metadata without evaluating rows.

y object, default=None

Ignored. Present for scikit-learn estimator compatibility.

**fit_params dict

Accepted for the scikit-learn method signature. Additional fit parameters are unsupported and raise TypeError when supplied.

Returns

X_transformed polars.DataFrame, polars.LazyFrame or pandas.DataFrame

Result of fitting and transforming the same input. Default preserves input kind; set_output can force eager, lazy, or pandas output.

See also

fit: Learn the fitted schema and feature names. transform: Apply the fitted transformer and its validation policies.

get_feature_names_out

[source]
get_feature_names_out(input_features=None)

Return the fitted output column names.

Use these names to label model inputs or inspect a fitted pipeline. Returns a copy of the names recorded during fitting; it does not transform data, collect lazy rows, or refit the transformer. Supplied input names are checked against all fitted input names in exact order. For SchemaValidator, inference-only extra columns are not included; this is the fitted contract rather than a live query's column list.

Parameters

input_features array-like of str or None, default=None

If supplied, names must exactly match feature_names_in_.

Returns

feature_names_out ndarray of shape (n_features_out_,), dtype=object

Copy of fitted output names. SchemaValidator excludes inference-only extras. SchemaSpec includes preserved training extras. A selector lists selected names, including any renames; a passthrough lists unchanged selected names. An engineer lists all output names.

Raises

sklearn.exceptions.NotFittedError

If the transformer has not been fitted.

ValueError

If supplied input names differ from fitted names or order.

get_params

[inherited API]
get_params(deep=True)

Get estimator constructor parameters.

Parameters

deep bool, default=True

Include nested estimator parameters when present.

Returns

params dict

Constructor parameters and, when requested, nested parameters. Output configuration is managed separately by set_output.

set_params

[inherited API]
set_params(**params)

Set constructor parameters using scikit-learn conventions.

Parameters

**params dict

Constructor parameter names and values. Nested estimator parameters use the step__parameter convention where supported.

Returns

self estimator instance

This estimator with updated configuration.

Raises

ValueError

If a parameter name is unknown.

Notes

Changing parameters does not immediately validate configuration or clear fitted attributes. Refit after contract changes. Invalid parameter values fail at the next relevant fit or transform operation.

set_output

[source]
set_output(*, transform=None)

Configure native Polars output or pandas compatibility output.

Call this when a downstream consumer needs a particular frame container. It only stores a choice for subsequent transform/fit_transform calls; it does not fit, transform, or collect data at configuration time. Use "default" to follow each call's input kind, "polars" for eager Polars, "polars-lazy" for a lazy plan, or "pandas" for an eager pandas DataFrame. Converted pandas input is treated as eager Polars by default. Pandas input permission remains separate: enable convert_pandas on any transformer receiving pandas, even when pandas output is requested.

Parameters

transform {"default", "polars", "polars-lazy", "pandas"} or None, default=None

Default preserves the input kind. "polars" collects an eager DataFrame; "polars-lazy" returns a LazyFrame. None leaves the configuration unchanged. "pandas" collects if necessary and converts the result to a pandas DataFrame. Requires the pandas extra.

Returns

self estimator instance

This transformer.

Raises

ValueError

If an unsupported output container is requested.

Notes

Prefer native Polars output for speed and to avoid conversion overhead. Pandas output is provided for compatibility and uses Polars to_pandas defaults, including a fresh RangeIndex and dtype/null conversion. Conversion raises ImportError if pandas or PyArrow is unavailable. Input conversion remains separately controlled by convert_pandas. Automatic scikit-learn output wrapping and global output settings are bypassed to preserve native names and permitted extra columns. Output choices survive cloning. None leaves the current choice intact; "default" resets input-kind preservation. Fit always returns self and does not collect lazy rows. Forcing lazy output from eager input may wrap an already evaluated result; use lazy input to defer evaluation.

See also

Shared estimator conventions · Production guidance · Persistence

Concepts and worked workflows.