MatchVariables

class feature_engine.preprocessing.MatchVariables(fill_value=nan, missing_values='raise', match_dtypes=False, verbose=True)

MatchVariables() ensures that the same variables observed in the train set are present in the test set. If the dataset to transform contains variables that were not present in the train set, they are dropped. If the dataset to transform lacks variables that were present in the train set, these variables are added to the dataframe with a value determined by the user (np.nan by default).

train = pd.DataFrame({
    "Name": ["tom", "nick", "krish", "jack"],
    "City": ["London", "Manchester", "Liverpool", "Bristol"],
    "Age": [20, 21, 19, 18],
    "Marks": [0.9, 0.8, 0.7, 0.6],
})

test = pd.DataFrame({
    "Name": ["tom", "sam", "nick"],
    "Age": [20, 22, 23],
    "Marks": [0.9, 0.7, 0.6],
    "Hobbies": ["tennis", "rugby", "football"]
})

match_columns = MatchVariables()

match_columns.fit(train)

df_transformed = match_columns.transform(test)

Note that in the returned dataframe, the variable “Hobbies” was removed and the variable “City” was added with np.nan:

df_transformed

    Name    City  Age  Marks
0    tom  np.nan   20    0.9
1    sam  np.nan   22    0.7
2   nick  np.nan   23    0.6

The order of the variables in the transformed dataset is also adjusted to match that observed in the train set.

More details in the User Guide.

Parameters

fill_value: integer, float or string. Default=np.nan

The values for the variables that will be added to the transformed dataset.

missing_values: string, default=’ignore’

Indicates if missing values should be ignored or raised. If ‘raise’ the transformer will return an error if the the datasets to fit or transform contain missing values. If ‘ignore’, missing data will be ignored when learning parameters or performing the transformation.

match_dtypes: bool, default=False

Indicates whether the dtypes observed in the train set should be applied to variables in the test set.

verbose: bool, default=True

If True, the transformer will print out the names of the variables that are added and / or removed from the dataset.

Attributes

feature_names_in_:

The variables present in the train set, in the order observed during fit.

n_features_in_:

The number of features in the train set used in fit.

dtype_dict_:

If match_dtypes is set to True, then this attribute will exist, and it will contain a dictionary of variables and their corresponding dtypes.

Examples

>>> import pandas as pd
>>> from feature_engine.preprocessing import MatchVariables
>>> X_train = pd.DataFrame(dict(x1 = ["a","b","c"], x2 = [4,5,6]))
>>> X_test = pd.DataFrame(dict(x1 = ["c","b","a","d"],
>>>                             x2 = [5,6,4,7],
>>>                             x3 = [1,1,1,1]))
>>> mv = MatchVariables(missing_values="ignore")
>>> mv.fit(X_train)
>>> mv.transform(X_train)
x1  x2
0  a   4
1  b   5
2  c   6
>>> mv.transform(X_test)
The following variables are dropped from the DataFrame: ['x3']
  x1  x2
0  c   5
1  b   6
2  a   4
3  d   7

>>> import pandas as pd
>>> from feature_engine.preprocessing import MatchVariables
>>> X_train = pd.DataFrame(dict(x1 = ["a","b","c"],
>>>                             x2 = [4,5,6], x3 = [1,1,1]))
>>> X_test = pd.DataFrame(dict(x1 = ["c","b","a","d"], x2 = [5,6,4,7]))
>>> mv = MatchVariables(missing_values="ignore")
>>> mv.fit(X_train)
>>> mv.transform(X_train)
  x1  x2  x3
0  a   4   1
1  b   5   1
2  c   6   1
>>> mv.transform(X_test)
The following variables are added to the DataFrame: ['x3']
  x1  x2  x3
0  c   5 NaN
1  b   6 NaN
2  a   4 NaN
3  d   7 NaN

Methods

fit:	Identify the variable names in the train set.
fit_transform:	Fit to the data. Then transform it.
get_feature_names_out:	Get output feature names for transformation.
get_params:	Get parameters for this estimator.
set_params:	Set the parameters of this estimator.
transform:	Add or delete variables to match those observed in the train set.

fit(X, y=None)

Learns and stores the names of the variables in the training dataset.

Parameters

X: pandas dataframe of shape = [n_samples, n_features]

The input dataframe.

y: None

y is not needed for this transformer. You can pass y or None.

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_feature_names_out(input_features=None)

Get output feature names for transformation. In other words, returns the variable names of transformed dataframe.

Parameters

input_featuresarray or list, default=None

This parameter exits only for compatibility with the Scikit-learn pipeline.

• If None, then feature_names_in_ is used as feature names in.

• If an array or list, then input_features must match feature_names_in_.

Returns

feature_names_out: list

Transformed feature names.

rtype

List[Union[str, int]] ..

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns

routingMetadataRequest

A MetadataRequest encapsulating routing information.

get_params(deep=True)

Get parameters for this estimator.

Parameters

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns

paramsdict

Parameter names mapped to their values.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters

**paramsdict

Estimator parameters.

Returns

selfestimator instance

Estimator instance.

transform(X)

Drops variables that were not seen in the train set and adds variables that were in the train set but not in the data to transform. In other words, it returns a dataframe with matching columns.

Parameters

X: pandas dataframe of shape = [n_samples, n_features]

The data to transform.

Returns

X_new: Pandas dataframe, shape = [n_samples, n_features]

The dataframe with variables that match those observed in the train set.

rtype

DataFrame ..