DropMissingData

class feature_engine.imputation.DropMissingData(missing_only=True, threshold=None, variables=None)

DropMissingData() will delete rows containing missing values. It provides similar functionality to pandas.drop_na().

It works for numerical and categorical variables. You can enter the list of variables for which missing values should be evaluated. Alternatively, the imputer will evaluate missing data in all variables in the dataframe.

More details in the User Guide.

Parameters

variables: list, default=None

The list of variables to consider for the imputation. If None, the imputer will evaluate missing data in all variables in the dataframe. Alternatively, the imputer will evaluate missing data only in the variables in the list.

Note that if missing_only=True only variables with missing data in the train set will be considered to drop a row, which might be a subset of the indicated list.

missing_only: bool, default=True

If True, rows will be dropped when they show missing data in variables that had missing data during fit(). If False, rows will be dropped if there is missing data in any of the variables. This parameter only works when threshold=None, otherwise it is ignored.

threshold: int or float, default=None

Require that percentage of non-NA values in a row to keep it. If threshold=1, all variables need to have data to keep the row. If threshold=0.5, 50% of the variables need to have data to keep the row. If threshold=0.01, 10% of the variables need to have data to keep the row. If thresh=None, rows with NA in any of the variables will be dropped.

Attributes

variables_:

The variables for which missing data will be examined to decide if a row is dropped. The attribute variables_ is different from the parameter variables when the latter is None, or when only a subset of the indicated variables show NA in the train set if missing_only=True.

feature_names_in_:

List with the names of features seen during fit.

n_features_in_:

The number of features in the train set used in fit.

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from feature_engine.imputation import DropMissingData
>>> X = pd.DataFrame(dict(
>>>        x1 = [np.nan,1,1,0,np.nan],
>>>        x2 = ["a", np.nan, "b", np.nan, "a"],
>>>        ))
>>> dmd = DropMissingData()
>>> dmd.fit(X)
>>> dmd.transform(X)
    x1 x2
2  1.0  b

Methods

fit:	Find the variables for which missing data should be evaluated.
fit_transform:	Fit to data, then transform it.
get_feature_names_out:	Get output feature names for transformation.
get_params:	Get parameters for this estimator.
set_params:	Set the parameters of this estimator.
return_na_data:	Returns a dataframe with the rows that contain missing data.
transform:	Remove rows with missing data.

fit(X, y=None)

Find the variables for which missing data should be evaluated to decide if a row should be dropped.

Parameters

X: pandas dataframe of shape = [n_samples, n_features]

The training data set.

y: pandas Series, default=None

y is not needed in this imputation. You can pass None or y.

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_feature_names_out(input_features=None)

Get output feature names for transformation. In other words, returns the variable names of transformed dataframe.

Parameters

input_featuresarray or list, default=None

This parameter exits only for compatibility with the Scikit-learn pipeline.

• If None, then feature_names_in_ is used as feature names in.

• If an array or list, then input_features must match feature_names_in_.

Returns

feature_names_out: list

Transformed feature names.

rtype

List[Union[str, int]] ..

get_params(deep=True)

Get parameters for this estimator.

Parameters

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns

paramsdict

Parameter names mapped to their values.

return_na_data(X)

Returns the subset of the dataframe with the rows with missing values. That is, the subset of the dataframe that would be removed with the transform() method. This method may be useful in production, for example if we want to store or log the removed observations, that is, rows that will not be fed into the model.

Parameters

X_na: pandas dataframe of shape = [n_samples_with_na, features]

The subset of the dataframe with the rows with missing data.

:rtype: :py:class:`~pandas.core.frame.DataFrame`

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters

**paramsdict

Estimator parameters.

Returns

selfestimator instance

Estimator instance.

transform(X)

Remove rows with missing data.

Parameters

X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to be transformed.

Returns

X_new: pandas dataframe

The complete case dataframe for the selected variables, of shape [n_samples - n_samples_with_na, n_features]

rtype

DataFrame ..