DropMissingData#

class feature_engine.imputation.DropMissingData(missing_only=True, threshold=None, variables=None)[source]#

DropMissingData() will delete rows containing missing values. It provides similar functionality to pandas.drop_na().

It works for numerical and categorical variables. You can enter the list of variables for which missing values should be evaluated. Alternatively, the imputer will evaluate missing data in all variables in the dataframe.

More details in the User Guide.

Parameters
missing_only: bool, default=True

If True, rows will be dropped when they show missing data in variables with missing data in the train set, that is, in the data set used in fit(). If False, rows will be dropped if there is missing data in any of the variables. This parameter only works when threshold=None, otherwise it is ignored.

variables: list, default=None

The list of variables to consider for the imputation. If None, the imputer will evaluate missing data in all variables in the dataframe. Alternatively, the imputer will evaluate missing data only in the variables in the list.

Note that if missing_only=True only variables with missing data in the train set will be considered to drop a row, which might be a subset of the indicated list.

threshold: int or float, default=None

Require that percentage of non-NA values in a row to keep it. If threshold=1, all variables need to have data to keep the row. If threshold=0.5, 50% of the variables need to have data to keep the row. If threshold=0.01, 10% of the variables need to have data to keep the row. If thresh=None, rows with NA in any of the variables will be dropped.

Attributes
variables_:

The variables for which missing data will be examined to decide if a row is dropped. The attribute variables_ is different from the parameter variables when the latter is None, or when only a subset of the indicated variables show NA in the train set if missing_only=True.

n_features_in_:

The number of features in the train set used in fit.

Methods

fit:

Find the variables for which missing data should be evaluated.

transform:

Remove rows with missing data.

fit_transform:

Fit to the data, then transform it.

return_na_data:

Returns a dataframe with the rows that contain missing data.
fit(X, y=None)[source]#

Find the variables for which missing data should be evaluated to decide if a row should be dropped.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The training data set.

y: pandas Series, default=None

y is not needed in this imputation. You can pass None or y.

fit_transform(X, y=None, **fit_params)[source]#

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters
Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns
X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_params(deep=True)[source]#

Get parameters for this estimator.

Parameters
deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns
paramsdict

Parameter names mapped to their values.

return_na_data(X)[source]#

Returns the subset of the dataframe with the rows with missing values. That is, the subset of the dataframe that would be removed with the transform() method. This method may be useful in production, for example if we want to store or log the removed observations, that is, rows that will not be fed into the model.

Parameters
X_na: pandas dataframe of shape = [n_samples_with_na, features]

The subset of the dataframe with the rows with missing data.

:rtype:py:class:~pandas.core.frame.DataFrame
set_params(**params)[source]#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters
**paramsdict

Estimator parameters.

Returns
selfestimator instance

Estimator instance.

transform(X)[source]#

Remove rows with missing data.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to be transformed.

Returns
X_new: pandas dataframe

The complete case dataframe for the selected variables, of shape [n_samples - n_samples_with_na, n_features]

:rtype:py:class:~pandas.core.frame.DataFrame

previous

AddMissingIndicator

next

Categorical Encoding