DropMissingData#
- class feature_engine.imputation.DropMissingData(missing_only=True, threshold=None, variables=None)[source]#
DropMissingData() will delete rows containing missing values. It provides similar functionality to pandas.drop_na().
It works for numerical and categorical variables. You can enter the list of variables for which missing values should be evaluated. Alternatively, the imputer will evaluate missing data in all variables in the dataframe.
More details in the User Guide.
- Parameters
- variables: list, default=None
The list of variables to consider for the imputation. If None, the imputer will evaluate missing data in all variables in the dataframe. Alternatively, the imputer will evaluate missing data only in the variables in the list.
Note that if
missing_only=True
only variables with missing data in the train set will be considered to drop a row, which might be a subset of the indicated list.- missing_only: bool, default=True
If
True
, rows will be dropped when they show missing data in variables that had missing data duringfit()
. IfFalse
, rows will be dropped if there is missing data in any of the variables. This parameter only works whenthreshold=None
, otherwise it is ignored.- threshold: int or float, default=None
Require that percentage of non-NA values in a row to keep it. If
threshold=1
, all variables need to have data to keep the row. Ifthreshold=0.5
, 50% of the variables need to have data to keep the row. Ifthreshold=0.01
, 10% of the variables need to have data to keep the row. Ifthresh=None
, rows with NA in any of the variables will be dropped.
- Attributes
- variables_:
The variables for which missing data will be examined to decide if a row is dropped. The attribute
variables_
is different from the parametervariables
when the latter isNone
, or when only a subset of the indicated variables show NA in the train set ifmissing_only=True
.- feature_names_in_:
List with the names of features seen during
fit
.- n_features_in_:
The number of features in the train set used in fit.
Examples
>>> import pandas as pd >>> import numpy as np >>> from feature_engine.imputation import DropMissingData >>> X = pd.DataFrame(dict( >>> x1 = [np.nan,1,1,0,np.nan], >>> x2 = ["a", np.nan, "b", np.nan, "a"], >>> )) >>> dmd = DropMissingData() >>> dmd.fit(X) >>> dmd.transform(X) x1 x2 2 1.0 b
Methods
fit:
Find the variables for which missing data should be evaluated.
fit_transform:
Fit to data, then transform it.
get_feature_names_out:
Get output feature names for transformation.
get_params:
Get parameters for this estimator.
set_params:
Set the parameters of this estimator.
return_na_data:
Returns a dataframe with the rows that contain missing data.
transform:
Remove rows with missing data.
- fit(X, y=None)[source]#
Find the variables for which missing data should be evaluated to decide if a row should be dropped.
- Parameters
- X: pandas dataframe of shape = [n_samples, n_features]
The training data set.
- y: pandas Series, default=None
y is not needed in this imputation. You can pass None or y.
- fit_transform(X, y=None, **fit_params)[source]#
Fit to data, then transform it.
Fits transformer to
X
andy
with optional parametersfit_params
and returns a transformed version ofX
.- Parameters
- Xarray-like of shape (n_samples, n_features)
Input samples.
- yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).
- **fit_paramsdict
Additional fit parameters.
- Returns
- X_newndarray array of shape (n_samples, n_features_new)
Transformed array.
- get_feature_names_out(input_features=None)[source]#
Get output feature names for transformation. In other words, returns the variable names of transformed dataframe.
- Parameters
- input_featuresarray or list, default=None
This parameter exits only for compatibility with the Scikit-learn pipeline.
If
None
, thenfeature_names_in_
is used as feature names in.If an array or list, then
input_features
must matchfeature_names_in_
.
- Returns
- feature_names_out: list
Transformed feature names.
- get_params(deep=True)[source]#
Get parameters for this estimator.
- Parameters
- deepbool, default=True
If True, will return the parameters for this estimator and contained subobjects that are estimators.
- Returns
- paramsdict
Parameter names mapped to their values.
- return_na_data(X)[source]#
Returns the subset of the dataframe with the rows with missing values. That is, the subset of the dataframe that would be removed with the
transform()
method. This method may be useful in production, for example if we want to store or log the removed observations, that is, rows that will not be fed into the model.- Parameters
- X_na: pandas dataframe of shape = [n_samples_with_na, features]
The subset of the dataframe with the rows with missing data.
- :rtype: :py:class:`~pandas.core.frame.DataFrame`
- set_params(**params)[source]#
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as
Pipeline
). The latter have parameters of the form<component>__<parameter>
so that it’s possible to update each component of a nested object.- Parameters
- **paramsdict
Estimator parameters.
- Returns
- selfestimator instance
Estimator instance.
- transform(X)[source]#
Remove rows with missing data.
- Parameters
- X: pandas dataframe of shape = [n_samples, n_features]
The dataframe to be transformed.
- Returns
- X_new: pandas dataframe
The complete case dataframe for the selected variables, of shape [n_samples - n_samples_with_na, n_features]
- rtype
DataFrame
..