- class feature_engine.selection.DropHighPSIFeatures(split_col=None, split_frac=0.5, split_distinct=False, cut_off=None, switch=False, threshold=0.25, bins=10, strategy='equal_frequency', min_pct_empty_bins=0.0001, missing_values='raise', variables=None)#
DropHighPSIFeatures drops features which Population Stability Index (PSI) value is above a given threshold. The PSI of a numerical feature is an indication of the shift in its distribution; a feature with high PSI could therefore be considered unstable.
A bigger PSI value indicates a bigger shift in the feature distribution.
Different thresholds can be used to assess the magnitude of the distribution shift according to the PSI value. The most commonly used thresholds are:
Below 10%, the variable has not experienced a significant shift.
Above 25%, the variable has experienced a major shift.
Between those two values, the shift is intermediate.
To compute the PSI the DropHighPSIFeatures splits the dataset in two:
First and foremost, the user should enter one variable which will be used to guide the data split. This variable can be of any data type. If the user does not enter a variable name, DropHighPSIFeatures will use the dataframe index.
Second, the user has the option to specify a proportion of observations to put in each data set, or alternatively, provide a cut-off value.
If the user specifies a proportion through the
split_fracparameter, the data will be sorted to accommodate that proportion. If
split_fracis 0.5, 50% of the observations will go to either basis or test sets. If
split_fracis 0.6, 60% of the samples will go to the basis data set and the remaining 40% to the test set.
split_distinctis True, the data will be sorted considering unique values in the selected variables. Check the parameter below for more details.
If the user defines a numeric cut-off value or a specific date using the
cut_offparameter, the observations with value <= cut-off will go to the basis data set and the remaining ones to the test set. For categorical values this means they are sorted alphabetically and cut accordingly.
If the user passes a list of values in the
cut-off, the observations with the values in the list, will go to the basis set, and the remaining ones to the test set.
- split_col: string or int, default=None.
The variable that will be used to split the dataset into the basis and test sets. If None, the dataframe index will be used.
split_colcan be a numerical, categorical or datetime variable. If
split_colis a categorical variable, and the splitting criteria is given by
split_frac, it will be assumed that the labels of the variable are sorted alphabetically.
- split_frac: float, default=0.5.
The proportion of observations in each of the basis and test dataframes. If
split_fracis 0.6, 60% of the observations will be put in the basis data set.
split_distinctis True, the indicated fraction may not be achieved exactly. See parameter
split_distinctfor more details.
cut_offis not None,
split_fracwill be ignored and the data split based on the
- split_distinct: boolean, default=False.
split_fracis applied to the vector of unique values in
split_colinstead of being applied to the whole vector of values. For example, if the values in
split_colare [1, 1, 1, 1, 2, 2, 3, 4] and
split_fracis 0.5, we have the following: -
split_distinct=Falsesplits the vector in two equally sized parts: [1, 1, 1, 1] and [2, 2, 3, 4]. This involves that 2 dataframes with 4 observations each are used for the PSI calculations. -
split_distinct=Truecomputes the vector of unique values in
split_col([1, 2, 3, 4]) and splits that vector in two equal parts: [1, 2] and [3, 4]. The number of observations in the two dataframes used for the PSI calculations is respectively 6 ([1, 1, 1, 1, 2, 2]) and 2 ([3, 4]).
- cut_off: int, float, date or list, default=None
Threshold to split the dataset based on the
split_colvariable. If int, float or date, observations where the
split_colvalues are <= threshold will go to the basis data set and the rest to the test set. If
cut_offis a list, the observations where the
split_colvalues are within the list will go to the basis data set and the remaining observations to the test set. If
cut_offis not None, this parameter will be used to split the data and
split_fracwill be ignored.
- switch: boolean, default=False.
If True, the order of the 2 dataframes used to determine the PSI (basis and test) will be switched. This is important because the PSI is not symmetric, i.e., PSI(a, b) != PSI(b, a)).
- threshold: float, default = 0.25.
The threshold to drop a feature. If the PSI for a feature is >= threshold, the feature will be dropped. The most common threshold values are 0.25 (large shift) and 0.10 (medium shift).
- bins: int, default = 10
Number of bins or intervals. For continuous features with good value spread, 10 bins is commonly used. For features with lower cardinality or highly skewed distributions, lower values may be required.
- strategy: string, default=’equal_frequency’
If the intervals into which the features should be discretized are of equal size or equal number of observations. Takes values “equal_width” for equally spaced bins or “equal_frequency” for bins based on quantiles, that is, bins with similar number of observations.
- min_pct_empty_bins: float, default = 0.0001
Value to add to empty bins or intervals. If after sorting the variable values into bins, a bin is empty, the PSI cannot be determined. By adding a small number to empty bins, we can avoid this issue. Note, that if the value added is too large, it may disturb the PSI calculation.
- missing_values: str, default=’raise’
Whether to perform the PSI feature selection on a dataframe with missing values. Takes values ‘raise’ or ‘ignore’. If ‘ignore’, missing values will be dropped when determining the PSI for that particular feature. If ‘raise’ the transformer will raise an error and features will not be selected.
- variables: list, default=None
The list of variables to evaluate. If None, the transformer will evaluate all numerical variables in the dataset.
List with the features that will be dropped.
The variables to evaluate.
Dictionary containing the PSI value per feature.
Value used to split the dataframe into basis and test. This value is computed when not given as parameter.
The number of features in the train set used in fit.
Find features with high PSI values.
Remove features with high PSI values.
Fit to the data. Then transform it.
- fit(X, y=None)#
Find features with high PSI values.
- Xpandas dataframe of shape = [n_samples, n_features]
The training dataset.
- ypandas series. Default = None
y is not needed in this transformer. You can pass y or None.
- fit_transform(X, y=None, **fit_params)#
Fit to data, then transform it.
Fits transformer to
ywith optional parameters
fit_paramsand returns a transformed version of
- Xarray-like of shape (n_samples, n_features)
- yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).
Additional fit parameters.
- X_newndarray array of shape (n_samples, n_features_new)
Get parameters for this estimator.
- deepbool, default=True
If True, will return the parameters for this estimator and contained subobjects that are estimators.
Parameter names mapped to their values.
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as
Pipeline). The latter have parameters of the form
<component>__<parameter>so that it’s possible to update each component of a nested object.
- selfestimator instance
Return dataframe with selected features.
- X: pandas dataframe of shape = [n_samples, n_features].
The input dataframe.
- X_new: pandas dataframe of shape = [n_samples, n_selected_features]
Pandas dataframe with the selected features.