How to properly use MeanEncoder for categorical encoding in a k fold loop-CodePudding

I want to use MeanEncoder from the feature-engine in my k-fold loop for encoding categorical data. It seems that after the tranform step the encoder introduces NaN values for certain columns in my dataset. The code is as follows

from sklearn.model_selection import KFold

from sklearn import linear_model

kf = KFold(n_splits=2)

linear_reg = linear_model.LinearRegression()

kfold_rmse = []

X = housing.drop(columns=['Price'], axis=1)
y = housing['Price']

for train_index, test_index in kf.split(X):
    
    X_train, X_test = X.iloc[train_index, :], X.iloc[test_index, :]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    X_train.drop(columns=['BuildingArea','YearBuilt', 'Rooms'], axis=1, inplace=True)
    X_test.drop(columns=['BuildingArea','YearBuilt', 'Rooms'], axis=1, inplace=True)
    
    random_imputer = RandomSampleImputer(variables=['Car', 'CouncilArea'])
    random_imputer.fit(X_train)

    X_train = random_imputer.transform(X_train)
    X_test = random_imputer.transform(X_test)

    X_train[descrete_var] = X_train[descrete_var].astype('O')
    X_test[descrete_var] = X_test[descrete_var].astype('O')

    mean_encoder = MeanEncoder(variables=categorical_var descrete_var)
    mean_encoder.fit(X_train,y_train)
    print(X_test.isnull().mean()) # <--------- No NaN columns


    X_train = mean_encoder.transform(X_train)
    X_test = mean_encoder.transform(X_test)
    
    print(X_test.isnull().mean()) # # <--------- NaN columns introduced

    # Fit the model
    # linear_reg_model = linear_reg.fit(X_train, y_train) 
    # y_pred_linear_reg = linear_reg_model.predict(X_test)
    
    # # Calculate the RMSE for each fold and append it
    # rmse = mean_squared_error(y_test, y_pred_linear_reg, squared=False)
    # kfold_rmse.append(rmse)

For further context, here is the output I get:

...
Suburb           0.0
Type             0.0
Method           0.0
SellerG          0.0
Distance         0.0
Postcode         0.0
Bedroom2         0.0
Bathroom         0.0
Car              0.0
Landsize         0.0
CouncilArea      0.0
Regionname       0.0
Propertycount    0.0
Month_name       0.0
day              0.0
Year             0.0
dtype: float64
Suburb           0.000000
Type             0.000000
Method           0.000000
SellerG          0.014138
Distance         0.000000
Postcode         0.000000
Bedroom2         0.000000
Bathroom         0.000295
...
Month_name       0.000000
day              0.191605
Year             0.000000

This obviously causes problems for the model prediction because LinearRegression can not accept NaN values. I think this may be an issue with how I'm using MeanEncoder in the loop with kfold. Is there something I'm doing wrong or not understanding about either the k-fold process or MeanEncoder?

CodePudding user response：

Your test folds contain categories unseen at training time, and the encoder by default encodes those as NaN. From the documentation:

errors: string, default=’ignore’

Indicates what to do, when categories not present in the train set are encountered during transform. If ‘raise’, then rare categories will raise an error. If ‘ignore’, then rare categories will be set as NaN and a warning will be raised instead.