export SHAP waterfall plot to dataframe-CodePudding

I am working on a binary classification using random forest model, neural networks in which am using SHAP to explain the model predictions. I followed the tutorial and wrote the below code to get the waterfall plot shown below

row_to_show = 20
data_for_prediction = ord_test_t.iloc[row_to_show]  # use 1 row of data here. Could use multiple rows if desired
data_for_prediction_array = data_for_prediction.values.reshape(1, -1)
rf_boruta.predict_proba(data_for_prediction_array)
explainer = shap.TreeExplainer(rf_boruta)
# Calculate Shap values
shap_values = explainer.shap_values(data_for_prediction)
shap.plots._waterfall.waterfall_legacy(explainer.expected_value[0], shap_values[0],ord_test_t.iloc[row_to_show])

This generated the plot as shown below

However, I want to export this to dataframe and how can I do it?

I expect my output to be like as shown below. I want to export this for the full dataframe. Can you help me please?

CodePudding user response：

If I recall correctly, you can do something like this with pandas

    import pandas as pd
    
    shap_values = explainer.shap_values(data_for_prediction)
    shap_values_df = pd.DataFrame(shap_values)

to get the feature names, you should do something like this (if data_for_prediction is a dataframe):

feature_names = data_for_prediction.columns.tolist()
shap_df = pd.DataFrame(shap_values.values, columns=feature_names)

CodePudding user response：

I'm a currenty using that :

def getShapReport(classifier,X_test):
   shap_values = shap.TreeExplainer(classifier).shap_values(X_test)
   shap.summary_plot(shap_values, X_test)
   shap.summary_plot(shap_values[1], X_test)
   return pd.DataFrame(shap_values[1])

It first displays the shap values for the model, and for each prediction after that, and finally it returns the dataframe for the positive class(i'm on an imbalance context)

It is for a Tree explainer and not a waterfall, but it is basically the same.