How to export SHAP local explanations to dataframe?-CodePudding

I am working on a binary classification using random forest and trying out SHAP to explain the model predictions.

However, I would like to convert the SHAP local explanation plots with values into a pandas dataframe for each instance.

Is there any one here who can help me with exporting SHAP local explanations to pandas dataframe for each instance?

I know that SHAPASH has .to_pandas() method but couldn't find anything like that in SHAP

I tried something like below based on the SO post here but it doesn't help

feature_names = shap_values.feature_names
    shap_df = pd.DataFrame(shap_values.values, columns=feature_names)
    vals = np.abs(shap_df.values).mean(0)
    shap_importance = pd.DataFrame(list(zip(feature_names, vals)), columns=['col_name', 'feature_importance_vals'])
    shap_importance.sort_values(by=['feature_importance_vals'], ascending=False, inplace=True)

I expect my output something like below. Here, negative sign indicates feature contribution for class 0 and positive values indicates feature contribution for class 1

subject_id       Feature importance      value (contribution)
   1                       F1                  31
   1                       F2                  27
   1                       F3                  20
   1                       F5                  - 10
   1                       F9                  - 29

CodePudding user response：

If you have a model like this:

import xgboost
import shap
import warnings
warnings.filterwarnings("ignore")

# train XGBoost model
X,y = shap.datasets.boston()
model = xgboost.XGBRegressor().fit(X, y)

# explain the model's predictions using SHAP values
# (same syntax works for LightGBM, CatBoost, and scikit-learn models)
background = shap.maskers.Independent(X, max_samples=100)
explainer = shap.Explainer(model, background, algorithm="tree")
sv = explainer(X)

you can decompose your results like this:

sv.base_values[0]

22.342787810446044

sv.values[0]

array([-7.68297079e-01, -4.38205232e-02,  3.46814548e-01, -4.06731364e-03,
       -3.17875379e-01, -5.37296545e-01,  2.68567768e-01, -1.30198611e 00,
       -4.83524088e-01, -4.39375216e-01,  2.94188969e-01,  2.43096180e-02,
        4.63890554e 00])

model.predict(X.iloc[[0]])

array([24.019339], dtype=float32)

Which is exactly equal to:

sv.base_values[0]   sum(sv.values[0])

24.01933200249436

If you want to put results to Pandas df:

pd.DataFrame(sv.values[0], index = X.columns)

         0
CRIM    -0.768297
ZN      -0.043821
INDUS    0.346815
CHAS    -0.004067
NOX     -0.317875
RM      -0.537297
AGE      0.268568
DIS     -1.301986
RAD     -0.483524
TAX     -0.439375
PTRATIO  0.294189
B        0.024310
LSTAT    4.638906

Alternatively, if you wish everything arranged row-wise:

pd.DataFrame(
    np.c_[sv.base_values, sv.values],
    columns = ["bv"]   list(X.columns)
)