matplotlib - Databricks 中的形状渲染

问题描述

我正在尝试可视化 Shap 指数来解释 xgboost 机器学习模型。可以在 google collab 中实现这一点，但在 Databricks 中很难实现。

import shap
import pandas as pd
import numpy as np
import matplotlib
import xgboost as xgb

df = pd.read_csv("/dbfs/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv").iloc[:,1:]

#label encode the categorical features:
df = pandasData.copy() 
df["clarity"] = df["clarity"].astype('category')
df["color"] = df["color"].astype('category')
df["cut"] = df["cut"].astype('category')

df["clarity_cat"] = df["clarity"].cat.codes
df["color_cat"] = df["color"].cat.codes
df["cut_cat"] = df["cut"].cat.codes
df= df.drop(["clarity","color","cut"],axis = 1)

Y = df['price']
X = df.drop(['price'], axis=1)

X_traintest, X_valid, Y_traintest, Y_valid = train_test_split(X, Y, test_size=0.3, random_state=7)

model = xgb.XGBRegressor(learning_rate=0.02, n_estimators=3161, max_depth=3)
model.fit(X_traintest, Y_traintest, eval_metric="rmse", verbose = False)

目前这有效

shap.initjs() # load JS visualization code to notebook

explainer = shap.TreeExplainer(model) # explain the model's predictions using SHAP values

shap_values = explainer.shap_values(X_traintest)
    shap_explain = shap.force_plot(explainer.expected_value, shap_values[0,:], X.iloc[0,:]) # visualize the first prediction's explanation
    display(shap_display)

并生产在此处输入图像描述

这不起作用（但在 Colab 中起作用）：

shap.initjs()
shap_display = shap.dependence_plot('y', shap_values, X_traintest)
display(shap_display) #tried matplotlib=True/False)

最终看起来像这样

在此处输入图像描述

什么时候应该是这样的

在此处输入图像描述

标签： matplotlibpysparkdatabricksazure-databricks

matplotlib - Databricks 中的形状渲染

问题描述

解决方案

推荐阅读