python - sklearn 使用带有自定义指标的 RandomizedSearchCV 并捕获异常
问题描述
我在带有随机森林分类器的 sklearn 中使用 RandomizedSearchCV 函数。要查看不同的指标,我正在使用自定义评分
from sklearn.metrics import make_scorer, roc_auc_score, recall_score, matthews_corrcoef, balanced_accuracy_score, accuracy_score
acc = make_scorer(accuracy_score)
auc_score = make_scorer(roc_auc_score)
recall = make_scorer(recall_score)
mcc = make_scorer(matthews_corrcoef)
bal_acc = make_scorer(balanced_accuracy_score)
scoring = {"roc_auc_score": auc_score, "recall": recall, "MCC" : mcc, 'Bal_acc' : bal_acc, "Accuracy": acc }
这些自定义记分器用于随机搜索
rf_random = RandomizedSearchCV(estimator=rf, param_distributions=random_grid, n_iter=100, cv=split, verbose=2,
random_state=42, n_jobs=-1, error_score=np.nan, scoring = scoring, iid = True, refit="roc_auc_score")
现在的问题是,当我使用自定义拆分时,AUC 会抛出异常,因为这个精确拆分只有一个类标签。
我不想更改拆分,因此是否有可能在 RandomizedSearchCV 或 make_scorer 函数中捕获这些异常?因此,例如,如果没有计算其中一个指标(由于异常),只需输入 NaN 并继续下一个模型。
编辑:显然 error_score 不包括模型训练,但不包括度量计算。如果我使用例如 Accuracy 一切正常,我只会在只有一个类标签的折叠处收到警告。如果我使用例如 AUC 作为指标,我仍然会抛出异常。
在这里得到一些想法会很棒!
解决方案:定义一个自定义记分器,但有例外:
def custom_scorer(y_true, y_pred, actual_scorer):
score = np.nan
try:
score = actual_scorer(y_true, y_pred)
except ValueError:
pass
return score
这导致了一个新的指标:
acc = make_scorer(accuracy_score)
recall = make_scorer(custom_scorer, actual_scorer=recall_score)
new_auc = make_scorer(custom_scorer, actual_scorer=roc_auc_score)
mcc = make_scorer(custom_scorer, actual_scorer=matthews_corrcoef)
bal_acc = make_scorer(custom_scorer,actual_scorer=balanced_accuracy_score)
scoring = {"roc_auc_score": new_auc, "recall": recall, "MCC" : mcc, 'Bal_acc' : bal_acc, "Accuracy": acc }
这又可以传递给 RandomizedSearchCV 的打分参数
我发现的第二个解决方案是:
def custom_auc(clf, X, y_true):
score = np.nan
y_pred = clf.predict_proba(X)
try:
score = roc_auc_score(y_true, y_pred[:, 1])
except Exception:
pass
return score
这也可以传递给评分参数:
scoring = {"roc_auc_score": custom_auc, "recall": recall, "MCC" : mcc, 'Bal_acc' : bal_acc, "Accuracy": acc }
(改编自这个答案)
解决方案
您可以拥有一个通用记分器,它可以将其他记分器作为输入,检查结果,捕获它们抛出的任何异常并在它们上返回一个固定值。
def custom_scorer(y_true, y_pred, actual_scorer):
score = np.nan
try:
score = actual_scorer(y_true, y_pred)
except Exception:
pass
return score
然后您可以使用以下方法调用它:
acc = make_scorer(custom_scorer, actual_scorer = accuracy_score)
auc_score = make_scorer(custom_scorer, actual_scorer = roc_auc_score,
needs_threshold=True) # <== Added this to get correct roc
recall = make_scorer(custom_scorer, actual_scorer = recall_score)
mcc = make_scorer(custom_scorer, actual_scorer = matthews_corrcoef)
bal_acc = make_scorer(custom_scorer, actual_scorer = balanced_accuracy_score)
重现示例:
import numpy as np
def custom_scorer(y_true, y_pred, actual_scorer):
score = np.nan
try:
score = actual_scorer(y_true, y_pred)
except Exception:
pass
return score
from sklearn.metrics import make_scorer, roc_auc_score, accuracy_score
acc = make_scorer(custom_scorer, actual_scorer = accuracy_score)
auc_score = make_scorer(custom_scorer, actual_scorer = roc_auc_score,
needs_threshold=True) # <== Added this to get correct roc
from sklearn.datasets import load_iris
X, y = load_iris().data, load_iris().target
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV, KFold
cvv = KFold(3)
params={'criterion':['gini', 'entropy']}
gc = GridSearchCV(DecisionTreeClassifier(), param_grid=params, cv =cvv,
scoring={"roc_auc": auc_score, "accuracy": acc},
refit="roc_auc", n_jobs=-1,
return_train_score = True, iid=False)
gc.fit(X, y)
print(gc.cv_results_)
推荐阅读
- django - Django REST Framework GET 检索具有 NAME LIKE %NAME% 的行
- azure - 如何在 Azure Pipelines 中跨多个阶段传播矩阵定义
- python - 存储给定百分比的形状轮廓坐标
- typescript - 使用 keyof 进行高级 Typescript 推理
- python - 带有 partial_fit() 的 MLPRegressor
- sql-server - SQL Server 查询存储顶部资源执行计数不正确?
- java - 在 Javax.Swing 应用程序中有效处理幻数
- javascript - 基于来自父级的道具反应动态导入
- microsoft-graph-api - office365 Graph API 仅列出管理员用户
- javascript - 如何在不重定向的反应路由器链接中创建按钮组件?