python - 如何使用 GridSearchCV 和 sklearn Pipeline 用训练数据的估算值估算测试数据
问题描述
我正在努力改进我在这里找到的 Kaggle 房价竞赛的提交。我正在使用此处提供的爱荷华州数据。我正在尝试使用管道(sklearn.pipeline.Pipeline)训练和测试我的模型,使用 GridSearchCV(sklearn.model_selection.GridSearchCV)进行交叉验证,并使用和使用 XGBRegressor(xgboost.XGBRegressor)。选择的特征具有分类数据和必须估算的 NaN 值 (sklearn.impute.SimpleImputer()。初始设置:
import pandas as pd
from xgboost import XGBRegressor
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.impute import SimpleImputer
# Path of the file to read.
iowa_file_path = '../input/train.csv'
original_home_data = pd.read_csv(iowa_file_path)
home_data = original_home_data.copy()
# delete rows where SalePrice is Nan
home_data.dropna(axis=0, subset=['SalePrice'], inplace=True)
# Create a target object and call it y
y = home_data.SalePrice
# Create X
features = ['LotArea', 'YearBuilt', '1stFlrSF', '2ndFlrSF', 'FullBath', 'BedroomAbvGr', 'TotRmsAbvGrd']
extra_features = ['OverallCond', 'GarageArea', 'LotFrontage', 'OverallQual', 'BsmtFinSF1', 'BsmtUnfSF', 'TotalBsmtSF', 'GrLivArea', 'MoSold']
categorical_data = ['LotShape', 'MSZoning', 'Neighborhood', 'BldgType', 'HouseStyle', 'Foundation', 'KitchenQual']
features.extend(extra_features)
features.extend(categorical_data)
X = home_data[features]
分类数据是由以下人员热编码的:
X = pd.get_dummies(X, prefix='OHE', columns=categorical_data)
通过以下方式收集具有缺失值的列:
cols_with_missing = (col for col in X.columns if X[col].isnull().any())
for col in cols_with_missing:
X[col + '_was_missing'] = X[col].isnull()
然后拆分训练和验证数据:
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1, test_size=0.25)
train_X, val_X = train_X.align(val_X, join='left', axis=1)
然后创建管道以使用回归器估算 NaN 的平均值
my_pipeline = Pipeline([('imputer', SimpleImputer()), ('xgbrg', XGBRegressor())])
param_grid = {
'xgbrg__n_estimators': [10, 50, 100, 500, 1000],
'xgbrg__learning_rate': [0.01, 0.04, 0.05, 0.1, 0.5, 1]
}
fit_params = {
'xgbrg__early_stopping_rounds': 10,
'xgbrg__verbose': False,
'xgbrg__eval_set': [(np.array(val_X), val_y)]
}
然后我初始化了交叉验证器:
searchCV = GridSearchCV(my_pipeline, cv=5, param_grid=param_grid, return_train_score=True, scoring='neg_mean_absolute_error')
然后我安装了我的交叉验证器:
searchCV = GridSearchCV(my_pipeline, cv=5, param_grid=param_grid, return_train_score=True, scoring='neg_mean_absolute_error')
并拟合模型(注意下一行):
searchCV.fit(X=np.array(train_X), y=train_y, **fit_params)
然后我对测试数据做了同样的事情(一个热编码,用 NaN 获取列,)
# path to file you will use for predictions
test_data_path = '../input/test.csv'
# read test data file using pandas
test_data = pd.read_csv(test_data_path)
# create test_X which comes from test_data but includes only the columns you used for prediction.
original_test_X = test_data[features]
test_X = original_test_X.copy()
# to one hot encode the data
test_X = pd.get_dummies(test_X, prefix='OHE', columns=categorical_data)
for col in cols_with_missing:
test_X[col + '_was_missing'] = test_X[col].isnull()
# to align the training and test data and discard columns not in the training data
X, test_X = X.align(test_X, join='inner', axis=1)
然后,我尝试用训练数据的平均值转换测试数据,以估算测试数据中的 NaN 值:
test_X = my_pipeline.named_steps['imputer'].transform(test_X)
然后我得到这个错误:
NotFittedError: This SimpleImputer instance is not fitted yet. Call 'fit' with appropriate arguments before using this method.
所以我什至不能用这条线进行预测:
test_preds = searchCV.predict(test_X)
这里可能有什么问题?
拟合后如何使用我的管道转换另一个数据集?
如果我尝试为测试数据创建一个新的 SimpleImputer() 实例并为 NaN 插补并执行 fit_transform:
test_pipeline = SimpleImputer()
test_X = test_pipeline.fit_transform(test_X)
我添加并运行:
test_preds = searchCV.predict(test_X)
我收到以下错误:
ValueError: X has 72 features per sample, expected 74
- 这里有什么问题?
解决方案
在缺失数据阶段优化我的模型时,我遇到了相同的“这个 SimpleImputer 实例尚未安装”错误。经过大量的试验和错误,以下为我做了诀窍:
在准备训练数据的同一循环中准备测试数据。基本上,应该同时为训练和测试数据运行“for col in cols_with_missing”循环。我也是这个领域的新手(上周刚开始),但我猜如果你单独运行 col 循环来训练和测试数据,我猜这个错误可能是由于列中的不匹配而发生的。
我的代码片段有效:
cols_with_missing = (col for col in X_train.columns
if X_train[col].isnull().any())
for col in cols_with_missing:
imputed_X_train_plus[col + '_was_missing'] = imputed_X_train_plus[col].isnull()
imputed_X_test_plus[col + '_was_missing'] = imputed_X_test_plus[col].isnull()
imputed_final_test_plus[col + 'was_missing'] = imputed_final_test_plus[col].isnull()
推荐阅读
- c++ - C++ 如何将 Array 理解为指针的观点?
- python - KeyPressEvent 没有焦点
- docker - Atlassian Bamboo Server 中 Docker 中的 Docker 权限被拒绝
- php - 使用 URL 路由器会导致标头授权检索问题
- flutter - 如何在颤动中构建动态范围滑块?
- python - GitCommandError:Cmd('git')失败,原因是:退出代码(1),pygithub
- c# - 获取 Azure Blob 容器
- vue.js - 如何清理 Vue.Js 中的输入
- python - Python 词频
- python - 如何在屏幕上获取枚举对象