python - 我无法理解我在 Kaggle 比赛中的训练和测试数据做错了什么
问题描述
在机器学习 kaggle 微课程中,您可以找到这些数据集和代码来帮助您为比赛制作预测模型:https [put your user name here]
: //www.kaggle.com//exercise-categorical-variables/edit
它为您提供了两个数据集:1 个训练数据集和 1 个测试数据集,您将使用它们来进行预测并提交以查看您在比赛中的排名
所以在:
第 5 步:生成测试预测并提交结果
我写了这段代码:
EDITED
# Read the data
X = pd.read_csv('../input/train.csv', index_col='Id')
X_test = pd.read_csv('../input/test.csv', index_col='Id')
# Remove rows with missing target, separate target from predictors
X.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X.SalePrice
X.drop(['SalePrice'], axis=1, inplace=True)
# To keep things simple, we'll drop columns with missing values
cols_with_missing = [col for col in X.columns if X[col].isnull().any()]
X.drop(cols_with_missing, axis=1, inplace=True)
X_test.drop(cols_with_missing, axis=1, inplace=True)
#print(X_test.shape, X.shape)
X_test.head()
# Break off validation set from training data
X_train, X_valid, y_train, y_valid = train_test_split(X, y,
train_size=0.8, test_size=0.2,
random_state=0)
X_train.head()
#Asses Viability of method ONE-HOT
# Get number of unique entries in each column with categorical data
object_nunique = list(map(lambda col: X_train[col].nunique(), object_cols))
d = dict(zip(object_cols, object_nunique))
# Print number of unique entries by column, in ascending order
sorted(d.items(), key=lambda x: x[1])
# Columns that will be one-hot encoded ####<<<<I THINK THAT THE PROBLEM STARTS HERE>>>>#####
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]
##############For X_train
# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))
# One-hot encoding removed index; put it back
OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index
# Remove categorical columns (will replace with one-hot encoding)
num_X_train = X_train.drop(object_cols, axis=1)
num_X_valid = X_valid.drop(object_cols, axis=1)
# Add one-hot encoded columns to numerical features
OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)
OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)
##############For X_test
low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]
# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
#Se não retirar os NAs a linha abaixo dá erro
X_test.dropna(axis = 0, inplace=True)
OH_cols_test = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))
#print(OH_cols_test.shape, OH_cols_train.shape)
# One-hot encoding removed index; put it back
OH_cols_test.index = X_test.index
# Remove categorical columns (will replace with one-hot encoding)
num_X_test = X_test.drop(object_cols, axis=1)
# Add one-hot encoded columns to numerical features
OH_X_test = pd.concat([num_X_test, OH_cols_test], axis=1)
#print(OH_X_test.shape ,OH_X_valid.shape)
# Define and fit model
model = RandomForestRegressor(n_estimators=100, criterion='mae', random_state=0)
model.fit(OH_X_train, y_train)
# Get validation predictions and MAE
preds_test = model.predict(OH_X_test)
# Save predictions in format used for competition scoring
output = pd.DataFrame({'Id': X_test.index,
'SalePrice': preds_test})
output.to_csv('submission.csv', index=False)
当我尝试预处理数据集时,我得到不同的行用于训练数据和测试数据。然后我无法拟合模型并做出预测。
我认为我应该只拆分测试数据集来完成所有这些操作,但是y
比它多 1 行X_test
,然后我就无法进行拆分。
所以我认为我必须使用训练数据集进行拆分,然后将其拟合为测试数据集的预测
解决方案
我相信你的问题发生在这一行:
low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]
您正在为您的独特列引用 X_test。按照Kaggle教程,您应该引用 X_train,如下所示:
# Columns that will be one-hot encoded
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]
您似乎在这一行进一步犯了同样的错误:
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))
您已将其标记为 one-hot 编码训练列,但您使用的是 X_test 而不是 X_train。您正在混淆您的训练和测试集处理,这不是一个好主意。这一行应该是:
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
我建议您再次查看教程中的代码块,并确保您的所有数据集和变量都正确匹配,以便您处理正确的训练和测试数据。
推荐阅读
- pipenv - python pipenv:如何定义 Pipfile 文件夹位置
- node.js - Nestjs 是否支持同时在操作上获取和发布?
- docker - Docker:数据库、客户端和 API 之间的分离级别
- laravel - 如何在 laravel 的查询生成器中使用 with
- css - 如何使用 w3.css 制作全角和响应式表格
- python - Raspberry Pi Reboot 时运行 Python3 Web 服务器程序
- postgresql - 如何使用 golang-migrate 进行迁移
- database - 关于可空外键及其替代方案的实践
- mvel - 访问 MVEL ASTNode 信息
- moodle - 如何为moodle mobile设置自定义网址?