python - 为什么在相同数据上训练准确率为 99%,而预测准确率为 81%?
问题描述
我查找了与该问题类似的问题,但我仍然不明白为什么它会给出这样的结果。模型训练高达 99% 的准确率是正常的,但是当用于预测相同的确切数据时,它给出的准确率较低,在这种情况下为 81%?它不应该返回 99% 的准确率吗?
此外,当我呈现新的看不见的数据时,预测准确度是糟糕的 17%。这肯定是不对的。我知道模型在呈现新数据时应该低于模型的准确度,但不会差到 17%。
这是上下文的代码。为了便于阅读,我放了评论:
# Step 1) Split Data into Training and Prediction Sets
num_split_df_at = int(0.75*len(df))
np_train_data = df.iloc[0:num_split_df_at, columns_index_list].to_numpy()
np_train_target = list(df.iloc[0:num_split_df_at, 4])
np_predict_data = df.iloc[num_split_df_at:len(df), columns_index_list].to_numpy()
np_predict_target = list(df.iloc[num_split_df_at:len(df), 4])
# Step 2) Split Training Data into Training and Validation Sets
x_train, x_test, y_train, y_test = train_test_split(np_train_data, np_train_target, random_state=0)
# Step 3) Reshape Training and Validation Sets to (49, 5)
# prints: "(3809, 245)"
print(x_train.shape)
# prints: "(1270, 245)"
print(x_test.shape)
x_train = x_train.reshape(x_train.shape[0], round(x_train.shape[1]/5), 5)
x_test = x_test.reshape(x_test.shape[0], round(x_test.shape[1]/5), 5)
y_train = np.array(y_train)- 1
y_test = np.array(y_test)- 1
# prints: "(3809, 49, 5)"
print(x_train.shape)
# prints: "[0 1 2 3 4 5 6 7 8 9]"
print(np.unique(y_train))
# prints: "10"
print(len(np.unique(y_train)))
input_shape = (x_train.shape[1], 5)
# Step 4) Run Model
adam = keras.optimizers.Adam(learning_rate=0.0001)
model = Sequential()
model.add(Conv1D(512, 5, activation='relu', input_shape=input_shape))
model.add(Conv1D(512, 5, activation='relu'))
model.add(MaxPooling1D(3))
model.add(Conv1D(512, 5, activation='relu'))
model.add(Conv1D(512, 5, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
model.fit(x_train, y_train, batch_size=128, epochs=150, validation_data=(x_test, y_test))
print(model.summary())
model.save('model_1')
# Step 5) Predict on Exact Same Trained Data - Should Return High Accuracy
np_train_data = np_train_data.reshape(np_train_data.shape[0], round(np_train_data.shape[1]/5), 5)
np_train_target = np.array(np_train_target)- 1
predict_results = model.predict_classes(np_train_data)
print(accuracy_score(predict_results, np_train_target))
# Step 6) Predict on Validation Set
np_predict_data = np_predict_data.reshape(np_predict_data.shape[0], round(np_predict_data.shape[1]/5), 5)
np_predict_target = np.array(np_predict_target)- 1
predict_results = model.predict_classes(np_predict_data)
print(accuracy_score(predict_results, np_predict_target))
以下是预测结果:
我的输出可能的分类结果是:
[1 2 3 4 5 6 7 8 9 10]转换为“sparse_categorical_crossentropy”的[0 1 2 3 4 5 6 7 8 9]
解决方案
这是因为 Keras 模型的训练精度/损失是按批次计算的,然后取平均值(请参见此处)。相反,验证指标/性能是在所有传递的数据上同时计算的。
这只是为了在这个虚拟示例中进行验证。我们训练一个神经网络并将相同的训练数据作为有效数据传递。通过这种方式,我们可以在训练结束时比较 (a) 训练 acc、(b) 验证 acc 和 (c) accuracy_score。正如我们所看到的 (b) = (c) 但由于上述原因 (a) 与 (c) 和 (b) 不同
timestamp, features, n_sample = 45, 2, 1000
n_class = 10
X = np.random.uniform(0,1, (n_sample, timestamp, features))
y = np.random.randint(0,n_class, n_sample)
model = Sequential()
model.add(Conv1D(8, 3, activation='relu', input_shape=(timestamp, features)))
model.add(MaxPooling1D(3))
model.add(Conv1D(8, 3, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(n_class, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(X, y, batch_size=128, epochs=5, validation_data=(X, y))
history.history['accuracy'][-1] # (a)
history.history['val_accuracy'][-1] # (b)
accuracy_score(y, np.argmax(model.predict(X), axis=1)) # (c)
推荐阅读
- docker - 在 alpine docker 容器上运行 selenium 测试时出错
- javascript - 显示和格式化嵌套的 SVG
- c - 在 C 中使用块结构进行宏替换(K&R 问题 4.14)
- android - 带有 https 方案的 Android Facebook 应用程序链接
- spring-boot - 在某些方法中如何不考虑@NotBlank
- android - 测试 Rx Java 间隔
- excel - 在excel VBA中没有for
- css - 如何不垂直溢出表格单元格并打印表格?
- python - 从不断变化的字符串列表中写入排序的 csv 文件。蟒蛇 2.7
- vb6 - 如何制作将输出打印到命令提示符的 VB6 控制台应用程序