python - 通过先前训练的模型预测看不见的数据

问题描述

我正在使用 Scikit-learn 执行监督机器学习。我有两个数据集。第一个数据集包含具有 X 特征和 Y 标签的数据。第二个数据集仅包含 X 个特征，但没有 Y 标签。我可以成功地为训练/测试数据执行 LinearSVC 并获得测试数据集的 Y 标签。

现在，我想使用我为第一个数据集训练的模型来预测第二个数据集的标签。如何在 Scikit-learn 中使用从第一个数据集到第二个数据集（看不见的标签）的预训练模型？

我尝试的代码片段： 来自以下评论的更新代码：

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import pandas as pd
import pickle


# ----------- Dataset 1: for training ----------- #
# Sample data ONLY
some_text = ['Books are amazing',
             'Harry potter book is awesome. It rocks',
             'Nutrition is very important',
             'Welcome to library, you can find as many book as you like',
             'Food like brocolli has many advantages']
y_variable = [1,1,0,1,0]

# books = 1 : y label
# food = 0 : y label

df = pd.DataFrame({'text':some_text,
                   'y_variable': y_variable
                          })

# ------------- TFIDF process -------------#
tfidf = TfidfVectorizer()
features = tfidf.fit_transform(df['text']).toarray()
labels = df.y_variable
features.shape


# ------------- Build Model -------------#
model = LinearSVC()
X_train, X_test, y_train, y_test= train_test_split(features,
                                                 labels,
                                                 train_size=0.5,
                                                 random_state=0)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)


# Export model
pickle.dump(model, open('model.pkl', 'wb'))
# Read the Model
model_pre_trained = pickle.load(open('model.pkl','rb'))


# ----------- Dataset 2: UNSEEN DATASET ----------- #

some_text2 = ['Harry potter books are amazing',
             'Gluten free diet is getting popular']

unseen_df = pd.DataFrame({'text':some_text2}) # Notice this doesn't have y_variable. This the is the data set I am trying to predict y_variable labels 1 or 0.


# This is where the ERROR occurs
X_unseen = tfidf.fit_transform(unseen_df['text']).toarray()
y_pred_unseen = model_pre_trained.predict(X_unseen) # error here: 
# ValueError: X has 11 features per sample; expecting 26


print(X_unseen.shape) # prints (2, 11)
print(X_train.shape) # prints (2, 26)


# Looking for an output like this for UNSEEN data
# Looking for results after predicting unseen and no label data. 
text                                   y_variable
Harry potter books are amazing         1
Gluten free diet is getting popular    0

正如我上面尝试的那样，它不必是泡菜代码。我正在寻找是否有人有建议，或者是否有任何预构建功能可以从 scikit 进行预测？

标签： pythonpython-3.xmachine-learningscikit-learn

如您所见，您的第一个tfidf是将输入转换为 26 个特征，而第二个tfidf是将它们转换为 11 个特征。因此发生错误是因为X_train与的形状不同X_unseen。提示告诉您，每个观察的X_unseen特征少于model训练接收的特征数量。

加载model第二个脚本后，您将在文本中安装另一个矢量化器。也就是说，tfidf第一个脚本和tfidf第二个脚本是不同的对象。为了使用进行预测model，您需要X_unseen使用原始的进行转换tfidf。为此，您必须导出原始矢量化器，将其加载到新脚本中并使用它转换新数据，然后再将其传递给model.

### Do this in the first program
# Dump model and tfidf
pickle.dump(model, open('model.pkl', 'wb'))
pickle.dump(tfidf, open('tfidf.pkl', 'wb'))

### Do this in the second program
model = pickle.load(open('model.pkl', 'rb'))
tfidf = pickle.load(open('tfidf.pkl', 'rb'))

# Use `transform` instead of `fit_transform`
X_unseen = tfidf.transform(unseen_df['text']).toarray()

# Predict on `X_unseen`
y_pred_unseen = model_pre_trained.predict(X_unseen)

python - 通过先前训练的模型预测看不见的数据

问题描述

解决方案

推荐阅读