时间戳

首页 > 解决方案 > 如何在经过训练的模型上使用新句子进行情感分析？

machine-learning - 如何在经过训练的模型上使用新句子进行情感分析？

问题描述

我使用朴素贝叶斯训练了一个模型。我的准确率很高，但是现在我想给一个句子然后我想看看它的情绪。这是我的代码：

# data Analysis
import pandas as pd

# data Preprocessing and Feature Engineering
from textblob import TextBlob
import re
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer

# Model Selection and Validation
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
import joblib

import warnings
import mlflow

warnings.filterwarnings("ignore")

train_tweets = pd.read_csv('data/train.csv')

tweets = train_tweets.tweet.values
labels = train_tweets.label.values

processed_features = []

for sentence in range(0, len(tweets)):
    # Remove all the special characters
    processed_feature = re.sub(r'\W', ' ', str(tweets[sentence]))

    # remove all single characters
    processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature)

    # Remove single characters from the start
    processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature)

    # Substituting multiple spaces with single space
    processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I)

    # Removing prefixed 'b'
    processed_feature = re.sub(r'^b\s+', '', processed_feature)

    # Converting to Lowercase
    processed_feature = processed_feature.lower()

    processed_features.append(processed_feature)


vectorizer = TfidfVectorizer(max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))
processed_features = vectorizer.fit_transform(processed_features).toarray()

X_train, X_test, y_train, y_test = train_test_split(processed_features, labels, test_size=0.2, random_state=0)

text_classifier = MultinomialNB()
text_classifier.fit(X_train, y_train)

predictions = text_classifier.predict(X_test)

print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))
print(accuracy_score(y_test, predictions))


joblib.dump(text_classifier, 'model.pkl')

如您所见，我正在保存我的模型。现在，我想给出这样的输入：

new_sentence = "I am very happy today"
model.predict(new_sentence)

我想看到这样的输出：

sentence = "I am very happy today"
sentiment = Positive

我怎样才能做到这一点？

标签： machine-learningscikit-learnnlpsentiment-analysis

解决方案

首先，将预处理放在一个函数中：

def preproc(tweets):
    processed_features = []

    for sentence in range(0, len(tweets)):
        # Remove all the special characters
        processed_feature = re.sub(r'\W', ' ', str(tweets[sentence]))

        # remove all single characters
        processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature)

        # Remove single characters from the start
        processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature)

        # Substituting multiple spaces with single space
        processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I)

        # Removing prefixed 'b'
        processed_feature = re.sub(r'^b\s+', '', processed_feature)

        # Converting to Lowercase
        processed_feature = processed_feature.lower()

        processed_features.append(processed_feature)

    return processed_features

processed_features = preproc(tweets)
vectorizer = TfidfVectorizer(max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))
processed_features = vectorizer.fit_transform(processed_features).toarray()

然后使用它来预处理测试字符串并将其提供给分类器transform：

# feeding two 1-sentence tweets:
test = preproc([["I hate this book."], ["I love this movie."]])
predictions = text_classifier.predict(vectorizer.transform(test).toarray())
print(predictions)

现在，根据您在数据集中拥有的标签以及train_tweets.label.values编码方式，您将获得可以解析为字符串的不同输出。例如，如果数据集中的标签编码为 1=positive 和 0=negative，您可能会得到 [0,1]。

推荐阅读