时间戳

首页 > 解决方案 > 如何将单词转换为浮点值

python - 如何将单词转换为浮点值

问题描述

对于我的一个副项目，我正在尝试构建一个朴素贝叶斯模型，该模型可以根据标题检测一条新闻是否是假的。到目前为止，这是我的代码：

import numpy as np
import pandas as pd
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer

data = pd.read_csv("/Users/amanpuranik/Desktop/fake-news-detection/data.csv")
data = data[['Headline', "Label"]]
print(data)

x = data[["Headline"]]
y = data[["Label"]]

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.1, random_state=1)

tfidf_vectorizer=TfidfVectorizer(stop_words='english', max_df=0.7)

model = MultinomialNB()
model.fit(x_train, y_train)

当我运行它时，我收到一个错误，告诉我标题无法转换为浮点值。由于标题是由一堆单词组成的，我想知道我的下一步是什么，因为我不确定如何将一个单词转换为浮点数。

标签： pythonscikit-learnmultinomial

解决方案

如果我理解正确，您想使用第一个向量化文本TfidfVectorizer，然后尝试使用MultinomialNB模型对结果向量进行分类。我建议您将这两个步骤包装在一个pipeline中，以便更轻松地部署模型、交叉验证或添加更多步骤。

import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline

data = pd.DataFrame({'Headline': ['Are Lizard Immigrants Stealing our Oil???',
                                  'Trade Summit Proceeds As Planned'],
                     'Label': ['Fake', 'Real']})

print(data)

X = data[['Headline']]
y = data['Label']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, random_state = 1)

tfidf_vectorizer=TfidfVectorizer(stop_words = 'english')

model = MultinomialNB()

pipeline = Pipeline([('vectorizer', tfidf_vectorizer), ('classifier', model)])

pipeline.fit(X_train, y_train)

print(pipeline)

输出：

                                    Headline Label
0  Are Lizard Immigrants Stealing our Oil???  Fake
1           Trade Summit Proceeds As Planned  Real

Pipeline(memory=None,
         steps=[('vectorizer',
                 TfidfVectorizer(analyzer='word', binary=False,
                                 decode_error='strict',
                                 dtype=<class 'numpy.float64'>,
                                 encoding='utf-8', input='content',
                                 lowercase=True, max_df=1.0, max_features=None,
                                 min_df=1, ngram_range=(1, 1), norm='l2',
                                 preprocessor=None, smooth_idf=True,
                                 stop_words='english', strip_accents=None,
                                 sublinear_tf=False,
                                 token_pattern='(?u)\\b\\w\\w+\\b',
                                 tokenizer=None, use_idf=True,
                                 vocabulary=None)),
                ('classifier',
                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],
         verbose=False)

请注意，当我从数据框中提取代码时，我从代码中删除了内括号y，因为它应该是一维的。

推荐阅读