machine-learning - 文本分类时的多个输入参数 - Scikit learn

Question

我是机器学习的新手。我正在尝试进行一些文本分类。'CleanDesc' 有文本句子。'output' 有相应的输出。最初我尝试使用一个输入参数，即文本字符串（newMerged.cleanDesc）和一个输出参数（newMerged.output）

finaldata = newMerged[['id','CleanDesc','type','output']]

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(newMerged.CleanDesc)

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

clf = MultinomialNB().fit(X_train_tfidf, newMerged.output)    
testdata = newMerged.ix[1:200]
X_test_counts = count_vect.transform(testdata.CleanDesc)
X_test_tfidf = tfidf_transformer.transform(X_test_counts)

predicted = clf.predict(X_new_tfidf)

这工作正常。但是准确率很低。我想再包含一个参数（newMerged.type）作为输入，以及尝试改进它的文本。我可以这样做吗？我该怎么做。newMerged.type 不是文本。它只是一个像“HT”这样的两个字符串。我尝试按以下方式进行操作，但失败了，

finaldata = newMerged[['id','CleanDesc','type','output']]

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(newMerged.CleanDesc)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

clf = MultinomialNB().fit([[X_train_tfidf,newMerged.type]], 
newMerged.output)    
testdata = newMerged.ix[1:200]
X_test_counts = count_vect.transform(testdata.CleanDesc)
X_test_tfidf = tfidf_transformer.transform(X_test_counts)

predicted = clf.predict([[X_new_tfidf, testdata.type]])

标签： machine-learningscikit-learnnlptext-classification

Answer 1

您必须使用 sicpy 中的 hstack 将数组附加到稀疏矩阵。

试试这个！

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import LabelBinarizer
from scipy.sparse import hstack
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())

print(X.shape)

#

['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
(4, 9)

您需要对分类变量进行编码。

cat_varia= ['s','ut','ss','ss']
lb=LabelBinarizer()
feature2=lb.fit_transform(cat_varia)

appended_X = hstack((X, feature2))

import pandas as pd
pd.DataFrame(appended_X.toarray())

#

    0   1   2   3   4   5   6   7   8   9   10  11
0   0.000000    0.469791    0.580286    0.384085    0.000000    0.000000    0.384085    0.000000    0.384085    1.0 0.0 0.0
1   0.000000    0.687624    0.000000    0.281089    0.000000    0.538648    0.281089    0.000000    0.281089    0.0 0.0 1.0
2   0.511849    0.000000    0.000000    0.267104    0.511849    0.000000    0.267104    0.511849    0.267104    0.0 1.0 0.0
3   0.000000    0.469791    0.580286    0.384085    0.000000    0.000000    0.384085    0.000000    0.384085    0.0 1.0 0.0