时间戳

首页 > 解决方案 > 情绪分析，朴素贝叶斯精度

python - 情绪分析，朴素贝叶斯精度

问题描述

我正在尝试为推文的情感分类形成一个朴素贝叶斯分类器脚本。我将我的整个代码粘贴在这里，因为我知道如果我不这样做，我会下地狱。所以我基本上使用 NLTK 的语料库作为训练数据，然后我抓取了一些推文作为测试数据。我对它们进行预处理并提取一袋单词。分类器训练没有问题，当我执行以下操作时

print(classifier.classify(bag_of_words('This is magnificent')))

它正确输出'pos'。

现在我的问题是如何使用 ntlk.util 精度计算精度。我愿意

print(nltk.classify.accuracy(classifier, proc_set))

我收到以下错误：

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-   packages/nltk/classify/util.py", line 87, in accuracy
  results = classifier.classify_many([fs for (fs, l) in gold])
  AttributeError: 'NaiveBayesClassifier' object has no attribute 'classify_many'

我也试过这个

test_set_final=[]
for tweet in proc_test:
test_set_final.append((bag_of_words(tweet),   classifier.classify(bag_of_words(tweet))))

print(nltk.classify.accuracy(classifier, test_set_final))

我得到了同样的错误

print(nltk.classify.accuracy(classifier, test_set_final))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/nltk/classify/util.py", line 87, in accuracy
results = classifier.classify_many([fs for (fs, l) in gold])
AttributeError: 'NaiveBayesClassifier' object has no attribute 'classify_many'

我是 100% 我错过了一些对机器学习者来说非常明显的东西。我想但是已经 3 天了，我正在慢慢失去理智，所以任何帮助将不胜感激。

代码 ->

import nltk
import ast
import string
import re
import csv
import textblob
import pandas as pd
import numpy as np
import itertools
from textblob import TextBlob
from textblob import Word
from textblob.classifiers import NaiveBayesClassifier
from nltk.corpus import twitter_samples
from nltk.corpus import stopwords
from nltk.corpus import wordnet as wd
from nltk.tokenize import word_tokenize
from sklearn.feature_extraction.text import CountVectorizer
from random import shuffle
from nltk.classify.util import accuracy
from autocorrect import spell

stopwords = stopwords.words('english')
lemmatizer = nltk.WordNetLemmatizer().lemmatize
punct=['"','$','%','&','\',''','(',')','+',',','-     ','.','/',':',';','<','=','>','@','[','\',','^','_','`','{','|','}','~']

emoticons_happy = set([
':)', ';)', ':o)', ':]', ':3', ':c)', ':>', '=]', '8)', '=)', ':}',
':^)', ':-D', ':D', ': D','8-D', '8D', 'x-D', 'xD', 'X-D', 'XD', '=-D', '=D',
'=-3', '=3', ':-))', ':-)', ":'-)", ":')", ':*', ':^*', '>:P', ':-P', ':P', 'X-P',
'x-p', 'xp', 'XP', ':-p', ':p', '=p', ':-b', ':b', '>:)', '>;)', '>:-)',
'<3',':*', ':p'
])

emoticons_sad = set([
':L', ':-/', '>:/', ':S', '>:[', ':@', ':-(', ':[', ':-||', '=L', ':<',
':-[', ':-<', '=\\', '=/', '>:(', ':-(', '>.<', ":'-(", ":'(", ':\\', ':-c',
':c', ':{', '>:\\', ';('
])
emoticons = emoticons_happy.union(emoticons_sad)


def pre_process(tweet):

    tweet = re.sub(r'https?:\/\/.*[\r\n]*', '', tweet)

    tweet = re.sub(r'#', '', tweet)

    tweet=''.join([i for i in tweet if not i.isdigit()])

    tweet=re.sub(r'([.,/#!$%^&*;:{}=_`~-])([.,/#!$%^&*;:{}=_`~-]+)\1+', r'\1',tweet)

    tweet = re.sub(r'@[A-Za-z0-9]+', '', tweet)

    tweet=''.join([i for i in tweet if i not in emoticons])

    tweet=''.join([i for i in tweet if i not in punct])

    tweet=' '.join([i for i in tweet.split() if i not in stopwords])

    tweet=tweet.lower()

    tweet=lemmatize(tweet)

    return tweet

def get_wordnet_pos(treebank_tag):
    if treebank_tag.startswith('J'):
        return wd.ADJ
    elif treebank_tag.startswith('V'):
        return wd.VERB
    elif treebank_tag.startswith('N'):
        return wd.NOUN
    elif treebank_tag.startswith('R'):
        return wd.ADV
    else:
        return wd.NOUN

def lemmatize(tt):
    pos = nltk.pos_tag(nltk.word_tokenize(tt))
    lemm = [lemmatizer(sw[0], get_wordnet_pos(sw[1])) for sw in pos]
    sentence= ' '.join([i for i in lemm])

    return sentence


test_tweets=[]
file=open('scraped_tweets.csv', 'r')
reader = csv.reader(file)
for line in reader:
    line=line[1]
    test_tweets.append(line)

pos_tweets = twitter_samples.strings('positive_tweets.json')
neg_tweets = twitter_samples.strings('negative_tweets.json')



proc_train_pos=[]
for tweet in pos_tweets:
    proc_train_pos.append(pre_process(tweet))
proc_train_neg=[]
for tweet in neg_tweets:
    proc_train_neg.append(pre_process(tweet))
proc_test=[]
for tweet in test_tweets:
    proc_test.append(pre_process(tweet))


def bag_of_words(tweet):
    words_dictionary = dict([word, True] for word in tweet.split())    
    return words_dictionary

pos_tweets_set = []
for tweet in proc_train_pos:
    pos_tweets_set.append((bag_of_words(tweet), 'pos'))    

neg_tweets_set = []
for tweet in proc_train_neg:
    neg_tweets_set.append((bag_of_words(tweet), 'neg'))

shuffle(pos_tweets_set)
shuffle(neg_tweets_set)
train_set = pos_tweets_set+neg_tweets_set

classifier = NaiveBayesClassifier(train_set)
print('Training is done')

#print(classifier.classify(bag_of_words('This is magnificent'))) #output 'pos'

print(nltk.classify.accuracy(classifier, proc_set))

标签： pythonsentiment-analysisnaivebayes

解决方案

好吧，正如错误消息所说，您尝试使用的分类器 ( NaiveBayesClassifier) 没有classify_many该nltk.classify.util.accuracy函数所需的方法。

（参考：https ://www.nltk.org/_modules/nltk/classify/naivebayes.html ）

现在，这看起来像是一个 NLTK 错误，但您可以自己轻松获得答案：

from sklearn.metrics import accuracy_score

y_predicted = [classifier.classify(x) for x in proc_set]

accuracy = accuracy_score(y_true, y_predicted)

y_true与输入相对应的情绪值在哪里proc_set（不过，我没有看到您在上面显示的代码中实际创建）。

希望有帮助。

编辑：

或者，不使用sklearnaccuracy 函数，而是使用纯 Python：

hits = [yp == yt for yp, yt in zip(y_predicted, y_true)]

accuracy = sum(hits)/len(hits)

推荐阅读