python - 停用词与 sklearn CountVectorizer 中的预处理不一致

Question

我一直在尝试使用 sklearn 库中的 Count Vectorizer，但是我收到警告说我使用的西班牙语停用词列表与预处理不一致。

是不是编码问题？？

stopwords.txt 文件每行有一个单词，例如：

一个埃尔洛斯...

custom_stop_words = []
with open( "stopwords.txt", "r" ,encoding='latin_1') as fin:
    for line in fin.readlines():
        custom_stop_words.append( line.strip() )
# note that we need to make it hashable
print("Stopword list has %d entries" % len(custom_stop_words) )


from sklearn.feature_extraction.text import CountVectorizer
# use a custom stopwords list, set the minimum term-document frequency to 20
vectorizer = CountVectorizer(stop_words = custom_stop_words, min_df = 2,strip_accents='unicode',encoding="latin-1")
A = vectorizer.fit_transform(df_final['visita'])
print( "Created %d X %d document-term matrix" % (A.shape[0], A.shape[1]) )

标签： pythonscikit-learntext-miningcountvectorizer