时间戳

首页 > 解决方案 > word2vec 错误：“_Token”对象不可迭代

python - word2vec 错误：“_Token”对象不可迭代

问题描述

我正在尝试将句子列表按顺序输入 gensim.models.Word2Vec，但它会生成 TypeError: '_Token' object is not iterable。我应该怎么办？

    embedding_model= Word2Vec()
    for index, sentence_list in df.iterrows():
        embedding_model = Word2Vec(sentence_list, size=100, window=5, min_count=2, workers=2)
        embedding_model.train(tokenized_contents, total_examples=len(tsentence_list), epochs=10)

标签： pythongensimword2vec

解决方案

感谢你的快速回复。错误是我放置纯句子而不是标记列表的错误。但是，我仍然在为 Word2Vec 的顺序馈送而苦苦挣扎。这是我的示例数据、代码和错误：

tokenized_contents: ['こんにちは', '!', '掲示', '板', 'が', 'でき', 'まし', 'た', 'ね', '!', 'これ', 'から', 'も', 'よろしく', 'お', '願い', 'し', 'ます', '!']

embedding_model= Word2Vec()
for index, tokenized_contents in df.iterrows():
    embedding_model = Word2Vec(tokenized_contents, size=100, window=5, min_count=1, workers=4)
    embedding_model.build_vocab(tokenized_contents)

embedding_model.train(tokenized_contents, total_examples=len(tokenized_contents), epochs=10)

Error Msg:
Traceback (most recent call last):

    embedding_model.build_vocab(tokenized_contents)
  File "/anaconda3/envs/japan/lib/python3.6/site-packages/gensim/models/base_any2vec.py", line 484, in build_vocab
    trim_rule=trim_rule, **kwargs)
  File "/anaconda3/envs/japan/lib/python3.6/site-packages/gensim/models/word2vec.py", line 1318, in prepare_vocab
    self.sort_vocab(wv)
  File "/anaconda3/envs/japan/lib/python3.6/site-packages/gensim/models/word2vec.py", line 1184, in sort_vocab
    raise RuntimeError("cannot sort vocabulary after model weights already initialized.")
RuntimeError: cannot sort vocabulary after model weights already initialized.
'''

推荐阅读