时间戳

首页 > 解决方案 > HuggingFace BPE 培训师错误 - 培训分词器

python - HuggingFace BPE 培训师错误 - 培训分词器

问题描述

我正在尝试使用可迭代而不是来自文件来训练 ByteLevelBPETokenizer。当我实例化训练器时，一定有什么我做错了，但我不知道它是什么。当我尝试使用我的数据集（来自 Kaggle 的服装数据）+ BpeTrainer 训练标记器时，出现错误。

**TypeError**: 'tokenizers.trainers.BpeTrainer' object cannot be interpreted as an integer

我正在使用 Colab

第 1 步：安装标记器并下载 Kaggle 数据

!pip install tokenizers

# Download clothing data from Kaggle
# https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews/version/1?select=Womens+Clothing+E-Commerce+Reviews.csv

第二步：上传文件

# use colab file upload
from google.colab import files
uploaded = files.upload()

第 3 步：清理数据（删除浮点数）并运行训练器

import io
import pandas as pd  

# convert the csv to a dataframe so it can be parsed
data = io.BytesIO(uploaded['clothing_dataset.csv']) 
df = pd.read_csv(data)

# convert the review text to a list so it can be passed as iterable to tokenizer
clothing_data = df['Review Text'].to_list()

# Remove float values from the data
clean_data =  []    
for item in clothing_data:
  if type(item) != float:
    clean_data.append(item)   


from tokenizers import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing
from tokenizers import trainers, pre_tokenizers
from tokenizers.trainers import BpeTrainer
from pathlib import Path


# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer(lowercase=True)

# Intantiate BpeTrainer
trainer = BpeTrainer(
    vocab_size=20000,
    min_frequence = 2,
    show_progress=True,
    special_tokens=["<s>","<pad>","</s>","<unk>","<mask>"],)

# Train the tokenizer
tokenizer.train_from_iterator(clean_data, trainer)

错误 - 我可以看到培训师是 BpeTrainer 类型。

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-103-7738a7becb0e> in <module>()
     34 
     35 # Train the tokenizer
---> 36 tokenizer.train_from_iterator(clean_data, trainer)

/usr/local/lib/python3.7/dist-packages/tokenizers/implementations/byte_level_bpe.py in train_from_iterator(self, iterator, vocab_size, min_frequency, show_progress, special_tokens)
    119             show_progress=show_progress,
    120             special_tokens=special_tokens,
--> 121             initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
    122         )
    123         self._tokenizer.train_from_iterator(iterator, trainer=trainer)

TypeError: 'tokenizers.trainers.BpeTrainer' object cannot be interpreted as an integer

有趣的注意：如果我设置输入 trainer=trainer 我得到这个

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-104-64737f948e6d> in <module>()
     34 
     35 # Train the tokenizer
---> 36 tokenizer.train_from_iterator(clean_data, trainer=trainer)

TypeError: train_from_iterator() got an unexpected keyword argument 'trainer'

标签： pythonnlpgoogle-colaboratoryhuggingface-tokenizers

解决方案

我以前没有使用train_from_iterator过，但是查看HF 文档似乎您应该使用生成器功能。所以像：

def clothing_generator():
  for item in clothing_data:
    if type(item) != float:
      yield item

其次是：

tokenizer.train_from_iterator(clothing_generator(), trainer)

可能有帮助？

推荐阅读