nlp - huggingface 的 ReformerForMaskedLM 配置问题
问题描述
我正在尝试将所有拥抱脸的 ...ForMaskedLM 传递给 FitBert 模型以完成填空任务,并查看哪个预训练在我准备的数据上产生最佳结果。但是在改革者模块中,我有这个错误说我需要做'config.is_decoder = False'但我真的不明白这意味着什么(这是我第一次使用huggingface)。我试图将 ReformerConfig(is_decoder=False) 传递给模型,但仍然得到相同的错误。我怎样才能解决这个问题?
我的代码:
pretrained_weights = ['google/reformer-crime-and-punishment',
'google/reformer-enwik8']
configurations = ReformerConfig(is_decoder=False)
for weight in pretrained_weights:
print(weight)
model = ReformerForMaskedLM(configurations).from_pretrained(weight)
tokenizer = ReformerTokenizer.from_pretrained(weight)
fb = FitBert(model=model, tokenizer=tokenizer)
predicts = []
for _, row in df.iterrows():
predicts.append(fb.rank(row['question'], options=[row['1'], row['2'], row['3'], row['4']])[0])
print(weight,':', np.sum(df.anwser==predicts) / df.shape[0])
错误:
AssertionError Traceback (most recent call last)
<ipython-input-5-a6016e0015ba> in <module>()
4 for weight in pretrained_weights:
5 print(weight)
----> 6 model = ReformerForMaskedLM(configurations).from_pretrained(weight)
7 tokenizer = ReformerTokenizer.from_pretrained(weight)
8 fb = FitBert(model=model, tokenizer=tokenizer)
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
1032
1033 # Instantiate model.
-> 1034 model = cls(config, *model_args, **model_kwargs)
1035
1036 if state_dict is None and not from_tf:
/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in __init__(self, config)
2304 assert (
2305 not config.is_decoder
-> 2306 ), "If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention."
2307 self.reformer = ReformerModel(config)
2308 self.lm_head = ReformerOnlyLMHead(config)
AssertionError: If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.
解决方案
您可以通过单独加载模型配置并将其作为from_pretrained()
方法的参数提供来覆盖某些模型配置。这将确保您使用正确的模型配置和您所做的更改:
from transformers import ReformerConfig, ReformerForMaskedLM
config = ReformerConfig.from_pretrained('google/reformer-crime-and-punishment')
print(config.is_decoder)
config.is_decoder=False
print(config.is_decoder)
model = ReformerForMaskedLM.from_pretrained('google/reformer-crime-and-punishment', config=config)
输出:
True
False
推荐阅读
- javascript - express-validator isAfter 总是假的
- c# - 如何将安装程序单选按钮对话框中的输入用作文件系统 DefaultLocation 的一部分的变量
- python - 有没有办法错开或躲避多个 matplotlib.plot 误差线?
- java - Android File.listFiles() 返回 null
- python - 我只能在python中将某些变量设置为true时记录吗?
- c++ - 我的代码无法将输入验证为整数
- javascript - 如何在JS中存储链接并通过html动态打开?
- ios - 观察者类没有收到通知
- sql-server - SQL Server、LIKE 与 RTRIM 性能
- php - 在登陆 URL 中通过修剪 PHP $_GET 获取哈希