word-embedding - 在本地下载预训练的句子转换器模型

Question

我正在使用 SentenceTransformers 库（此处：https : //pypi.org/project/sentence-transformers/#pretrained-models）使用预训练模型 bert-base-nli-mean-tokens 创建句子的嵌入。我有一个应用程序将部署到无法访问 Internet 的设备上。到这里，已经回答了，如何保存模型在本地下载预训练的BERT模型。然而，我坚持从本地保存的路径加载保存的模型。

当我尝试使用上述技术保存模型时，这些是输出文件：

('/bert-base-nli-mean-tokens/tokenizer_config.json',
 '/bert-base-nli-mean-tokens/special_tokens_map.json',
 '/bert-base-nli-mean-tokens/vocab.txt',
 '/bert-base-nli-mean-tokens/added_tokens.json')

当我尝试将其加载到内存中时，使用

tokenizer = AutoTokenizer.from_pretrained(to_save_path)

我越来越

Can't load config for '/bert-base-nli-mean-tokens'. Make sure that:

- '/bert-base-nli-mean-tokens' is a correct model identifier listed on 'https://huggingface.co/models'

- or '/bert-base-nli-mean-tokens' is the correct path to a directory containing a config.json 

标签： word-embeddingbert-language-modelhuggingface-tokenizerssentence-transformers

Answer 1

您可以像这样下载和加载模型

from sentence_transformers import SentenceTransformer
modelPath = "local/path/to/model

model = SentenceTransformer('bert-base-nli-stsb-mean-tokens')
model.save(modelPath)
model = SentenceTransformer(modelPath)

这对我有用。您可以查看 SBERT 文档以了解 SentenceTransformer 类的模型详细信息 [此处][1]

[1]: https://www.sbert.net/docs/package_reference/SentenceTransformer.html#:~:text=class,Optional%5Bstr%5D%20%3D%20None )