We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
在将torch代码迁移到MindSpore框架的时候,用到了cybertron替代transformers。 原始代码是: from transformers import BertTokenizer model_name_or_path = "dmis-lab/biobert-base-cased-v1.1" tfm_tokenizer = BertTokenizer.from_pretrained(model_name_or_path) a = tfm_tokenizer("this is not ok",padding ="max_length", max_length=25, truncation=True, return_tensors="pt") 输出结果如下图:
from transformers import BertTokenizer
model_name_or_path = "dmis-lab/biobert-base-cased-v1.1"
tfm_tokenizer = BertTokenizer.from_pretrained(model_name_or_path)
a = tfm_tokenizer("this is not ok",padding ="max_length", max_length=25, truncation=True, return_tensors="pt")
对应的cybertron代码如下: import cybertron model_name_or_path = "dmis-lab/biobert-base-cased-v1.1" ms_tokenizer = cybertron.BertTokenizer.load(model_name_or_path) ms_tokenizer("this is not ok", padding="max_length", max_length=self.max_length, truncation=True, return_tensors="pt") 运行报错信息如下图:
import cybertron
ms_tokenizer = cybertron.BertTokenizer.load(model_name_or_path)
ms_tokenizer("this is not ok", padding="max_length", max_length=self.max_length, truncation=True, return_tensors="pt")
看了下cybertron的源代码,tokenizer的方法没有实现完全,想请作者帮忙修复一下这个问题,感谢
The text was updated successfully, but these errors were encountered:
lvyufeng
No branches or pull requests
在将torch代码迁移到MindSpore框架的时候,用到了cybertron替代transformers。
原始代码是:
from transformers import BertTokenizer
model_name_or_path = "dmis-lab/biobert-base-cased-v1.1"
tfm_tokenizer = BertTokenizer.from_pretrained(model_name_or_path)
a = tfm_tokenizer("this is not ok",padding ="max_length", max_length=25, truncation=True, return_tensors="pt")
输出结果如下图:
对应的cybertron代码如下:
import cybertron
model_name_or_path = "dmis-lab/biobert-base-cased-v1.1"
ms_tokenizer = cybertron.BertTokenizer.load(model_name_or_path)
ms_tokenizer("this is not ok", padding="max_length", max_length=self.max_length, truncation=True, return_tensors="pt")
运行报错信息如下图:
看了下cybertron的源代码,tokenizer的方法没有实现完全,想请作者帮忙修复一下这个问题,感谢
The text was updated successfully, but these errors were encountered: