Fix tokenizer mismatch bug between model and tokenizer for THUDM/glm-…

[darkSuperman]

Fix tokenizer mismatch bug between model and tokenizer for THUDM/glm-4-9b example

[LaurentMazare]

Did you try out the change? There doesn't seem to be a tokenizer.json file in the repo that you've switched to as far as I can tell.https://huggingface.co/THUDM/glm-4-9b/tree/main

[darkSuperman]

Sorry I misread that. Also I found this branch used natively within Transformers, and it also provides a tokenizer.json, but loading the model requires some changes Are you interested in using this branch to modify the glm4 example? https://huggingface.co/THUDM/glm-4-9b/tree/refs%2Fpr%2F15

[LaurentMazare]

The main reason why this model uses the tokenizer from codegeex4 is that the two tokenizers should be identical. When you look at the two sentencepiece model they have the same hash, here and here so I don't think the current version is actually fine or maybe I'm missing something here?

[darkSuperman]

Yes, they are the same. Also, when I was running the inference, I encountered some problems and it seemed that I could not finish the inference. I am trying it and I will give you feedback if I have more information.