You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, thank you for creating the ChatGPTSwift library. I noticed that the tokenizer currently used is the BPE tokenizer for ChatGPT-3, which is different from the Unigram language model tokenizer used by GPT-3.5.
Since we need to manually count the used tokens in stream mode, I was wondering if there is a plan to implement the GPT-3.5 tokenizer in the ChatGPTSwift library.
Thank you for your consideration.
The text was updated successfully, but these errors were encountered:
Hello, thank you for creating the ChatGPTSwift library. I noticed that the tokenizer currently used is the BPE tokenizer for ChatGPT-3, which is different from the Unigram language model tokenizer used by GPT-3.5.
Since we need to manually count the used tokens in stream mode, I was wondering if there is a plan to implement the GPT-3.5 tokenizer in the ChatGPTSwift library.
Thank you for your consideration.
The text was updated successfully, but these errors were encountered: