Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MODULE] - Tiktoken Token Counter #359

Open
LeonardPuettmannKern opened this issue Sep 28, 2023 · 0 comments
Open

[MODULE] - Tiktoken Token Counter #359

LeonardPuettmannKern opened this issue Sep 28, 2023 · 0 comments
Labels
cognition enhancement New feature or request

Comments

@LeonardPuettmannKern
Copy link
Contributor

Please describe the module you would like to add to bricks
Similar to #347 but returning the amount of tokens instead of classes. See here: https://platform.openai.com/tokenizer

Do you already have an implementation?
Tiktoken has a basic implementation in their docs, e.g. for gpt-3.5.-turbo. We just need to calculate the number of tokens of a text input.

Additional context
Also use this to calculate the complexity of a paragraph for RAG; for instance, > 1000 tokens means a high complexity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cognition enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant