Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix bug in python benchmark script #1206

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

thevishalagarwal
Copy link
Contributor

@thevishalagarwal thevishalagarwal commented Jan 29, 2025

Bug: if we use random token ids (with flag --use_random_token), decoding and encoding generates a different set of tokens which is not equal to the original set. This changes the number of prompt tokens and generates incorrect result during benchmarking. e.g.

original_tokens = np.random.randint(100, size=(1, 50))
prompt = tokenizer.decode(original_tokens )
new_tokens = tokenizer.encode(prompt)

Earlier the number of tokens was 50 but in new_tokens it may not be 50.

I have removed the step of encoding the prompt again to solve for this but the tokenization latency also need to be removed.
IMO, it is not an important metric as tokenizer is never a bottleneck and this easily resolves the above problem/bug.

Alternatively, if we do not use the --use_random_token, then the default way of generating tokens/prompt is very slow.

@thevishalagarwal
Copy link
Contributor Author

@baijumeswani Can you please review this? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant