New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question] The end-to-end generation speed and W4A4 #16

Open

aur61 opened this issue Oct 13, 2024 · 0 comments

aur61 commented Oct 13, 2024 •

edited

Loading

Great job, starred! I do have a few questions:

Did you test the e2e generation speed, specifically in terms of tokens/second or the latency of the first token?
For the W4A4, the speedup is about 1. Could you share the reason behind this, and is there any potential for improvement?
For the W4A4, did you compare it with fp16 or bf16?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment