-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Beaver fix; working towards another model #13
Conversation
Ziya works if you use I'll have results on HERM in ~10 minutes, and then I'll run it on pref sets. I can include these changes in our docker image when this is merged. |
@jacob-morrison on what GPU? Quantization should be exactly the same in the |
It should be, but it's clearly not based on my testing. I was running this on an A100 and I couldn't get it to successfully run on a single example with: batch_size=1, max_length=1024. After switching to direct_load, I could change back to: batch_size=32, max_length=2048, with no other changes. We also already use direct_load for all other models of the same size or bigger, and I'd imagine we'd run into the same issue for those if we tried to use the |
Good to know, I'll add something to the docs that the quantization doesn't always work right for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Jacob, can you try running this new Ziya model?