-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chore fill anemoi models #1
Conversation
Allows Distributed Data Parallel Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Simon Lang <[email protected]> Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: Mihai Alexe <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
Encompasses encoder, processor and decoder structure. Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: Simon Lang <[email protected]> Co-authored-by: mihai.alexe <[email protected]>
Dynamic data handling and indexing Co-authored-by: Florian Pinault <[email protected]> Co-authored-by: Baudouin Raoult <[email protected]> Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: mihai.alexe <[email protected]> Co-authored-by: Simon Lang <[email protected]>
Pytorch model interface Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Simon Lang <[email protected]> Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: Mihai Alexe <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: Mihai Alexe <[email protected] Co-authored-by: Florian Pinault <[email protected]> Co-authored-by: Baudouin Raoult <[email protected]>
test config missing Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Simon Lang <[email protected]> Co-authored-by: Helen Theissen <[email protected]>
--------- Co-authored-by: Steffen Tietsche <[email protected]> Co-authored-by: Helen Theissen <[email protected]> Co-authored-by: Sara Hahner <[email protected]> Co-authored-by: Mariana Clare <[email protected]> Reviewed-by: Mariana Clare <[email protected]>
Distributed folder has no init. |
Co-authored-by: Matthew Chantry <[email protected]>
Some tests fail with flash-attention, I'm not sure that should be a blocker, but just an issue to fix. |
t, "(batch grid) (heads vars) -> batch heads grid vars", batch=batch_size, heads=self.num_heads | ||
), | ||
(query, key, value), | ||
query, key, value = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this being refactored?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Readability mostly.
query, key, value = map( | ||
lambda t: einops.rearrange(t, "batch heads grid vars -> batch grid heads vars"), | ||
(query, key, value), | ||
query, key, value = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above, what's the pro of the refactor?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mostly readability
Initial implementation of anemoi-models components including the following modules: