-
Notifications
You must be signed in to change notification settings - Fork 451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make garbage collection interval configurable #533
Conversation
olmo/config.py
Outdated
@@ -987,6 +987,11 @@ class TrainConfig(BaseConfig): | |||
How often to log to the console. | |||
""" | |||
|
|||
gen1_gc_interval: int = 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: it might be nice to provide an easy way to turn this off (e.g. setting it to None)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! 989f799
(I assumed you mean disabling manual collection, i.e. not messing with it at all)
@@ -987,9 +987,10 @@ class TrainConfig(BaseConfig): | |||
How often to log to the console. | |||
""" | |||
|
|||
gen1_gc_interval: int = 1 | |||
gen1_gc_interval: Optional[int] = 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just checking, we want the default behavior to be messing with GC?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I think so. There might be a better default interval for smaller models, but this was a huge improvement for the 70B.
No description provided.