Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Record batch time #278

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

isamu-isozaki
Copy link
Contributor

@isamu-isozaki isamu-isozaki commented Feb 10, 2024

This piece of code helped me identify how much time is spent in the data processing stage, the forward pass, and backward pass so that I know where to optimize.
The main steps blocking this pr will be

  • Figure out how to log with trainer since I think the def log function is not present currently because of refactoring

@isamu-isozaki isamu-isozaki marked this pull request as draft February 10, 2024 20:25
piercus added a commit to piercus/refiners that referenced this pull request Feb 13, 2024
@isamu-isozaki isamu-isozaki marked this pull request as ready for review February 16, 2024 17:20
@isamu-isozaki
Copy link
Contributor Author

I'm thinking if this can become a callback

Copy link
Member

@limiteinductive limiteinductive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this is gonna be useful! Maybe you can take inspiration on how the TrainingClock Callback is implemented? Let's try to pollute as little as possible the core of the Trainer

@isamu-isozaki isamu-isozaki marked this pull request as draft February 20, 2024 18:23
@isamu-isozaki isamu-isozaki marked this pull request as ready for review February 21, 2024 03:33
@isamu-isozaki
Copy link
Contributor Author

@limiteinductive do you think I should make a timer config or merge timer with clock?

@limiteinductive
Copy link
Member

@limiteinductive do you think I should make a timer config or merge timer with clock?

I think it makes sense to merge with Clock and extend its config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants