Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restarting from previous checkpoint #40

Open
jbrry opened this issue Oct 26, 2023 · 0 comments
Open

Restarting from previous checkpoint #40

jbrry opened this issue Oct 26, 2023 · 0 comments

Comments

@jbrry
Copy link

jbrry commented Oct 26, 2023

Hi, do you know what the best way to resume training from a previous checkpoint would be? Let's assume I am training for 100k steps but I have a 24-hour time limit, and I just have the following checkpoints available:

ls checkpoints/pretrain
vanilla_11081_12.0%.pth  vanilla_11081_25.0%.pth  vanilla_11081_50.0%.pth

Given that the generator and discriminator are instantiated as separated models, do we point them to the same .pth file? Also, I believe the .from_pretrained() method requires a single config.json so how do we merge the two configs if that is necessary?

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant