Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to restart a training session that has completed #198

Open
bwanab opened this issue Jul 12, 2023 · 3 comments
Open

How to restart a training session that has completed #198

bwanab opened this issue Jul 12, 2023 · 3 comments

Comments

@bwanab
Copy link

bwanab commented Jul 12, 2023

I've written a reversi (Othello) game interface. For params.jl, I used a modified version of the mancala params.jl. For my first run, I set num_iters to 5. The results are somewhat encouraging, so I'd like to continue with num_iters set much higher, but when I invoke Scipts.train(exp), I get the message: "Training completed".

I tried changing num_iters in params.jl and in the session params.json, but same result in both cases.

Is there a way to get the training to resume from the point it has reached, or do I just have to delete the session and start over?

@jonathan-laurent
Copy link
Owner

The reason is that when loading a session, hyperparameters are loaded from "params.data" and not "params.json" (you can look at the source in session.jl). The former is not written in human-readable format. The latter is human-readable but not used to load sessions.

I agree this is not very elegant. The reason I had to do this is that last I checked, JSON3.jl was pretty good to at dumping complex data into JSON without much boilerplate but not as good at parsing back this data. See this issue.

Don't hesitate to contribute an improvement here.

@bwanab
Copy link
Author

bwanab commented Jul 13, 2023

Thanks. I looked at the code and see what you mean. But, from your answer I take it that there's no other reason why one couldn't restart a training session assuming that some reasonably elegant mechanism exited to set the value of env.num_iters to a higher value.

@smart-fr
Copy link

smart-fr commented Jul 13, 2023

There is a way, though not very elegant: #118 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants