Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chapter 5: can't reproduce "often end up with more than 95% accuracy in less than 10 epochs" #17

Open
jeffhgs opened this issue Feb 21, 2019 · 6 comments

Comments

@jeffhgs
Copy link

jeffhgs commented Feb 21, 2019

Chapter 5 reads:

But it’s noteworthy to observe that you often end up with more than 95% accuracy in less than 10 epochs.

However, I don't share this experience.

I have captured some scenarios in a branch.

https://github.com/jeffhgs/deep_learning_and_the_game_of_go/blob/test_nn_chapter_5/code/dlgo/nn/test_nn.py

You can run as:

(cd code/dlgo/nn && python test_nn.py)

The program downsamples mnist train and test data, train, repeats, and then computes summary statistics.

The program uses the unittest framework for regression testing, and it also emits json documents between scenarios for offline analysis.

The program depends on the incomplete branch I made for #12.

The typical fitting accuracy I get for the full mnist dataset is around 57%.

For purposes of helping your reproduction, I am holding back all inessential changes, including fix for performance related #11.

I tried to run all the configurations listed in the test, but unfortunately my job just ran past the 4h timeout I assigned it, so I'll have to up it and try again. I will attach the 4 instance hours worth of data I have.

This is my configuration:

host: AWS t2.large
OS: ubuntu
Python: 3.6.8
@jeffhgs
Copy link
Author

jeffhgs commented Feb 21, 2019

@jeffhgs
Copy link
Author

jeffhgs commented Feb 23, 2019

Data from a longer run. Note this timed out also, so doesn't end cleanly. But you can clearly see the fit measurements. Each full data set run just under 2h for 10 epochs.

test_nn_jeffhgs_2019-02-20_try3.txt

@maxpumperla
Copy link
Owner

that's pretty strange. last time I evaluated this, I certainly ended up with 95% plus, most of the time. I mean, not that it really matters, the lesson is that the algo learns. But I get your frustration, sorry for the inconvenience.

My suspicion is that I used a different learning rate altogether for my experiments.

@arisliang
Copy link

arisliang commented Oct 13, 2021

For me, the network doesn't train most of the time.

image

@arisliang
Copy link

Shallower and smaller network architecture, and dense weight initial value scaled with 0.1 worked better for me.

image

@Rock-New
Copy link

I met the same problem. What can I do to solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants