Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't crash with ZeroDivisionError if initial test ELBO is the best test ELBO #312

Merged
merged 1 commit into from
Apr 23, 2024

Conversation

alecw
Copy link
Contributor

@alecw alecw commented Dec 4, 2023

The --final-elbo-fail-fraction test checks that the final test ELBO isn't much worse than the best test ELBO. It checks the (distance between final test ELBO and best test ELBO)/(distance between initial test ELBO and best test ELBO).

This causes a ZeroDivisionError if best test ELBO == initial test ELBO.

Solution:
If --final-elbo-fail-fraction is enabled && final test ELBO < best test ELBO && initial test ELBO == best test ELBO
then training is considered to have failed.

@sjfleming
Copy link
Member

Hi @alecw , any chance you could rebase this on top of dev post #345 ? I think the test_elbo referred to here should now probably be model.loss['test']['elbo']

@alecw
Copy link
Contributor Author

alecw commented Apr 22, 2024

Hi Stephen, yes, fixed conflicts, addressed removed local variable and rebased. thanks!

@sjfleming
Copy link
Member

Thanks so much!

@sjfleming sjfleming merged commit a71e183 into broadinstitute:dev Apr 23, 2024
4 checks passed
@alecw alecw deleted the aw_retry_div_0 branch August 21, 2024 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants