Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation does not always start at the end of an epoch #136

Open
mc4117 opened this issue Feb 14, 2025 · 2 comments
Open

Validation does not always start at the end of an epoch #136

mc4117 opened this issue Feb 14, 2025 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@mc4117
Copy link
Member

mc4117 commented Feb 14, 2025

What happened?

If training batches are limited and the total number of batches is not reached then validation does not start. This bug is fixed by setting train batched to null.

What are the steps to reproduce the bug?

Set dataloader.training.limit_batches > number of batches in dataset.

Version

0.3.3

Platform (OS and architecture)

All

Relevant log output

Accompanying data

No response

Organisation

No response

@mc4117 mc4117 added the bug Something isn't working label Feb 14, 2025
@Rilwan-Adewoyin
Copy link
Member

So i think this issue is more related to pytorch lightning not knowing the length of the dataset prior to training run starting.

If it does know the dataset length then regardless of what limit batches value is entered the val loop is not skipped and the end of validation plots are also not skipped

This entails implementing the len property on the Dataset class

@mc4117
Copy link
Member Author

mc4117 commented Feb 20, 2025

We are now working on implementing a length property on the dataset class

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: No status
Development

No branches or pull requests

4 participants