Don't reshuffle eval data each "epoch" #229
Conversation
olmo/util.py
Outdated
| if isinstance(dataloader.sampler, DistributedSampler): | ||
| if update_epoch_seed and isinstance(dataloader.sampler, DistributedSampler): | ||
| epoch = dataloader.sampler.epoch + 1 | ||
| dataloader.sampler.set_epoch(epoch) |
There was a problem hiding this comment.
I don't understand how skipping this makes sure it starts from scratch every epoch. I thought we'd have to reset the data loader somehow every time.
There was a problem hiding this comment.
Each validation loop through a dataset is a complete loop / epoch over the dataset (less the extra examples, since drop_last=True). The DistributedSampler reshuffles each epoch with a specific seed that's shared across all processes. That seed never changes unless you call DistributedSampler.set_epoch(). See https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler.
There was a problem hiding this comment.
But the fact that this is confusing, err kind of works by exploiting a nuance of the DistributedSampler, tells me that this probably isn't the best way.
There was a problem hiding this comment.
Do we still need cycle_through_epoch then? Or at least, do we still need this change?
Right |
Keep eval data order consistent across iterations. Add paths to metadata in
MemMapDatasetclass.