Logs are not complete and scrambled

[fvisin]

The logs of my jobs are missing some lines (I print one line every batch, approx 6 out of 7 are missing!) and sometimes the lines are scrambled (the outputs of different epochs are interleaved). Note that these are two separate issues, i.e., I checked that the missing lines are not just somewhere else in the log, they are actually missing.

It might have to do with progressbar2, but I have no evidence of this. The only reason for this claim is that there might be an incompatibility with progressbar rewriting multiple times the same line.

[mgermain]

I've never heard of this one. Might be related to progressbar2 which I've never tried with smartdispatch. I don't know if @MarcCote knows more about this issue. For now, can you tell us on which cluster you've seen this issue and do you have code to reproduce?

[fvisin]

I was using Helios. You can almost certaintly reproduce it with this code (I'm using the same main loop, which is where the progressbar is called):
https://github.com/fvisin/reseg

Thank you for investigating the issue!