detect corrupted state in case app dies during middle of shuffile_migrate

[adammoody]

Currently, shuffile_migrate overwrites files in place when moving data. This is useful to conserve storage space, and it is necessary in the case of only being able to fit a single dataset in cache at a time.

However, in introduces a significant risk in that one has corrupt files during the time that shuffile_migrate is executing. If an application dies during that time, the files may be left in a corrupt state.

We should add logic to detect this condition and return an error to notify the user. If using er, the er library does its own tracking to detect this condition. However, we should add something for apps that may be using shuffile directly without going through the er component.

[adammoody]

One could set a CORRUPT flag just before starting the migrate in the shuffile metadata file. After completing, one could update this flag to VALID. If when calling migrate() one finds the flag is already set to CORRUPT, then one must assume the files were left in a corrupt state.