S3P - Checkpointing (e.g. log all copied files & option to skip files listed in a source file)

[mmuru-w]

It would be nice to have a checkpointing support in S3P while copying files. Currently, it is all or nothing and no indication of the start/last file copied. It would be nice to include them in the status summary at the end. Any suggestion how to get the processed file information from S3P?

[shanebdavis]

That's a great idea! It might not be simple to do, though. S3P is massively parallel, and if it fails partway through, it'll be in the middle of many simultaneous scans. The only reliable way to do it would be to log every file already copied and skip copying them on the next run.

The s3p sync command kind-of does this already. It'll quickly skip over files previously copied, and in many cases is just the ticket. However, I realize it's not quite semantically the same as re-running s3p cp and skipping previously copied files.

[RobQuistNL]

unfortunately the s3 sync command clears out the target folder for me and starts all over again :(