Rollback failed append and continue

[forman]

nc2zarr should be able to rollback the last append on error and allow for appending remaining data in another call.

Example: After 5.5 hours of processing the generation of my CCI SST dataset suddenly failed, probably caused by a temporary problem with the JASMIN Object Storage API.

...
2021-06-03 20:51:52,984: INFO: nc2zarr: Opening done: took 0.51 seconds
2021-06-03 20:51:52,984: DEBUG: nc2zarr: Appending dataset...
2021-06-03 21:11:03,252: INFO: nc2zarr: Appending dataset done: took 1,150.27 seconds
2021-06-03 21:11:03,253: INFO: nc2zarr: Processing input 18 of 35: /work/scratch-nopw/esacci/sst/ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1999-v02.0-fv01.0.zarr
2021-06-03 21:11:03,253: DEBUG: nc2zarr: Opening...
2021-06-03 21:11:03,398: INFO: nc2zarr: Opening done: took 0.14 seconds
2021-06-03 21:11:03,398: DEBUG: nc2zarr: Appending dataset...
2021-06-03 21:27:52,166: ERROR: nc2zarr: Appending dataset failed: took 1,008.77 seconds
2021-06-03 21:27:52,181: ERROR: nc2zarr: Converting failed: took 19,698.26 seconds
Traceback (most recent call last):
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/s3fs/core.py", line 234, in _call_s3
    return await method(**additional_kwargs)
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchUpload: An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apps/slurm/spool/slurmd/job56316565/slurm_script", line 33, in <module>
    sys.exit(load_entry_point('nc2zarr', 'console_scripts', 'nc2zarr')())
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  ...
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/fsspec/asyn.py", line 222, in _pipe
    await asyncio.gather(
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/s3fs/core.py", line 777, in _pipe_file
    await self._call_s3(
  File "/home/users/forman/miniconda3/envs/xcube/lib/python3.8/site-packages/s3fs/core.py", line 252, in _call_s3
    raise translate_boto_error(err) from err
FileNotFoundError: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.