Catch FES2014 FileNotFound error as Error Code > 1

[robbibt]

Intermittent failures to access the EFS are causing the following errors (below). If we catch these and instead raise an Error code > 1, this would cause Argo to retry these tiles automatically.

  |   | 2022-09-09 05:52:33 | 2022-09-08T19:52:33.492373797Z stderr F Error: exit status 1
-- | -- | -- | --
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.443744421Z stderr F   warnings.warn(
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.443717685Z stderr F /usr/local/lib/python3.8/dist-packages/geopandas/_compat.py:111: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.10.3-CAPI-1.16.1). Conversions between both will be slow.
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417600673Z stdout F Modelling tides using FES2014 tide model
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417549978Z stdout F Creating reduced resolution tide modelling array
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417584179Z stderr F FileNotFoundError: ['/var/share/fes2014/ocean_tide/2n2.nc', '/var/share/fes2014/ocean_tide/eps2.nc', '/var/share/fes2014/ocean_tide/j1.nc', '/var/share/fes2014/ocean_tide/k1.nc', '/var/share/fes2014/ocean_tide/k2.nc', '/var/share/fes2014/ocean_tide/l2.nc', '/var/share/fes2014/ocean_tide/la2.nc', '/var/share/fes2014/ocean_tide/m2.nc', '/var/share/fes2014/ocean_tide/m3.nc', '/var/share/fes2014/ocean_tide/m4.nc', '/var/share/fes2014/ocean_tide/m6.nc', '/var/share/fes2014/ocean_tide/m8.nc', '/var/share/fes2014/ocean_tide/mf.nc', '/var/share/fes2014/ocean_tide/mks2.nc', '/var/share/fes2014/ocean_tide/mm.nc', '/var/share/fes2014/ocean_tide/mn4.nc', '/var/share/fes2014/ocean_tide/ms4.nc', '/var/share/fes2014/ocean_tide/msf.nc', '/var/share/fes2014/ocean_tide/msqm.nc', '/var/share/fes2014/ocean_tide/mtm.nc', '/var/share/fes2014/ocean_tide/mu2.nc', '/var/share/fes2014/ocean_tide/n2.nc', '/var/share/fes2014/ocean_tide/n4.nc', '/var/share/fes2014/ocean_tide/nu2.nc', '/var/share/fes2014/ocean_tide/o1.nc', '/var/share/fes2014/ocean_tide/p1.nc', '/var/share/fes2014/ocean_tide/q1.nc', '/var/share/fes2014/ocean_tide/r2.nc', '/var/share/fes2014/ocean_tide/s1.nc', '/var/share/fes2014/ocean_tide/s2.nc', '/var/share/fes2014/ocean_tide/s4.nc', '/var/share/fes2014/ocean_tide/sa.nc', '/var/share/fes2014/ocean_tide/ssa.nc', '/var/share/fes2014/ocean_tide/t2.nc']
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417578454Z stderr F     raise FileNotFoundError(output_file)
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417574655Z stderr F   File "/usr/local/lib/python3.8/dist-packages/pyTMD/model.py", line 1425, in pathfinder
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417570251Z stderr F     self.model_file = self.pathfinder(model_files)
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417566232Z stderr F   File "/usr/local/lib/python3.8/dist-packages/pyTMD/model.py", line 842, in elevation
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.41756248Z stderr F     model = pyTMD.model(directory, format="netcdf", compressed=False).elevation(model)
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417558321Z stderr F   File "/usr/local/lib/python3.8/dist-packages/dea_tools/coastal.py", line 700, in model_tides
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417553496Z stderr F     tide_df = model_tides(
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417549347Z stderr F   File "/usr/local/lib/python3.8/dist-packages/dea_tools/coastal.py", line 957, in pixel_tides
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417544578Z stderr F     ds["tide_m"], tides_lowres = pixel_tides(ds, resample=True, directory="/var/share")
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417540644Z stderr F   File "/usr/local/lib/python3.8/dist-packages/coastlines/raster.py", line 1099, in generate_rasters
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417535887Z stderr F     generate_rasters(
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417530877Z stderr F   File "/usr/local/lib/python3.8/dist-packages/coastlines/raster.py", line 1231, in generate_rasters_cli
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417526732Z stderr F Traceback (most recent call last):
  |   | 2022-09-09 05:52:32 | 2022-09-08T19:52:32.417511312Z stderr F 2022-09-08 19:52:32 ERROR Study area 1096: Failed to run process with error ['/var/share/fes2014/ocean_tide/2n2.nc', '/var/share/fes2014/ocean_tide/eps2.nc', '/var/share/fes2014/ocean_tide/j1.nc', '/var/share/fes2014/ocean_tide/k1.nc', '/var/share/fes2014/ocean_tide/k2.nc', '/var/share/fes2014/ocean_tide/l2.nc', '/var/share/fes2014/ocean_tide/la2.nc', '/var/share/fes2014/ocean_tide/m2.nc', '/var/share/fes2014/ocean_tide/m3.nc', '/var/share/fes2014/ocean_tide/m4.nc', '/var/share/fes2014/ocean_tide/m6.nc', '/var/share/fes2014/ocean_tide/m8.nc', '/var/share/fes2014/ocean_tide/mf.nc', '/var/share/fes2014/ocean_tide/mks2.nc', '/var/share/fes2014/ocean_tide/mm.nc', '/var/share/fes2014/ocean_tide/mn4.nc', '/var/share/fes2014/ocean_tide/ms4.nc', '/var/share/fes2014/ocean_tide/msf.nc', '/var/share/fes2014/ocean_tide/msqm.nc', '/var/share/fes2014/ocean_tide/mtm.nc', '/var/share/fes2014/ocean_tide/mu2.nc', '/var/share/fes2014/ocean_tide/n2.nc', '/var/share/fes2014/ocean_tide/n4.nc', '/var/share/fes2014/ocean_tide/nu2.nc', '/var/share/fes2014/ocean_tide/o1.nc', '/var/share/fes2014/ocean_tide/p1.nc', '/var/share/fes2014/ocean_tide/q1.nc', '/var/share/fes2014/ocean_tide/r2.nc', '/var/share/fes2014/ocean_tide/s1.nc', '/var/share/fes2014/ocean_tide/s2.nc', '/var/share/fes2014/ocean_tide/s4.nc', '/var/share/fes2014/ocean_tide/sa.nc', '/var/share/fes2014/ocean_tide/ssa.nc', '/var/share/fes2014/ocean_tide/t2.nc']