Unable to write ISCE format on MacOS (SystemError: Unkown GDAL Error)

[scottyhq]

The bug

For version 0.9.2. The example workflow fails saving the glo30 DEM in ISCE format running via a conda-install on MacOS. It seems like the conda-forge gdal doesn't have a functional ISCE driver because writing to other formats is fine.

hyp3-isce2/src/hyp3_isce2/dem.py

Lines 123 to 124 in 5713e16

	dem_profile['nodata'] = None
	dem_profile['driver'] = 'ISCE'

To Reproduce

conda activate hyp3-isce2
python -m hyp3_isce2 ++process insar_tops_burst \
  S1_136231_IW2_20200604T022312_VV_7C85-BURST \
  S1_136231_IW2_20200616T022313_VV_5D11-BURST \
  --looks 20x4 \
  --apply-water-mask False

Additional context

Seems a problem only on MacOS, conda-installs on linux are working fine.

(hyp3-isce2) ➜  conda list                      
gdal                      3.6.3           py311h619941e_1    conda-forge
libgdal                   3.6.3                h8ea55aa_1    conda-forge
rasterio                  1.3.6           py311hc41c901_0    conda-forge

2024-01-12 08:54:12,127 - hyp3_isce2.insar_tops_burst - INFO - DEM ROI: (53.07855985074144, 27.491831927404057, 54.15583739250667, 27.84714385769884)
Reading glo_30 Datasets: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.11it/s]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/scott/GitHub/hyp3-isce2/src/hyp3_isce2/__main__.py", line 51, in <module>
    main()
  File "/Users/scott/GitHub/hyp3-isce2/src/hyp3_isce2/__main__.py", line 47, in main
    sys.exit(process_entry_point.load()())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/scott/GitHub/hyp3-isce2/src/hyp3_isce2/insar_tops_burst.py", line 482, in main
    isce_output_dir = insar_tops_burst(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/scott/GitHub/hyp3-isce2/src/hyp3_isce2/insar_tops_burst.py", line 105, in insar_tops_burst
    dem_path = download_dem_for_isce2(dem_roi, dem_name='glo_30', dem_dir=dem_dir, buffer=0, resample_20m=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/scott/GitHub/hyp3-isce2/src/hyp3_isce2/dem.py", line 130, in download_dem_for_isce2
    with rasterio.open(dem_path, 'w', **dem_profile) as ds:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/scott/miniconda3/envs/hyp3-isce2/lib/python3.11/site-packages/rasterio/env.py", line 451, in wrapper
    return f(*args, **kwds)
           ^^^^^^^^^^^^^^^^
  File "/Users/scott/miniconda3/envs/hyp3-isce2/lib/python3.11/site-packages/rasterio/__init__.py", line 314, in open
    dataset = writer(
              ^^^^^^^
  File "rasterio/_io.pyx", line 1450, in rasterio._io.DatasetWriterBase.__init__
  File "rasterio/_err.pyx", line 222, in rasterio._err.exc_wrap_pointer
SystemError: Unkown GDAL Error. To debug: https://rasterio.readthedocs.io/en/latest/topics/errors.html#debugging-internal-gdal-functions

[jhkennedy]

Hmm, @forrestfwilliams, since you use a Mac, can you look at this?

[forrestfwilliams]

Hey @scottyhq are you on an Intel or ARM based Mac? This will change our debugging efforts.

[forrestfwilliams]

@scottyhq I wasn't able to re-create the error you're seeing on my 2021 M1 MacBook Pro.

Just FYI, hyp3-isce2 won't run correctly natively on an ARM platform, because there is no ARM-based version of ISCE2 available on conda-forge.

To get around this, you can set up a pseudo-Intel conda environment for hyp3-isce2 by running the commands below in hyp3-isce2's home directory:

export CENV_NAME=hyp3-isce2
export CONDA_SUBDIR=osx-64
export OSX_SDK_DIR=/Library/Developer/CommandLineTools/SDKs
conda create -y -n $CENV_NAME
conda activate $CENV_NAME
conda config --env --set subdir osx-64
mamba env update --name $CENV_NAME -f environment.yml

Try installing hyp3-isce2 this way and let me know how it goes.

[scottyhq]

Also on a 2019 intel. I just followed the development instructions, so maybe it's a version issue with some new build?... no rush on debugging, just wanted to flag it

conda environment

name: hyp3-isce2
channels:
  - conda-forge
dependencies:
  - affine=2.4.0=pyhd8ed1ab_0
  - aom=3.5.0=hf0c8a7f_0
  - asf_search=6.7.3=pyhd8ed1ab_0
  - attrs=23.2.0=pyh71513ae_0
  - blosc=1.21.5=heccf04b_0
  - boost-cpp=1.78.0=h31500c2_2
  - boto3=1.34.17=pyhd8ed1ab_0
  - botocore=1.34.17=pyhd8ed1ab_0
  - branca=0.7.0=pyhd8ed1ab_1
  - brotli=1.1.0=h0dc2134_1
  - brotli-bin=1.1.0=h0dc2134_1
  - brotli-python=1.1.0=py311hdf8f085_1
  - bzip2=1.0.8=h10d778d_5
  - c-ares=1.25.0=h10d778d_0
  - ca-certificates=2023.11.17=h8857fd0_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cairo=1.16.0=h904041c_1014
  - certifi=2023.11.17=pyhd8ed1ab_0
  - cfitsio=4.2.0=hd56cc12_0
  - cftime=1.6.3=py311hc9a392d_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - click-plugins=1.1.1=py_0
  - cligj=0.7.2=pyhd8ed1ab_1
  - cloudpickle=3.0.0=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - contourpy=1.2.0=py311h7bea37d_0
  - coverage=7.4.0=py311he705e18_0
  - curl=8.1.2=hbee3ae8_0
  - cycler=0.12.1=pyhd8ed1ab_0
  - cytoolz=0.12.2=py311h2725bcf_1
  - dask-core=2023.12.1=pyhd8ed1ab_0
  - dateparser=1.2.0=pyhd8ed1ab_0
  - dem_stitcher=2.5.4=pyhd8ed1ab_0
  - exceptiongroup=1.2.0=pyhd8ed1ab_2
  - expat=2.5.0=hf0c8a7f_1
  - ffmpeg=5.1.2=gpl_h8b4fe81_106
  - fftw=3.3.10=nompi_h4fa670e_108
  - fiona=1.9.3=py311h9687163_0
  - flake8=7.0.0=pyhd8ed1ab_0
  - flake8-blind-except=0.2.1=pyhd8ed1ab_0
  - flake8-builtins=2.2.0=pyhd8ed1ab_0
  - flake8-import-order=0.18.2=pyhd8ed1ab_0
  - folium=0.15.1=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_1
  - fontconfig=2.14.2=h5bb23bf_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.47.2=py311he705e18_0
  - freetype=2.12.1=h60636b9_2
  - freexl=1.0.6=hb7f2c08_1
  - fsspec=2023.12.2=pyhca7485f_0
  - gdal=3.6.3=py311h619941e_1
  - geopandas=0.14.2=pyhd8ed1ab_0
  - geopandas-base=0.14.2=pyha770c72_0
  - geos=3.11.2=hf0c8a7f_0
  - geotiff=1.7.1=h6d03f96_7
  - gettext=0.21.1=h8a4c099_0
  - giflib=5.2.1=hb7f2c08_3
  - gmp=6.3.0=h93d8f39_0
  - gnutls=3.7.9=h1951705_0
  - graphite2=1.3.13=h2e338ed_1001
  - h5py=3.8.0=nompi_py311h777de7a_100
  - harfbuzz=6.0.0=h08f8713_0
  - hdf4=4.2.15=h9804679_6
  - hdf5=1.12.2=nompi_h48135f9_101
  - hyp3lib=2.0.2=pyhd8ed1ab_0
  - icu=70.1=h96cf925_0
  - idna=3.6=pyhd8ed1ab_0
  - imagecodecs-lite=2019.12.3=py311h4a70a88_7
  - imageio=2.33.1=pyh8c1a49c_0
  - importlib-metadata=7.0.1=pyha770c72_0
  - importlib_metadata=7.0.1=hd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - isce2=2.6.3=py311h3432358_0
  - jasper=4.0.0=h794afb9_1
  - jinja2=3.1.3=pyhd8ed1ab_0
  - jmespath=1.0.1=pyhd8ed1ab_0
  - joblib=1.3.2=pyhd8ed1ab_0
  - json-c=0.16=h01d06f9_0
  - kealib=1.5.0=h5c1f988_0
  - kiwisolver=1.4.5=py311h5fe6e05_1
  - krb5=1.20.1=h049b76e_0
  - lame=3.100=hb7f2c08_1003
  - lazy_loader=0.3=pyhd8ed1ab_0
  - lcms2=2.15=h2dcdeff_1
  - lerc=4.0.0=hb486fe8_0
  - libaec=1.1.2=he965462_1
  - libblas=3.9.0=20_osx64_openblas
  - libbrotlicommon=1.1.0=h0dc2134_1
  - libbrotlidec=1.1.0=h0dc2134_1
  - libbrotlienc=1.1.0=h0dc2134_1
  - libcblas=3.9.0=20_osx64_openblas
  - libcurl=8.1.2=hbee3ae8_0
  - libcxx=16.0.6=hd57cbcb_0
  - libdeflate=1.17=hac1461d_0
  - libedit=3.1.20191231=h0678c8f_2
  - libev=4.33=h10d778d_2
  - libexpat=2.5.0=hf0c8a7f_1
  - libffi=3.4.2=h0d85af4_5
  - libgdal=3.6.3=h8ea55aa_1
  - libgfortran=5.0.0=13_2_0_h97931a8_1
  - libgfortran5=13.2.0=h2873a65_1
  - libglib=2.78.1=h6d9ecee_0
  - libiconv=1.17=hd75f5a5_2
  - libidn2=2.3.4=hb7f2c08_0
  - libjpeg-turbo=2.1.5.1=h0dc2134_1
  - libkml=1.3.0=hab3ca0e_1016
  - liblapack=3.9.0=20_osx64_openblas
  - liblapacke=3.9.0=20_osx64_openblas
  - libnetcdf=4.9.1=nompi_hb17d2b5_102
  - libnghttp2=1.58.0=h64cf6d3_0
  - libopenblas=0.3.25=openmp_hfef2a42_0
  - libopencv=4.7.0=py311hbc2f7b1_2
  - libopus=1.3.1=hc929b4f_1
  - libpng=1.6.39=ha978bb4_0
  - libpq=15.2=h3640bf0_0
  - libprotobuf=3.21.12=h7d26f99_2
  - librttopo=1.1.0=h5c328d2_13
  - libspatialindex=1.9.3=he49afe7_4
  - libspatialite=5.0.1=h595bb97_24
  - libsqlite=3.44.2=h92b6c6a_0
  - libssh2=1.11.0=hd019ec5_0
  - libtasn1=4.19.0=hb7f2c08_0
  - libtiff=4.5.0=hd920806_5
  - libunistring=0.9.10=h0d85af4_0
  - libvpx=1.11.0=he49afe7_3
  - libwebp-base=1.3.2=h0dc2134_0
  - libxcb=1.15=hb7f2c08_0
  - libxml2=2.10.3=h201ad9d_4
  - libxslt=1.1.37=h5d22bc9_0
  - libzip=1.10.1=hc158999_3
  - libzlib=1.2.13=h8a1eda9_5
  - llvm-openmp=17.0.6=hb6ac08f_0
  - locket=1.0.0=pyhd8ed1ab_0
  - lxml=4.9.2=py311h9f2bb26_0
  - lz4-c=1.9.4=hf0c8a7f_0
  - mapclassify=2.6.1=pyhd8ed1ab_0
  - markupsafe=2.1.3=py311h2725bcf_1
  - matplotlib-base=3.8.2=py311hd316c10_0
  - mccabe=0.7.0=pyhd8ed1ab_0
  - munch=4.0.0=pyhd8ed1ab_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - ncurses=6.4=h93d8f39_2
  - netcdf4=1.6.3=nompi_py311he6f34b2_100
  - nettle=3.9.1=h8e11ae5_0
  - networkx=3.2.1=pyhd8ed1ab_0
  - nspr=4.35=hea0b92c_0
  - nss=3.96=ha05da47_0
  - numpy=1.23.5=py311h62c7003_0
  - opencv=4.7.0=py311h6eed73b_2
  - openh264=2.3.1=hf0c8a7f_2
  - openjpeg=2.5.0=h13ac156_2
  - openmotif=2.3.8=h6b0d8c2_4
  - openssl=3.1.4=hd75f5a5_0
  - p11-kit=0.24.1=h65f8906_0
  - packaging=23.2=pyhd8ed1ab_0
  - pandas=2.1.4=py311h1eadf79_0
  - partd=1.4.1=pyhd8ed1ab_0
  - patsy=0.5.6=pyhd8ed1ab_0
  - pcre2=10.40=h1c4e4bc_0
  - pep8=1.7.1=py_0
  - pillow=9.5.0=py311h7cb0e2d_1
  - pip=23.3.2=pyhd8ed1ab_0
  - pixman=0.43.0=h73e2aa4_0
  - pluggy=1.3.0=pyhd8ed1ab_0
  - poppler=23.03.0=hc777cac_1
  - poppler-data=0.4.12=hd8ed1ab_0
  - postgresql=15.2=hbea33b9_0
  - proj=9.1.1=hf909084_2
  - pthread-stubs=0.4=hc929b4f_1001
  - py-opencv=4.7.0=py311h9bdd199_2
  - pycodestyle=2.11.1=pyhd8ed1ab_0
  - pyflakes=3.2.0=pyhd8ed1ab_0
  - pyparsing=3.1.1=pyhd8ed1ab_0
  - pyproj=3.5.0=py311hdc3c720_0
  - pyshp=2.3.1=pyhd8ed1ab_0
  - pysocks=1.7.1=pyha2e5f31_6
  - pytest=7.4.4=pyhd8ed1ab_0
  - pytest-console-scripts=1.4.1=pyhd8ed1ab_0
  - pytest-cov=4.1.0=pyhd8ed1ab_0
  - pytest-mock=3.12.0=pyhd8ed1ab_0
  - python=3.11.6=h30d4d87_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-tzdata=2023.4=pyhd8ed1ab_0
  - python_abi=3.11=4_cp311
  - pytz=2023.3.post1=pyhd8ed1ab_0
  - pywavelets=1.4.1=py311h4a70a88_1
  - pyyaml=6.0.1=py311h2725bcf_1
  - rasterio=1.3.6=py311hc41c901_0
  - readline=8.2=h9e318b2_1
  - regex=2023.12.25=py311he705e18_0
  - remotezip=0.12.2=pyhd8ed1ab_0
  - requests=2.31.0=pyhd8ed1ab_0
  - responses=0.24.1=pyhd8ed1ab_0
  - rtree=1.1.0=py311hbc1f44b_0
  - s3transfer=0.10.0=pyhd8ed1ab_0
  - scikit-image=0.20.0=py311hd84f3f5_1
  - scikit-learn=1.3.2=py311h66081b9_2
  - scipy=1.11.4=py311he0bea55_0
  - setuptools-scm=8.0.4=pyhd8ed1ab_0
  - setuptools_scm=8.0.4=hd8ed1ab_0
  - shapely=2.0.1=py311heb7bb94_1
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.1.10=h225ccf5_0
  - snuggs=1.4.7=py_0
  - sqlite=3.44.2=h7461747_0
  - statsmodels=0.14.1=py311hc9a392d_0
  - svt-av1=1.4.1=hf0c8a7f_0
  - tabulate=0.9.0=pyhd8ed1ab_1
  - tenacity=8.2.2=pyhd8ed1ab_0
  - threadpoolctl=3.2.0=pyha21a80b_0
  - tifffile=2020.6.3=py_0
  - tiledb=2.13.2=h8b9cbf0_0
  - tk=8.6.13=h1abcd95_1
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - toolz=0.12.0=pyhd8ed1ab_0
  - tqdm=4.66.1=pyhd8ed1ab_0
  - types-pyyaml=6.0.12.12=pyhd8ed1ab_0
  - typing-extensions=4.9.0=hd8ed1ab_0
  - typing_extensions=4.9.0=pyha770c72_0
  - tzcode=2023d=h10d778d_0
  - tzdata=2023d=h0c530f3_0
  - tzlocal=5.2=py311h6eed73b_0
  - uriparser=0.9.7=hf0c8a7f_1
  - urllib3=1.26.18=pyhd8ed1ab_0
  - x264=1!164.3095=h775f41a_2
  - x265=3.5=hbb4e6a2_3
  - xerces-c=3.2.4=h2007e90_1
  - xorg-kbproto=1.0.7=h35c211d_1002
  - xorg-libice=1.0.10=h0d85af4_0
  - xorg-libsm=1.2.3=h0d85af4_1000
  - xorg-libx11=1.6.12=h9a43bea_0
  - xorg-libxau=1.0.11=h0dc2134_0
  - xorg-libxdmcp=1.1.3=h35c211d_0
  - xorg-libxext=1.3.4=h01d97ff_0
  - xorg-libxft=2.3.4=hf9f1598_1
  - xorg-libxmu=1.1.3=h0b31af3_0
  - xorg-libxp=1.0.3=h10d778d_1002
  - xorg-libxrender=0.9.10=h01d97ff_1002
  - xorg-libxt=1.1.5=h01d97ff_1003
  - xorg-renderproto=0.11.1=h0d85af4_1002
  - xorg-xextproto=7.3.0=hb7f2c08_1003
  - xorg-xproto=7.0.31=h35c211d_1007
  - xyzservices=2023.10.1=pyhd8ed1ab_0
  - xz=5.2.6=h775f41a_0
  - yaml=0.2.5=h0d85af4_2
  - zipp=3.17.0=pyhd8ed1ab_0
  - zlib=1.2.13=h8a1eda9_5
  - zstd=1.5.5=h829000d_0
  - pip:
      - hyp3-isce2==0.9.2
      - setuptools==67.6.1
      - wheel==0.40.0

[forrestfwilliams]

@scottyhq what MacOS are you running? I'm on Sonoma 14.1 and can't recreate the issue.

[scottyhq]

Thanks for looking into it @forrestfwilliams, after some frustrating debugging I realized I had GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR set in my terminal (often these days since working mostly with URLs). And Because the ISCE driver seems to require sidecar files even to open new files, gdal unceremoniously exits :(, thus the traceback!

[jhkennedy]

@scottyhq oh, that's handy to know! I set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR often as well.