Skip to content

Latest commit

 

History

History
764 lines (537 loc) · 32 KB

CHANGELOG.md

File metadata and controls

764 lines (537 loc) · 32 KB

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to PEP 440 and uses Semantic Versioning.

[9.0.2]

Added

  • The static-analysis Github Actions workflow now includes mypy for type checking.

[9.0.1]

Changed

  • The static-analysis Github Actions workflow now uses ruff rather than flake8 for linting.

[9.0.0]

Changed

  • Orbit files are now retrieved using the s1-orbits library.

Removed

  • Removed support for the --esa-username and --esa-password command-line options and the ESA_USERNAME and ESA_PASSWORD environment variables.

[8.2.0]

Changed

[8.1.4]

Changed

  • Updated versions for Python and GitHub Actions dependencies.

[8.1.3]

Changed

  • Input SLCs for insar jobs are now downloaded in parallel in __main__.insar()
  • Consolidated python dependencies to minimize differences between local, docker, and github environments

[8.1.2]

Changed

  • Enabled LZW compression for intermediate GeoTIFFs generated by hyp3_gamma.water_mask to reduce required disk space
  • Additional minor edits to water mask language

[8.1.1]

Changed

  • Updated water mask language to reflect switch to OpenStreetMaps/ESA WorldCover based water map.

[8.1.0]

Added

  • Support for a new water masking dataset based off of OpenStreetMaps and ESA WorldCover data.

Removed

  • Polygon processing functions: split_geometry_on_antimeridian and get_envelope_wgs84 from water_mask.py.

[8.0.1]

Changed

  • Upgraded to hyp3lib>=3,<4 from >=2,<3
  • All requirements for the conda environment are once again installed via conda in environment.yml, eliminating the workaround for #421 implemented in v5.7.2.

[8.0.0]

Removed

  • Support for legacy SRTM/NED DEMs in RTC processing. All RTC processing now uses the Copernicus DEM.
    • legacy option for the --dem-name parameter of __main__.rtc()
    • legacy option for the --dem-name parameter of rtc_sentinel.main()
    • legacy option for the dem_name parameter of rtc_sentinel.rtc_sentinel_gamma()
    • EU_DEM_V11, GIMP, IFSAR, NED13, NED1, NED2, REMA, SRTMGL1, and SRTMGL3 entries from metadata.rtc.SUPPORTED_DEMS

[7.0.1]

Fixed

  • A bug that would cause some INSAR jobs, with apply-water-mask set to True to fail with a shapely.errors.TopologicalError.

[7.0.0]

Changed

  • Upgraded hyp3lib dependency to version 2.x.x.
  • As of HyP3-lib v2.0.0, the Copernicus Data Space Ecosystem (CDSE) will now be used for downloading Sentinel-1 orbit files from ESA.
  • CDSE credentials must be provided via the --esa-username and --esa-password command-line options or the ESA_USERNAME and ESA_PASSWORD environment variables.

[6.4.1]

Fixed

  • Incorrect / blank water masks in some areas (primarily Europe), by clipping the water mask to the envelope of the product before rasterization.

[6.4.0]

Added

  • The --phase-filter-parameter option now accepts a value of 0.0, which indicates that the adaptive phase filter will be skipped.
  • Documentation for the adaptive phase filter parameter in product readme and xml files.

[6.3.4]

Added

  • New DEM coverage file which allows for jobs over Armenia and Azerbaijan to use COP90. Fixes #462.

[6.3.3]

Fixed

  • #491 util.unzip_granule fails with StopIteration for recently acquired Sentinel-1 scenes

[6.3.2]

Fixed

  • Description of the range of the lv_phi to be -Pi to Pi in the readme.txt.j2 and the lv_phi_tif.xml.j2. Fixes #478.

[6.3.1]

Fixed

  • Phase unwrapping for very large interferograms is now performed in two range patches to keep the total memory requirement under 31,600 MB. Fixes #316.

Removed

  • The unwrapping_geocoding function and CLI for InSAR processing no longer allow specifying the number of range and/or azimuth patches to use when unwrapping. Unwrapping is always done with 1 azimuth patch and 1 or 2 range patches, depending on size.

[6.3.0]

Changed

[6.2.4]

Changed

  • Remove the NoData setting in the output dem.tif

[6.2.3]

Added

  • References to the DEM pixel value unit (meters) in the readme.md.txt file for the RTC and InSAR products

[6.2.2]

Changed

  • The initial DEM mosaic for RTC processing is now resampled to the desired output pixel spacing so that pixels for 20m products are always aligned to a common grid.

[6.2.1]

Changed

  • Updated language in RTC readme files to remove references to standard products

[6.2.0]

Added

  • Added resolution=20.0 option for __main__.rtc()

[6.1.0]

Added

  • A --phase-filter-paameter option has been added to the __main__ (HyP3) and ifm_sentinel.py entry points to specify the adaptive phase filter parameter used when processing InSAR products. The provided value is ultimately passed to the alpha argument of the adf GAMMA function, which uses the algorithm with a constant exponent as described in https://doi.org/10.1029/1998GL900033.

[6.0.1]

Fixed

  • Broken Copernicus DEM links.

[6.0.0]

Removed

  • Surface water extent map processing of RTC products, which is now entirely contained in asf-tools and its associated docker container image. This includes removing:
    • water_map entrypoint to create a water map product
    • asf_tools environment from the hyp3-gamma docker container image

[5.7.5]

Added

  • Information about image pixel spacing to RTC product README.md.txt

[5.7.4]

Added

  • ++omp-num-threads parameter to the main() entrypoint to limit the number of threads used by GAMMA during multiprocessing.

[5.7.3]

Added

  • Add decibel option in the README.md.txt for the RTC product

[5.7.2]

Changed

  • Upgraded to asf-tools v0.4.4 from v0.4.3

Fixed

  • In the conda environment, install hyp3lib and lxml via pip due to an libxml2 issue as described in #421

[5.7.1]

Changed

[5.7.0]

Changed

  • Upgraded to the base image used for the hyp3-gamma docker image to Ubuntu 22.04 from 20.04. This also changes:
    • the GAMMA software 20220630 build
    • Python to 3.10.6 from 3.8.10
    • Numpy to 1.21.5 from 1.17.4
    • GDAL to 3.4.1 from 3.0.4

[5.6.0]

Changed

  • Removed buffer from water mask during unwrapped_phase calculation
  • Updated metadata templates to reflect new water masking approach and link to new resources

[5.5.1]

Added

  • Added information about the end of mission for S1B to all product readme files and xml templates

[5.5.0]

Changed

[5.4.5]

Changed

  • Removed the intermediate initial/fuzzy geotiffs from the water_map product package.

[5.4.4]

Changed

  • insar now uploads unzipped product files (in addition to the zip archive) when using the --bucket option

[5.4.3]

Changed

  • Upgraded to asf-tools v0.4.2 from v0.4.1

[5.4.2]

Changed

  • Increase the window size and ignore pixels near an edge when determining the phase unwrapping reference point for InSAR products.
  • Upgraded to asf-tools v0.4.1 from v0.4.0

Fixed

  • Map coordinates for the phase unwrapping reference point are now reported correctly in the output metadata text file for InSAR products. Fixes #380

[5.4.1]

Added

  • Added links to reference material for the Minimum Cost Flow (MCF) phase unwrapping algorithm to the InSAR readme template and unwrapped phase png xml template
  • Added language to the InSAR readme template to clarify that the sign convention used for unwrapped phase is opposite to that used for the displacement maps
  • Added new HyP3 citation to the InSAR and RTC readme templates

[5.4.0]

Changed

[5.3.0]

Added

  • Added --include-flood-depth option for water_map to produce a map of estimated flood depths.

[5.2.1]

Changed

  • Output the reference point in SAR space in the meta text file. The reference point indexes start form 0.

[5.2.0]

Changed

  • The amplitude threshold used for the InSAR phase unwrapping validity mask was changed from 0.2 to 0.0 (in power scale). The coherence threshold remains at 0.1, so the validity mask is now calculated based only on coherence values.
  • The InSAR phase unwrapping reference point is now determined using a neighborhood coherence value. The pixels with the highest coherence are examined using a 9-pixel window to find the pixel with the highest neighborhood coherence value, which is then used as the reference point for phase unwrapping.

[5.1.8]

Changed

  • Upgraded to asf-tools v0.3.2 from v0.3.1

[5.1.7]

Fixed

  • asf-tools conda environment, which is used by the water mapping code, is now copied from the latest asf-tools image.

[5.1.6]

Fixed

  • asf-tools conda environment, which is used by the water mapping code, has been (temporarily) locked to the environment as created in HyP3 GAMMA v5.1.2 to resolve some dependency issues.

[5.1.5]

Changed

  • Upgraded to hyp3-lib v1.7.0 from v1.6.8

[5.1.4]

Changed

  • hyp3_gamma.dem will now pull the Copernicus DEM from the us-west-2 mirror located at s3://asf-dem-west/v2/COP30 to reduce cross-region traffic

[5.1.3]

Added

  • It is now possible to inject Earthdata username and password using environment variables: EARTHDATA_USERNAME and EARTHDATA_PASSWORD.

5.1.2

Fixed

  • Unable to open datasource error when providing rtc_sentinel.py with absolute path to SAFE directory. Fixes #332

5.1.1

Changed

  • Reference point information has been modified for clarity in the *.txt parameter file

5.1.0

Added

  • Ascending/descending and orbit number information have been added to the *.txt parameter file in InSAR products
  • The reference point coordinate information (in both SAR and map space) has been added to the .txt parameter file in InSAR products

Changed

  • The highest coherent pixel will be used as the starting point when unwrapping during InSAR processing

5.0.1

Changed

  • InSAR readme and unwrapped phase png xml templates have been updated to include more information about the fringe pattern in the unwrapped phase browse images

5.0.0

Added

  • An --include-displacement-maps argument to the insar entrypoint which will cause both the line-of-sight and vertical displacement maps to be included.

Changed

  • The include_los_displacement parameter to insar_sentinel_gamma in hyp3_gamma.insar.ifm_sentinel has changed to include_displacement_maps. Now, both the line-of-sight displacement and the vertical displacement maps will be included if this option is set to True.
  • InSAR README and XML templates have been updated to reflect the change in displacement map options and to include more information on adjusting unwrapped phase and displacement map values relative to a user-selected reference point.
  • Updated the URL for the Copernicus DEM information webpage.

Deprecated

  • The --include-los-displacment argument to the insar entrypoint is deprecated and may be removed at any time. --include-displacement-maps should be used instead as setting either option to true will cause both the line-of-sight and vertical displacement maps to be included.

4.9.0

Added

  • hyp3_metadata v1.3.0 has been merged into HyP3 GAMMA at hyp3_gamma.metadata generating HyP3 GAMMA products' metadata. See the metadata README for usage.

4.8.0

Changed

  • Upgraded to the latest available GAMMA software version and related dependencies
    • GAMMA software 20210701 from 20191209
    • Ubuntu 20.04 from 18.04
    • Python 3.8.10 from 3.6.9
    • GDAL 3.0.4 from 2.2.3

4.7.0

Added

  • Option to apply water masking to the unwrapped_phase calculation in the gamma processing
    • apply_water_mask parameter to ifm_sentinel.insar_sentinel_gamma function
    • --apply-water-mask parameter to insar entrypoint
    • -m parameter to ifm_sentinel.py script

4.6.3

Added

  • InSAR products now include an ellipsoidal incidence angle GeoTIFF (in addition to local incidence angle) when selecting the include_inc_map option

Changed

  • Upgraded to hyp3-metadata v1.2.5 from v1.2.4

4.6.2

Added

  • InSAR products now include a land/water mask GeoTIFF and a corresponding ArcGIS metadata xml file

Changed

  • Upgraded to hyp3-metadata v1.2.4 from v1.2.3

4.6.1

Changed

  • conda-env.yml has been renamed to environment.yml in-line with community practice
  • Upgraded to hyp3-lib v1.6.8 from v1.6.7

4.6.0

Added

  • water_map entrypoint to create a water map product
  • Option to output decibel scaled RTC products
    • --scale parameter to rtc entrypoint now accepts decibel
    • scale parameter to rtc_sentinel.rtc_sentinel_gamma now accepts decibel

Changed

  • Upgraded to hyp3-metadata v1.2.3 from v1.1.0

4.5.1

Added

  • --include-wrapped-phase option for the insar entrypoint to include the wrapped phase GeoTIFF in the output product

Changed

  • Updated the description of the Wrapped Interferogram in the InSAR product README, including the optional GeoTIFF

4.5.0

Added

  • Option to include DEM file with InSAR products
    • include_dem parameter to ifm_sentinel.insar_sentinel_gamma function
    • --include-dem parameter to insar entrypoint
    • -d parameter to ifm_sentinel.py script
  • Parameter metadata file for InSAR products now includes Slant range near, Slant range center and Slant range far attributes

4.4.0

Added

  • Option to include an Incidence Angle GeoTIFF with InSAR products
    • include_inc_map parameter to ifm_sentinel.insar_sentinel_gamma function
    • --include-inc-map parameter to insar entrypoint
    • -i parameter to ifm_sentinel.py script
  • Parameter metadata file for InSAR products now includes Spacecraft height and Earth radius at nadir attributes

4.3.0

Changed

  • InSAR processing via ifm_sentinel.py now leverages the Copernicus GLO-30 Public DEM
  • InSAR GeoTIFF files are now aligned to a common 40m (for 10x2 looks) or 80m (for 20x4 looks) pixel grid

4.2.0

Added

  • Option for RTC_GAMMA jobs to use either Copernicus GLO-30 Public DEM or legacy SRTM/NED DEMs for processing:
    • --dem-name option to rtc entrypoint and rtc_sentinel.py script
    • dem_name parameter to hyp3_gamma.rtc_sentinel.rtc_sentinel_gamma()
  • hyp3_gamma.dem module for preparing GeoTIFF mosaics of the Copernicus GLO-30 Public DEM

Changed

  • Upgraded to hyp3-metadata v0.4.2 from v0.2.0

4.1.2

Changed

  • Reverted change that allows ifm_sentinel.py to proceed with processing when no Restituted or Precision orbit file is found. Restituted or Precision orbit files are required.

4.1.1

Changed

  • ifm_sentinel.py will now proceed with processing when no Restituted or Precision orbit file is found
  • Upgraded to hyp3lib v1.6.7 from v1.6.6

4.1.0

Added

  • Ability to include an RGB decomposition GeoTIFF in RTC output
    • an --include-rgb option has been added to rtc_sentinel.py
    • an include_rgb keyword argument has been added to hyp3_gamma.rtc.rtc_sentinel.rtc_sentinel_gamma

Changed

  • Upgraded to hyp3lib v1.6.6 from v1.6.5

4.0.5

Changed

  • rtc now uploads unzipped product files (in addition to the zip archive) when using the --bucket option
  • Upgraded to hyp3lib v1.6.5 from v1.6.4

4.0.4

Changed

  • rtc_sentinel.py will now proceed with processing when no Restituted or Precision orbit file is found

4.0.3

Fixed

  • Resolved off-by-one error when computing days of separation for InSAR product names in ifm_sentinel.py

4.0.2

Changed

  • Upgraded to hyp3lib v1.6.4 from v1.6.3

4.0.1

Added

  • Added wrapped phase geotiff via command line option -m to ifm_sentinel.py

4.0.0

RTC processing via rtc_sentinel.py has been significantly refactored, and a number of legacy options have been removed.

Added

  • rtc.util.unzip_granule() unzips a S1 zip file and returns the .SAFE directory name

Changed

  • The parameters to rtc_sentinel.py and rtc.rtc_sentinel.rtc_sentinel_gamma() have been significantly revised. Review the corresponding help for more details:
    • rtc_sentinel.py --help
    • from hyp3_gamma.rtc.rtc_sentinel import rtc_sentinel_gamma; help(rtc_sentinel_gamma)
  • A refactored rtc.coregistration module has replaced rtc.check_coreg. The check_coreg.py entrypoint has been removed.
  • Upgraded to hyp3_metadata v0.2.0 from v0.1.4

Removed

  • Legacy support for the GIMP and REMA DEMs has been removed, including the rtc.smoothem module and the corresponding smoothem.py entrypoint.
  • User-provided DEMs in GAMMA format are no longer supported. DEMs in GeoTIFF format are still supported.
  • The output/out_name options to set the name of the output product have been removed.
  • The shape option to subset to the bounds of a shapefile has been removed. Subsetting to a bounding box or to a user-provided DEM are still supported.
  • The terms option has been removed. DEM matching will always use a one-term offset polynomial.
  • The par option to provide a pre-generated diff_par offset file has been removed.
  • The fail/dead_flag options have been removed. Processing will always proceed using dead reckoning when dem_matching is selected and co-registration fails.
  • hyp3_gamma.util.find_and_remove() has been removed.

3.1.0

Added

  • hyp3-insar-gamma v2.2.1 has been merged into HyP3 GAMMA at hyp3_gamma.insar to provide InSAR processing of Sentinel-1. hyp3_gamma now provides these additional entrypoints:
    • insar provides the InSAR process interface to HyP3
    • ifm_sentinel.py is the InSAR (science code) processor
    • interf_pwr_s1_lt_tops_proc.py for coregistering Sentinel-1 SLC data and DEMs
    • unwrapping_geocoding.py for unwrapping and geocoding Sentinel-1 INSAR products from GAMMA

Changed

  • hyp3-gamma entrypoint now allows InSAR processing through the ++process insar option

3.0.0

HyP3 RTC GAMMA as been transformed into HyP3 GAMMA; a processor based plugin

Changed

  • hyp3_rtc_gamma has been transformed into a hyp3_gamma package

    • hyp3_rtc_gamma.__init__ has moved to hyp3_gamma.__init__
    • hyp3_rtc_gamma.__main__ has moved to hyp3_gamma.__main__
    • hyp3_rtc_gamma.check_coreg has moved to hyp3_gamma.rtc.check_coreg
    • hyp3_rtc_gamma.rtc_sentinel has moved to hyp3_gamma.rtc.rtc_sentinel
    • hyp3_rtc_gamma.smoothem has moved to hyp3_gamma.rtc.smoothem

  • hyp3_gamma now provides these entrypoints:

    • hyp3_gamma is the main HyP3 entrypoint
    • rtc provides the RTC process interface to HyP3
    • rtc_sentinel.py is the RTC (science code) processor
    • check_coreg.py for checking the results of the GAMMA coregistration process
    • smoothdem.py for smoothing and filling holes in DEMs

2.4.1

Changed

  • rtc_sentinel.py now uses GAMMA's updated mk_geo_radcal2 instead of mk_geo_radcal to remove no-data values in lakes and rivers

  • Upgraded to hyp3lib v1.6.3 from v1.6.2

2.4.0

HyP3 v1 is no longer supported as of this release.

Added

  • A new --include-scattering-area option has been added to rtc_sentinel.py and hyp3_rtc_gamma_v2 to include a geotiff of scattering area in the product package. This supports creation of composites of RTC images using Local Resolution Weighting per Small (2012) https://doi.org/10.1109/IGARSS.2012.6350465.

Changed

  • Upgraded to hyp3_metadata v0.1.4 from v0.1.2

Removed

  • rtc_sentinel.py no longer creates a flattened backscatter image.
  • The hyp3_rtc_gamma package entrypoint and HyP3 v1 support has been removed.
  • rtc_gamma package entrypoint will now pass arguments to the hyp3_rtc_gamma_v2 entrypoint by default.

Fixed

  • rtc_sentinel.py now runs successfully when --nocrosspol is specified (#179)

2.3.4

Changed

  • Upgraded to hyp3lib v1.6.2 from v1.6.1

2.3.3

Changed

  • Upgraded to hyp3lib v1.6.1 from v1.6.0

2.3.2

Changed

  • Upgraded to hyp3lib v1.6.0 from v1.5.0
  • Upgraded to hyp3_metadata v0.1.2 from v0.1.1

2.3.1

Changed

  • README.txt has been renamed to README.md.txt
  • README.md.txt and .xml files are now generated using hyp3-metadata-templates v0.1.1
  • file_type tags are now applied to browse and thumbnail images uploaded to S3 to facilitate consistent ordering in HyP3 Api responses

Removed

  • Removed the all-white placeholder thumbnail from _ls_map.tif.xml

2.3.0

Added

  • Added cursory developer setup instructions in README.md

Changed

Removed

  • RTC Gamma products no longer include an ISO 19115-2 .iso.xml metadata file.
  • ASF MapReady is no longer a dependency and has been removed.

2.2.0

Changed

  • Files now get uploaded to S3 with a file_type in v2 tag indicating what the file is
  • Files no longer get a prefix based on file type in v2
  • Removed lo_flag from rtc_sentinel since it duplicates the functionality of resolution

2.1.1

Changed

  • Input scenes are now downloaded directly from NGAP distribution endpoint rather than ASF's datapool
  • Updates the output product README to include usage guidelines with acknowledgment and citation information
  • Removes the ESA_citation.txt file from the output product as it's included in the README
  • Drops the "This is an RTc product from ..." blurb from the bottom of product notification emails
  • Incompatible granules will fail much earlier in the processing chain

Removed

  • Non-functional support for non-Sentinel-1 products

2.1.0

Changed

  • Implemented new naming convention for output products. The names of output zips, folders, and files now all share a common format.
  • Revised README content:
    • Updated description of product naming scheme
    • Removed references to EUDEM, GEMP, and REMA DEMs that are no longer used since v2.0.0.
    • Clarified which RTC processing steps are performed when DEM matching is or is not requested.
  • Orbit files are now downloaded once at the start of processing, rather than once for each polarization image.
  • Orbit search priority is now POEORB from ESA, POEORB from ASF, RESORB from ESA, RESORB from ASF.
  • main_v2() now downloads Sentinel-1 data directly from ASF's NGAP distribution endpoint, rather than datapool.
  • Upgrade to hyp3-lib v1.4.1 from 1.3.0.
  • Install hyp3-lib via conda instead of pip

2.0.6

Fixed

  • Resolved issue where thumbnail filenames ended in ..png rather than .png

2.0.5

Added

  • The v2 entrypoint will now create and upload thumbnail images to S3 in addition to browse images

2.0.4

Changed

  • The v2 entrypoint will now upload browse images and thumnail images in addition to the zip file.
  • Upgrade to hyp3-lib 1.3.0. In particular, geotiff products no longer include overviews.
  • Eliminated seprate "low-res" (1024x) and "high-res" (2048x) browse image resolutions in favor of a single 2048x image.

2.0.3

Fixed

2.0.2

Changed

  • The v2 entrypoint will now make up to three retry attempts if it fails to download the input granule from the ASF archive.
  • Changed the name of the product README file to <product_name>.README.txt, e.g. S1A_IW_RT30_20170708T161200_G_gpn.README.txt
  • Calls to mk_geo_radcal will no longer include the -j do not use layover-shadow map in the calculation of pixel_area flag. The layover-shadow map will now be consistently applied to all products.
  • Upgraded to hyp3-lib v1.2.2
  • Removed custom blank_bad_data.py from mk_geo_radcal processing. Border pixels for older GRD products are now cleaned using the default make_edge setting of par_S1_GRD.

2.0.1

Changed

  • hyp3-v2 output products are now packaged in a .zip file similar to hyp3-v1 products

2.0.0

This is a significant refactor of hyp3-gamma into:

  • A pip installable package called hyp3_rtc_gamma
  • A stand alone, container-based HyP3 plugin

NOTE: There are significant changes to the overall repository structure and will break all existing HyP3 workflows!

Removed

  • Python 2. This package now requires Python 3.6+
  • A patched version of GAMMA's mk_geo_radcal and an associated blank_bad_data.py script

Added

  • A packaging and testing structure -- now pip installable and testing is done via pytest
    • Previous command line scripts are now registered entrypoints and created when the package is pip installed:
      • check_coreg.py
      • rtc_sentinel.py
      • smooth_dem_tiles.py
      • 'xml2meta.py'
  • A Dockerfile to build the HyP3 plugin
  • A CI/CD workflow setup, which will build and publish the docker container
  • The processing script that used to live in the now depreciated cloud-proj repository has been moved into the package as hyp3_rtc_gamma.__main__ and also registered as a hyp3_rtc_gamma entrypoint
  • A second, in development, entrypoint for HyP3 v2 has been added to hyp3_rtc_gamma.__main__

Changed

  • All of src/ is now contained in the hyp3_rtc_gamma package
  • All of etc/is now contained in hyp3_rtc_gamma.etc
  • The version number is now tracked automatically via git tags instead of in etc/version.txt