Scenario Failure: sentinel1_stats

[github-actions]

Benchmark scenario ID: sentinel1_stats
Benchmark scenario definition: https://github.com/ESA-APEx/apex_algorithms/blob/2cc41fbd25d36f0fcb6caa24ebb09de82caa062c/algorithm_catalog/vito/sentinel1_stats/benchmark_scenarios/sentinel1_stats.json
openEO backend: openeofed.dataspace.copernicus.eu

GitHub Actions workflow run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18652986817
Workflow artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18652986817#artifacts

Test start: 2025-10-20 13:23:29.505005+00:00
Test duration: 0:02:22.580305
Test outcome: ❌ failed

Last successful test phase: download-reference
Failure in test phase: compare:derived_from-change

Contact Information

Name	Organization	Contact
Pratichhya Sharma	VITO	Contact via VITO (VITO Website, GitHub)

Process Graph

{
  "s1stats1": {
    "process_id": "sentinel1_stats",
    "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/refs/heads/main/algorithm_catalog/vito/sentinel1_stats/openeo_udp/sentinel1_stats.json",
    "arguments": {
      "spatial_extent": {
        "east": 8.92,
        "north": 44.45,
        "south": 44.4,
        "west": 8.82
      },
      "temporal_extent": [
        "2023-05-01",
        "2023-07-30"
      ]
    },
    "result": true
  }
}

Error Logs

scenario = BenchmarkScenario(id='sentinel1_stats', description='Sentinel 1 statistics example', backend='openeofed.dataspace.cope...work/apex_algorithms/apex_algorithms/algorithm_catalog/vito/sentinel1_stats/benchmark_scenarios/sentinel1_stats.json'))
connection_factory = <function connection_factory.<locals>.get_connection at 0x7f0e4c465c60>
tmp_path = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0')
track_metric = <function track_metric.<locals>.track at 0x7f0e4c465f80>
track_phase = <function track_phase.<locals>.track at 0x7f0e4c465da0>
upload_assets_on_fail = <function upload_assets_on_fail.<locals>.collect at 0x7f0e4c466020>
request = <FixtureRequest for <Function test_run_benchmark[sentinel1_stats]>>

    @pytest.mark.parametrize(
        "scenario",
        [
            # Use scenario id as parameterization id to give nicer test names.
            pytest.param(uc, id=uc.id)
            for uc in get_benchmark_scenarios()
        ],
    )
    def test_run_benchmark(
        scenario: BenchmarkScenario,
        connection_factory,
        tmp_path: Path,
        track_metric,
        track_phase,
        upload_assets_on_fail,
        request,
    ):
        track_metric("scenario_id", scenario.id)

        with track_phase(phase="connect"):
            # Check if a backend override has been provided via cli options.
            override_backend = request.config.getoption("--override-backend")
            backend_filter = request.config.getoption("--backend-filter")
            if backend_filter and not re.match(backend_filter, scenario.backend):
                # TODO apply filter during scenario retrieval, but seems to be hard to retrieve cli param
                pytest.skip(
                    f"skipping scenario {scenario.id} because backend {scenario.backend} does not match filter {backend_filter!r}"
                )
            backend = scenario.backend
            if override_backend:
                _log.info(f"Overriding backend URL with {override_backend!r}")
                backend = override_backend

            connection: openeo.Connection = connection_factory(url=backend)

        with track_phase(phase="create-job"):
            # TODO #14 scenario option to use synchronous instead of batch job mode?
            job = connection.create_job(
                process_graph=scenario.process_graph,
                title=f"APEx benchmark {scenario.id}",
                additional=scenario.job_options,
            )
            track_metric("job_id", job.job_id)

        with track_phase(phase="run-job"):
            # TODO: monitor timing and progress
            # TODO: abort excessively long batch jobs? https://github.com/Open-EO/openeo-python-client/issues/589
            job.start_and_wait()
            # TODO: separate "job started" and run phases?

        with track_phase(phase="collect-metadata"):
            collect_metrics_from_job_metadata(job, track_metric=track_metric)

            results = job.get_results()
            collect_metrics_from_results_metadata(results, track_metric=track_metric)

        with track_phase(phase="download-actual"):
            # Download actual results
            actual_dir = tmp_path / "actual"
            paths = results.download_files(target=actual_dir, include_stac_metadata=True)

            # Upload assets on failure
            upload_assets_on_fail(*paths)

        with track_phase(phase="download-reference"):
            reference_dir = download_reference_data(
                scenario=scenario, reference_dir=tmp_path / "reference"
            )

        with track_phase(
            phase="compare", describe_exception=analyse_results_comparison_exception
        ):
            # Compare actual results with reference data
>           assert_job_results_allclose(
                actual=actual_dir,
                expected=reference_dir,
                tmp_path=tmp_path,
                rtol=scenario.reference_options.get("rtol", 1e-6),
                atol=scenario.reference_options.get("atol", 1e-6),
                pixel_tolerance=scenario.reference_options.get("pixel_tolerance", 0.0),
            )

tests/test_benchmarks.py:95:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

actual = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/actual')
expected = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference')

    def assert_job_results_allclose(
        actual: Union[BatchJob, JobResults, str, Path],
        expected: Union[BatchJob, JobResults, str, Path],
        *,
        rtol: float = _DEFAULT_RTOL,
        atol: float = _DEFAULT_ATOL,
        pixel_tolerance: float = _DEFAULT_PIXELTOL,
        tmp_path: Optional[Path] = None,
    ):
        """
        Assert that two job results sets are equal (with tolerance).

        :param actual: actual job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param expected: expected job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param rtol: relative tolerance
        :param atol: absolute tolerance
        :param pixel_tolerance: maximum fraction of pixels (in percent)
            that is allowed to be significantly different (considering ``atol`` and ``rtol``)
        :param tmp_path: root temp path to download results if needed.
            It's recommended to pass pytest's `tmp_path` fixture here
        :raises AssertionError: if not equal within the given tolerance

        .. versionadded:: 0.31.0

        .. warning::
            This function is experimental and subject to change.
        """
        issues = _compare_job_results(
            actual, expected, rtol=rtol, atol=atol, pixel_tolerance=pixel_tolerance, tmp_path=tmp_path
        )
        if issues:
>           raise AssertionError("\n".join(issues))
E           AssertionError: Issues for metadata file 'job-results.json':
E           Differing 'derived_from' links (0 common, 31 only in actual, 30 only in expected):
E             only in actual: {'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230715T171453_20230715T171518_049437_05F1DD_BA46', 'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230708T172324_20230708T172349_049335_05EEBC_2768', 'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230614T172323_20230614T172348_048985_05E406_C341', 'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230626T172323_20230626T172348_049160_05E954_34B8', 'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230620T052756_20230620T052821_049065_05E675_4DFC', 'https://services.sentinel-hub.com/api/v1/catalog/1.0.0/collections/sentinel-1-grd/items/S1A_IW_GRDH_1SDV_20230527T052755_20230527T052820_048715_05DBDD_CABC', 'https://services.sentinel-hub.com/api/v1/...
E             only in expected: {'/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/05/20/S1A_IW_GRDH_1SDV_20230520T053607_20230520T053632_048613_05D8C9_9AB5_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/06/01/S1A_IW_GRDH_1SDV_20230601T053608_20230601T053633_048788_05DE01_1BF4_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/07/19/S1A_IW_GRDH_1SDV_20230719T053610_20230719T053635_049488_05F366_9F2C_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/05/09/S1A_IW_GRDH_1SDV_20230509T172321_20230509T172346_048460_05D44B_3E56_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/07/26/S1A_IW_GRDH_1SDV_20230726T052758_20230726T052823_049590_05F68C_0A7F_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/07/20/S1A_IW_GRDH_1SDV_20230720T172325_20230720T172350_049510_05F40F_6113_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/07/14/S1A_IW_GRDH_1SDV_20230714T052758_20230714T052823_049415_05F130_0DBB_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2023/05/21/S1A_IW_GRDH_1SDV_20230521T172322_20230521T1....
E           Issues for file 'openEO.tif':
E           Left and right DataArray objects are not close
E           Differing values:
E           L
E               array([[[5.372932e-03, 1.490587e-04, ..., 8.591816e-03, 7.395972e-03],
E                       [2.654563e-03, 7.464876e-04, ..., 1.259500e-02, 1.754427e-02],
E                       ...,
E                       [2.604710e-05, 9.779212e-05, ..., 1.616370e-04, 4.526410e-04],
E                       [3.298784e-04, 7.654505e-05, ..., 3.207353e-05, 2.174394e-04]],
E
E                      [[1.026305e-01, 8.118022e-02, ..., 2.330827e-01, 3.507939e-01],
E                       [1.302835e-01, 1.302835e-01, ..., 3.507939e-01, 2.379648e-01],
E                       ...,
E                       [3.288277e-03, 7.816153e-03, ..., 6.296011e-02, 7.032564e-02],
E                       [5.563086e-03, 4.360230e-03, ..., 1.989181e-02, 6.659283e-02]],
E
E                      ...,
E
E                      [[9.895720e-02, 4.942647e-02, ..., 1.912273e-01, 2.575457e-01],
E                       [7.155539e-02, 5.260691e-02, ..., 2.461407e-01, 3.392899e-01],
E                       ...,
E                       [6.104632e-03, 4.811754e-03, ..., 1.314020e-02, 1.778121e-02],
E                       [5.298938e-03, 4.925177e-03, ..., 1.265550e-02, 1.437642e-02]],
E
E                      [[2.629619e-01, 1.828120e-01, ..., 5.444775e-01, 5.842698e-01],
E                       [1.976455e-01, 2.603058e-01, ..., 7.183174e-01, 6.919501e-01],
E                       ...,
E                       [1.284984e-02, 1.217198e-02, ..., 3.427796e-02, 4.603717e-02],
E                       [1.328379e-02, 1.367152e-02, ..., 3.276055e-02, 3.559327e-02]]],
E                     shape=(14, 558, 798), dtype=float32)
E           R
E               array([[[5.179015e-03, 6.743506e-04, ..., 6.575143e-03, 8.268312e-03],
E                       [2.680840e-03, 6.886612e-04, ..., 4.640952e-03, 1.225170e-02],
E                       ...,
E                       [1.069151e-05, 7.473860e-06, ..., 3.684950e-04, 7.085098e-05],
E                       [2.801024e-06, 1.215399e-06, ..., 2.321320e-04, 3.232815e-04]],
E
E                      [[4.369812e-02, 6.139704e-02, ..., 2.357572e-01, 1.049461e-01],
E                       [7.613949e-02, 1.043061e-01, ..., 1.248312e-01, 7.347236e-02],
E                       ...,
E                       [3.332000e-03, 6.064222e-03, ..., 5.764530e-02, 6.981783e-02],
E                       [4.810988e-03, 4.887321e-03, ..., 4.950251e-02, 8.110013e-02]],
E
E                      ...,
E
E                      [[6.717545e-02, 3.863911e-02, ..., 1.333968e-01, 1.126662e-01],
E                       [5.144660e-02, 4.761094e-02, ..., 1.429282e-01, 1.321035e-01],
E                       ...,
E                       [5.185403e-03, 4.838860e-03, ..., 1.324176e-02, 1.717496e-02],
E                       [5.710513e-03, 4.613311e-03, ..., 1.496293e-02, 1.626071e-02]],
E
E                      [[1.130307e-01, 1.201183e-01, ..., 5.564473e-01, 3.850443e-01],
E                       [1.332577e-01, 1.689987e-01, ..., 4.976607e-01, 3.429892e-01],
E                       ...,
E                       [1.394050e-02, 1.144980e-02, ..., 3.275811e-02, 3.861197e-02],
E                       [1.121384e-02, 1.270060e-02, ..., 3.564724e-02, 3.735264e-02]]],
E                     shape=(14, 558, 798), dtype=float32)
E           band 1: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0039566404884681e-06, max: 8.78488028049469, mean: 0.0, var: 0.0
E           band 1: differing pixels: 444401/445284 (99.8%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 2: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0104849934577942e-06, max: 501.474609375, mean: 0.09, var: 4.07
E           band 2: differing pixels: 445224/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 3: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0030344128608704e-06, max: 92.63527202606201, mean: 0.02, var: 0.18
E           band 3: differing pixels: 445138/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 4: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0060612112283707e-06, max: 153.75707817077637, mean: 0.02, var: 0.4
E           band 4: differing pixels: 445117/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 5: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0016956366598606e-06, max: 11.619417607784271, mean: 0.0, var: 0.0
E           band 5: differing pixels: 444926/445284 (99.9%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 6: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0067597031593323e-06, max: 40.20387780666351, mean: 0.01, var: 0.02
E           band 6: differing pixels: 445063/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 7: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.041218638420105e-06, max: 357.26934814453125, mean: 0.05, var: 2.0
E           band 7: differing pixels: 445197/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 8: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0009316611103714e-06, max: 26.900147199630737, mean: 0.02, var: 0.01
E           band 8: differing pixels: 445184/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 9: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.1138617992401123e-06, max: 10620.40673828125, mean: 1.83, var: 1753.34
E           band 9: differing pixels: 445275/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 10: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0104849934577942e-06, max: 2475.5344848632812, mean: 0.2, var: 58.2
E           band 10: differing pixels: 445173/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 11: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.014210283756256e-06, max: 3974.6932373046875, mean: 0.41, var: 171.23
E           band 11: differing pixels: 445228/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 12: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0067597031593323e-06, max: 27.79974865913391, mean: 0.03, var: 0.01
E           band 12: differing pixels: 445201/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 13: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0123476386070251e-06, max: 229.0928373336792, mean: 0.07, var: 0.68
E           band 13: differing pixels: 445196/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area
E           band 14: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0747462511062622e-06, max: 9809.591064453125, mean: 0.64, var: 848.56
E           band 14: differing pixels: 445250/445284 (100.0%), bbox ((485665.0, 4916305.0), (493635.0, 4921875.0)) - 100.0% of the area

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/openeo/testing/results.py:515: AssertionError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': send 'start'
0:00:10 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': created (progress 0%)
0:00:16 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': queued (progress 0%)
0:00:22 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': queued (progress 0%)
0:00:31 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': queued (progress 0%)
0:00:41 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': queued (progress 0%)
0:00:53 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': queued (progress 0%)
0:01:09 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': running (progress 9.1%)
0:01:28 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': running (progress 11.7%)
0:01:52 Job 'terrascope-j-2510201323334ec4b66fbebd33a2c2f6': finished (progress 100%)
----------------------------- Captured stderr call -----------------------------
Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.
Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.
------------------------------ Captured log call -------------------------------
INFO     conftest:conftest.py:131 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO     openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO     conftest:conftest.py:144 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO     conftest:conftest.py:148 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO     openeo.rest.connection:connection.py:255 Found OIDC providers: ['CDSE']
INFO     openeo.rest.auth.oidc:oidc.py:404 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:352 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.job:job.py:436 Downloading Job result asset 'openEO.tif' from https://openeo-cdse.vito.be/openeo/1.1/jobs/j-2510201323334ec4b66fbebd33a2c2f6/results/assets/NmE3N2ZjZDEtOWMwOC00NmU5LWI4NzUtNTRmYjk5OWFiMjAw/7115bf5bfe514f79289844c4e4af6581/openEO.tif?expires=1761571527 to /home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/actual/openEO.tif
INFO     apex_algorithm_qa_tools.scenarios:util.py:345 Downloading reference data for scenario.id='sentinel1_stats' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference'): start 2025-10-20 13:25:42.624782
INFO     apex_algorithm_qa_tools.scenarios:util.py:345 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-17378532865!tests_test_benchmarks.py__test_run_benchmark_sentinel1_stats_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference/job-results.json'): start 2025-10-20 13:25:42.625133
INFO     apex_algorithm_qa_tools.scenarios:util.py:351 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-17378532865!tests_test_benchmarks.py__test_run_benchmark_sentinel1_stats_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference/job-results.json'): end 2025-10-20 13:25:43.472505, elapsed 0:00:00.847372
INFO     apex_algorithm_qa_tools.scenarios:util.py:345 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-11743427213!tests_test_benchmarks.py__test_run_benchmark_sentinel1_stats_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference/openEO.tif'): start 2025-10-20 13:25:43.472963
INFO     apex_algorithm_qa_tools.scenarios:util.py:351 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-11743427213!tests_test_benchmarks.py__test_run_benchmark_sentinel1_stats_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference/openEO.tif'): end 2025-10-20 13:25:46.131666, elapsed 0:00:02.658703
INFO     apex_algorithm_qa_tools.scenarios:util.py:351 Downloading reference data for scenario.id='sentinel1_stats' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference'): end 2025-10-20 13:25:46.131912, elapsed 0:00:03.507130
INFO     openeo.testing.results:results.py:418 Comparing job results: PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/actual') vs PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_sentinel1_s0/reference')
WARNING  rasterio._env:__init__.py:356 CPLE_AppDefined in openEO.tif: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.
WARNING  rasterio._env:__init__.py:356 CPLE_AppDefined in openEO.tif: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 1:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 2:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│.................................................'............│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 3:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│.................................................'............│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 4:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 5:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 6:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│.................................................'............│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 7:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 8:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 9:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 10:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 11:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 12:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 13:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 14:
┌──────────────────────────────────────────────────────────────┐
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
│..............................................................│
└──────────────────────────────────────────────────────────────┘

[jdries]

@soxofaan this failure occured because the job got routed to terrascope, while it should always end up on CDSE.
The SENTINEL1_GRD collection on terrascope is 'hidden' on openeofed.

If CDSE is down, then this could still be acceptable behaviour, but afaik cdse is up most of the time.

[soxofaan]

quick investigation:

The aggregator receives this process graph here:

{
  "s1stats1": {
    "process_id": "sentinel1_stats",
    "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/refs/heads/main/algorithm_catalog/vito/sentinel1_stats/openeo_udp/sentinel1_stats.json",
    "arguments": {...   }
  }
}

And the aggregator does not "unroll" that UDP reference yet to detect the "best" backend (e.g. based on collection in this case)

-> ticket

• Unroll UDPs when trying to detect best backend to dispatch to Open-EO/openeo-aggregator#194

[github-actions]

Report of latest run:

Benchmark scenario ID: sentinel1_stats
Benchmark scenario definition: https://github.com/ESA-APEx/apex_algorithms/blob/b43116632be5e3cd6150fbff5ae21cb135763654/algorithm_catalog/vito/sentinel1_stats/benchmark_scenarios/sentinel1_stats.json
openEO backend: openeofed.dataspace.copernicus.eu

GitHub Actions workflow run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18811635897
Workflow artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18811635897#artifacts

Test start: 2025-10-26 02:21:26.689839+00:00
Test duration: 0:04:07.042862
Test outcome: ✅ passed