Simplify config getter and override system

[soxofaan]

This came up while trying to get Open-EO/openeo-geopyspark-driver#701 working:

configs defined in openeo-python-driver's OpenEoBackendConfig and used in openeo-python-driver can not be overriden from e.g. openeo-geopyspark-driver's gps_config_overrides as openeo-python-driver and openeo-geopyspark-driver have separate ConfigGetter instances and config_overrides only overrides one of them.

It probably pays off to simplify the config getter system a bit: just have a single ConfigGetter instance in the application. This implies that e.g. openeo-geopyspark-driver would overwrite the default one of openeo-python-driver.
Related: part of the simplification could also involve eliminating the subclass GpsConfigGetter

[soxofaan]

@pvbouwel got bitten by this one too

[pvbouwel]

I was trying out some workarounds suggested and used it in the backend_implementation fixture without success.
Only realised a bit late that that fixture returns backend when changing that to a yield I had better luck.

But rather than putting all the logic in that fixture it is likely cleaner to split it off

@pytest.fixture
def config_overrides() -> dict:
    # Default implementation does not have overrides
    return {}


@pytest.fixture
def set_config_overrides(config_overrides) -> typing.Generator[GpsBackendConfig, None, None]:
    openeo_driver_overrides = {}
    for config_key, config_override_value in config_overrides.items():
        if hasattr(OpenEoBackendConfig, config_key):
            openeo_driver_overrides[config_key] = config_override_value

    geopyspark_driver_overrides = {}
    for config_key, config_override_value in config_overrides.items():
        if hasattr(OpenEoBackendConfig, config_key):
            openeo_driver_overrides[config_key] = config_override_value
        elif hasattr(openeogeotrellis.config.GpsBackendConfig, config_key):
            geopyspark_driver_overrides[config_key] = config_override_value
        else:
            raise ValueError(f"config value {config_key} cannot be attributed to a config")
    with openeo_driver.testing.config_overrides(**openeo_driver_overrides):
        with openeogeotrellis.testing.gps_config_overrides(**geopyspark_driver_overrides, **openeo_driver_overrides):
            yield get_backend_config()

And use set_config_overrides argument in the backend_implementation fixture

[soxofaan]

sorry for late response, but I'm not completely following the intended scope of that suggestion. It involves GpsBackendConfig, but the openeo-python-driver code should be agnostic of that. Note that there are also other OpenEoBackendConfig subclasses (e.g. in openeo-aggregator), and we can not reasonably manage that from the openeo-python-driver side

[soxofaan]

I just noticed these kind of warnings in openeo-geopyspark-driver test runs:

WARNING openeo_driver.config.base:base.py:53 Ignoring invalid config arguments: {'job_dependencies_poll_interval_seconds', 'default_executor_memoryOverhead', 'default_soft_errors', 'gdalinfo_use_python_subprocess', 'etl_source_id', 'udp_registry_zookeeper_client_reuse', 'opensearch_enrich', 'batch_job_base_fee_credits', 'openeo_idp_details_file', 'udf_dependencies_pypi_index', 'setup_kerberos_auth', 'etl_api_config', 'vault_addr', 'batch_user_docker_mounts', 'zookeeper_hosts', 'calrissian_config', 'report_usage_sentinelhub_pus', 'fuse_mount_batchjob_s3_bucket', 'batch_job_config_dir', 'layer_catalog_files', 'udf_dependencies_install_mode', 's3_bucket_name', 'prometheus_api', 'gdalinfo_from_file', 'default_gdal_cachemax', 'udf_dependencies_sleep_after_install', 'capabilities_extras', 'zk_job_registry_max_specification_size', 'default_python_memory', 'default_opensearch_endpoint', 'etl_api', 'default_reading_strategy', 's3_region_proxy_endpoints', '...

which indicates there is something wrong with the config override system.

FYI: this warning can be made an exception (which makes it easier to drill down) by setting export OPENEO_CONFIG_STRICTNESS_MODE=strict