Some Collections in pygeoapi master and stable endpoints give error on queryables

[justb4]

Problem

On the demo server, log in and execute docker logs --follow pygeoapi_stable. This gives errors like:

[2024-03-27T14:23:58Z] {/usr/local/lib/python3.10/dist-packages/flask/app.py:828} ERROR - Exception on /collections/mapserver_world_map/queryables [GET]
Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api.py", line 1508, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api.py", line 1513, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/pygeoapi/pygeoapi/flask_app.py", line 213, in collection_queryables
    return get_response(api_.get_collection_queryables(request, collection_id))
  File "/pygeoapi/pygeoapi/api.py", line 228, in inner
    headers, status, content = func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 210, in inner
    return func(cls, req_out, *args[2:])
  File "/pygeoapi/pygeoapi/linked_data.py", line 60, in inner
    return func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 1517, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

Analysis

• Cannot reproduce these errors the https://demo.pygeoapi.io/stable/collections/mapserver_world_map seems to work.
• Some other remote client seems to trigger the errors, also for some other collections, not all, e.g. https://demo.pygeoapi.io/stable/collections/dutch_windmills/queryables works ok.
• Though taking from the error this gives an Exception: https://demo.pygeoapi.io/stable/collections/collections/mapserver_world_map/queryables
• Also on master endpoint: https://demo.pygeoapi.io/master/collections/mapserver_world_map/queryables
• As Update config for stable release service endpoint #52 has copied the config from the master endpoint, is there a facility (queryables?) not yet present in stable pygeoapi v0.16.0?

[justb4]

Edit: the problem is triggered from a remote client: https://qswega93.muc.moss.itn/wega-client/ not sure why it is issueing so many requests continuously.

[tomkralidis]

@justb4 is this still an issue?

[justb4]

Also today still see many errors, but looks like misconfigured clients. e.g. Requesting /collections/mapserver_world_map/queryables . Well /collections/mapserver_world_map exists, but is a 'OGC API Maps' endpoint (right?) without queryables. But well, if you run any server on the internet you always get a hose of requests, many probing vulnerabilities, these are caught at the Traefik, proxy level.
Will take quite some time to investigate but maybe for no use, this is a demo server after all...

[2024-07-24T13:10:04Z] {/pygeoapi/pygeoapi/api/__init__.py:1395} ERROR - Collection not found
[2024-07-24T13:33:01Z] {/usr/local/lib/python3.10/dist-packages/flask/app.py:838} ERROR - Exception on /collections/mapserver_world_map/queryables [GET]
Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 126, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 131, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_cors/extension.py", line 178, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/pygeoapi/pygeoapi/flask_app.py", line 261, in collection_queryables
    return execute_from_flask(itemtypes_api.get_collection_queryables, request,
  File "/pygeoapi/pygeoapi/flask_app.py", line 160, in execute_from_flask
    headers, status, content = api_function(api_, api_request, *args)
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 135, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested
[2024-07-24T13:33:01Z] {/usr/local/lib/python3.10/dist-packages/flask/app.py:838} ERROR - Exception on /collections/icoads-sst/queryables [GET]
Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 126, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 131, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 561, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_cors/extension.py", line 178, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/pygeoapi/pygeoapi/flask_app.py", line 261, in collection_queryables
    return execute_from_flask(itemtypes_api.get_collection_queryables, request,
  File "/pygeoapi/pygeoapi/flask_app.py", line 160, in execute_from_flask
    headers, status, content = api_function(api_, api_request, *args)
  File "/pygeoapi/pygeoapi/api/itemtypes.py", line 135, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 564, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested
[2024-07-24T14:57:10Z] {/

[tomkralidis]

@justb4 yes looks like a misconfigured client. If a collection does not have queryables, we should throw a 400.

I've issued a bugfix at geopython/pygeoapi#1757 to address.

[tomkralidis]

PR merged. We should be able test once the demo VM is triggered/updated.