Support for pmtiles S3 urls

[alamminsalo]

Added support for s3-urls to config.

This is a bit of a rough sketch currently.
The aws config struct does not seem to support sso-login credentials out of the box, I needed to add the credentials to environment variables along with AWS_REGION to get it working.

[nyurik]

This seems to be almost ready! The biggest challenge is ... sadly... how to test it? I wonder if we should set up some public s3 bucket with some small dummy pmtiles and use it in all the testing? @birkskyum @louwers any thoughts on where we can place it?

P.S. please rebase

[louwers]

https://pmtilestest.s3.eu-central-1.amazonaws.com/cb_2018_us_zcta510_500k.pmtiles

This is in a MapLibre owned bucket.

[nyurik]

@louwers why is a public readonly s3 bucket with only a small downloadable is a bad idea? Seems like it would be no different than exposing that same bucket with the http interface, and it would let us test s3 API

[louwers]

@nyurik I misunderstood and removed my earlier comment, I thought we need a S3 bucket for various kinds of test assets. Please use the link I provided! :)

[nyurik]

@louwers we do want to upload a bunch of test assets to an s3 bucket (by an admin), but that bucket would be used as a readonly source when running CI tests

[nyurik]

@louwers the link you posted above - do you have a full s3:// path to it?

[louwers]

That would be s3://pmtilestest/cb_2018_us_zcta510_500k.pmtiles

[CommanderStorm]

few nits about formatting in the docs

[alamminsalo]

I migrated to using pmtiles aws-s3-async feature, which uses aws-sdk-s3 and aws-config libraries.
They have better credentials configuration flexibility (eg. can use SSO credentials) and does not need the AWS_REGION environment variable to be set explicitly (can be also defined in profiles).
The support for them seems to have landed in pmtiles 0.11.

Edit: also added tests for the S3 backed based on your suggestions

[CommanderStorm]

The testcase for grcov is failing..

You likely need the same environment variables here:

martin/.github/workflows/grcov.yml

Line 62 in 321af56

CARGO_INCREMENTAL: '0'

[CommanderStorm]

@nyurik not sure what our policy on this is:
Which testcases should be added to

martin/tests/test.sh

Line 271 in 321af56

>&2 echo "***** Test server response for PMTiles source *****"

Is this one too specific or still fine to add?

[CommanderStorm]

Seems that even after adressing 94f8d43, there is one other issue that prevents this being merged.

I or @alamminsalo will need to investigate where this dying is coming from.