production related aspects

[iferencik]

Hello there,

I'd like to use this space to bring up some thoughts/aspects related to using martin in a production environment.

First of all, martin is already pretty good at this. However after attempting to use in my team (senior , intermediary and beginners) we realized we could really use a little bit of help with following:

1. have a config tool which would help with building a config file automatically
2. be able to select what tables/functions are going to be published and what are not
3. be able to specify what columns are going to be published and what are not
4. show a description property for a table/function
5. show the list of arguments for a given function source (similar to pg_tileserv or timvt)

Ideally martin would have an admin web interface where the users could log in (openID, etc) and configure the server (connect to DB and generate a config file). This would allow our interns to manage the service.

We made some small steps in this direction. I created a small python command line tool martin-config which is now publicly available here.

The tool leverages asyncpg and relies on interpreting information written into table/column comments to control what tables/layers are going to be included in the config file. While this is a good approach for deciding on columns, it probably is better in the long term to use postgres user rights (GRANT) privileges to decide in case of tables and functions.

I want to know your opinions/thoughts on implementing 4 and 5 in martin as I do not have rust know-how/experience.

4. Add a pub property description to FunctionSource struct which will be set in the config file or read from the COMMENT from the DB

5. add info about the arguments:

   • extend the FunctionSource struct to include args property (array)
   • add this info to config as arguments, example YAML

        arguments:
            - default:
              name: click_lon
              type: float8
            - default:
              name: click_lat
              type: float8
            - default: '500.0'
              name: radius
              type: float8

By convention the arguments with a default value are optional

No 5 will break some of existing code/concepts because, while it is easy to add this to the config file it is not possible (at least to my knowledge) to store/add this info to the function itself in the database, so it will not be possible to get a full FunctionSource object by running a query against the DB.

Thanks

[nyurik]

hi @iferencik , thx for good comments! I think some of them relate to my current rewrite of the configuration system (see #415 and a discussion #329 and possibly others). I don't think a web UI is in my personal roadmap, but PRs are obviously welcome. That said - I think it might make sense to stabilize the new feature set like multi-backend support and many other feature requests before implementing something more complex.

[iferencik]

hello @nyurik. It is clear to me the web UI is not in the list of priorities and it is needless to stress out the stabilization and multi-backend support are important.
At the same time however it is also important for martin to be used actively in production as this can be a source of ideas/features/docs.
For my team supporting descriptions for tables/functions and function arguments is very important because it will make the functions more intelligible and discoverable and we will probably try to take on this and for what it is worth we'll make a PR and the right time.

Thanks for all your work.

[nyurik]

@iferencik I agree about the other aspects of your proposal - I think it is very important to expand how PG can be queried, and make it easier to configure, i.e. run Martin in a "autodetect, save config, and exit" mode. Moreover, I was already hacking on a way to support OpenMapTiles querying style in the #380 draft PR. So my suggestion -- lets break your proposal up into smaller feature requests, and work on those.