Tutorial for generating the planet tiles?

[landc0ver]

I want to generate the planet tiles and have several questions about this work:

1. how long will it take to generate?
2. what about cpu, disk and memory requirements?
3. is there any tutorial to do this work?

[amatissart]

Basically all you need is to import the osm planet file into postgres and launch the tiles generation. To run these 2 steps separately using the "local" configuration:

docker-compose -f docker-compose.yml -f local-compose.yml run --rm -e INVOKE_OSM_URL=https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf load_db
docker-compose -f docker-compose.yml -f local-compose.yml run --rm -e INVOKE_TILES_PLANET=1 -e INVOKE_TILES_PARTS=80 load_db generate-tiles

You can find more details about the tiles.planet option in the python tasks: https://github.com/QwantResearch/kartotherian_config/blob/master/import_data/tasks.py#L430
This will take care of creating tiles generation jobs with some optimizations (to prevent useless generation on oceans, etc).
tiles.parts breaks each job into independent jobs, allowing it to run in multiple workers and/or machines

The import part is mostly i/o bound, and takes between 36 and 48 hours to run.
The duration of tiles generation will depend on the number of tilerator workers you can use.
Using 30 tilerator workers (with at least 24 CPUs dedicated to the PostgreSQL database), it takes around 36 hours to complete.

I hope that helps. Unfortunately there is no production-ready step-by-step tutorial at this point, as it would deeply depend on the infrastructure you are using, how databases are deployed, etc.

[landc0ver]

The import part has spent over 2 days but it still doesn't finish. The PostgreSQL database has 8CPUs and 32G Memory. It seems to be stuck in the clustering process of basemap. Is that normal?
Can I stop this clustering process and continue to run the next step?

[May  8 05:48:54] [INFO] [PostGIS] Analysing osm_landuse_polygon took: 579.765365ms
[May  8 05:54:03] [INFO] [PostGIS] Clustering osm_highway_linestring_gen2 on geohash took: 16m24.522862466s
[May  8 05:54:03] Analysing osm_highway_linestring_gen2
[May  8 05:54:03] [INFO] [PostGIS] Analysing osm_highway_linestring_gen2 took: 743.127181ms
[May  8 06:00:35] [INFO] [PostGIS] Clustering osm_landcover_polygon_gen2 on geohash took: 19m26.147765175s
[May  8 06:00:35] Analysing osm_landcover_polygon_gen2
[May  8 06:00:36] [INFO] [PostGIS] Analysing osm_landcover_polygon_gen2 took: 772.572877ms
[May  8 06:11:18] [INFO] [PostGIS] Clustering osm_highway_linestring_gen1 on geohash took: 27m46.401657897s
[May  8 06:11:18] Analysing osm_highway_linestring_gen1
[May  8 06:11:19] [INFO] [PostGIS] Analysing osm_highway_linestring_gen1 took: 918.753653ms
[May  8 06:12:04] [INFO] [PostGIS] Clustering osm_water_polygon on geohash took: 37m28.851767015s[May  8 06:12:04] Analysing osm_water_polygon
[May  8 06:12:04] [INFO] [PostGIS] Analysing osm_water_polygon took: 703.853971ms
[May  8 06:14:04] [INFO] [PostGIS] Indexing osm_highway_linestring on geohash took: 45m3.579287814s
[May  8 06:14:04] Clustering osm_highway_linestring on geohash
[May  8 06:15:12] [INFO] [PostGIS] Clustering osm_landcover_polygon_gen1 on geohash took: 35m56.964968188s
[May  8 06:15:12] Analysing osm_landcover_polygon_gen1
[May  8 06:15:13] [INFO] [PostGIS] Analysing osm_landcover_polygon_gen1 took: 605.185017ms
[May  8 07:09:45] [INFO] [PostGIS] Indexing osm_building_polygon on geohash took: 1h40m21.770888504s
[May  8 07:09:45] Clustering osm_building_polygon on geohash
[May  8 07:37:42] [INFO] [PostGIS] Clustering osm_waterway_linestring on geohash took: 2h0m40.01494207s
[May  8 07:37:42] Analysing osm_waterway_linestring
[May  8 07:37:43] [INFO] [PostGIS] Analysing osm_waterway_linestring took: 670.281878ms
[May  8 08:19:10] [INFO] [PostGIS] Clustering osm_building_multipolygon on geohash took: 2h40m11.508736618s
[May  8 08:19:10] Analysing osm_building_multipolygon
[May  8 08:19:11] [INFO] [PostGIS] Analysing osm_building_multipolygon took: 479.767429ms
[May  8 09:54:28] [INFO] [PostGIS] Clustering osm_landcover_polygon on geohash took: 4h10m43.881990422s
[May  8 09:54:28] Analysing osm_landcover_polygon
[May  8 09:54:28] [INFO] [PostGIS] Analysing osm_landcover_polygon took: 504.374453ms
[May  8 23:36:57] [INFO] [PostGIS] Clustering osm_highway_linestring on geohash took: 17h22m53.626420629s
[May  8 23:36:57] Analysing osm_highway_linestring
[May  8 23:37:04] [INFO] [PostGIS] Analysing osm_highway_linestring took: 7.108340863s