Not able to import europe map

[garysacca]

Hello All,

I was trying to import Europe map into Nominatim docker using a VM from Google (e2-standard-16) it would be 16 CPUs and 64 G RAM, despite I have set a bunch of resources container is exited after a while with not clear error message.

`+ trap stopServices TERM

• /app/config.sh
• id nominatim
• useradd -m -p geotab_nominatim nominatim
• IMPORT_FINISHED=/var/lib/postgresql/12/main/import-finished
• TOKENIZER_DIR=/nominatim/tokenizer
• '[' '!' -f /var/lib/postgresql/12/main/import-finished ']'
• /app/init.sh
• OSMFILE=/nominatim/data.osm.pbf
• '[' false = true ']'
• echo 'Skipping optional Wikipedia importance import'
  Skipping optional Wikipedia importance import
• '[' '' = true ']'
• echo 'Skipping optional GB postcode import'
  Skipping optional GB postcode import
• '[' '' = true ']'
• echo 'Skipping optional US postcode import'
  Skipping optional US postcode import
• '[' '' '!=' '' ']'
• '[' /nominatim/data/europe-latest.osm.pbf '!=' '' ']'
• echo Reading OSM extract from /nominatim/data/europe-latest.osm.pbf
  Reading OSM extract from /nominatim/data/europe-latest.osm.pbf
• OSMFILE=/nominatim/data/europe-latest.osm.pbf
• '[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
• sudo service postgresql start

• Starting PostgreSQL 12 database server [ OK ]

• sudo -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='''nominatim''''
• grep -q 1
• sudo -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='''www-data''''
• grep -q 1
• sudo -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '''geotab_nominatim''''
  ALTER ROLE
• sudo -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '''geotab_nominatim''''
  ALTER ROLE
• sudo -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
  DROP DATABASE
• chown -R nominatim:nominatim /nominatim
• cd /nominatim
• sudo -E -u nominatim nominatim import --osm-file /nominatim/data/europe-latest.osm.pbf --threads 16
  2022-02-08 11:07:42: Using project directory: /nominatim
  2022-02-08 11:07:43: Creating database
  2022-02-08 11:07:44: Setting up country tables
  2022-02-08 11:07:46: Importing OSM data file
  2022-02-08 11:07:46 osm2pgsql version 1.5.1
  2022-02-08 11:07:46 Database version: 12.9 (Ubuntu 12.9-0ubuntu0.20.04.1)
  2022-02-08 11:07:46 PostGIS version: 3.0
  2022-02-08 11:07:46 Parsing gazetteer style file '/usr/local/etc/nominatim/import-full.style'.
  NOTICE: table "place" does not exist, skipping`

I have tried to change memory for Postgres and either set some fix configs for container but I have always same outcome. Im lacking out ideas, since I already created a local volume and bind it with the docker, either way Im trying to use flatnode and PBF is download on machine too.

docker run -it --rm -e PBF_PATH=/nominatim/data/europe-latest.osm.pbf -e IMPORT_WIKIPEDIA=false -e NOMINATIM_PASSWORD=geotab_nominatim -v /startup/nominatim-data/postgres:/var/lib/postgresql/12/main/ -v /startup/nominatim-data/flatnode:/nominatim/flatnode -e NOMINATIM_FLATNODE_FILE=/nominatim/flatnode/flatnode.file -v /startup/nominatim-data/osm-maps:/nominatim/data -e POSTGRES_MAINTAINENCE_WORK_MEM=10GB -e POSTGRES_WORK_MEM=50MB -e POSTGRES_EFFECTIVE_CACHE_SIZE=24GB -e POSTGRES_SHARED_BUFFERS=50GB --shm-size=60G -p 8080:8080 --name nominatim --oom-kill-disable mediagis/nominatim:4.0

Anyone can help?

[mtmail]

The process stops when https://osm2pgsql.org/ imports data into Postgresql. Usually it's either running out of RAM, or the database server connection stops (e.g. network issue). Giving Postgresql more memory can have an adverse affect
https://nominatim.org/release-docs/latest/admin/Import/#notes-on-memory-usage

The most straight-forward approach, when using virtual cloud servers, is trying a machine setup with more RAM and reduce the RAM after the import is finished.