[Bug] Upgrade from 21.4.3-v1 to debian-master not possible

[enterdv]

Describe the bug
Upgrade from 21.4.3-v1 to debian-master not working. On alpine image PostgreSQL version 13. On Debian PostgreSQL version 11.14.

To Reproduce
Steps to reproduce the behavior:

1. In docker-compose change image version from 21.4.3-v1 to debian-master.
2. Run docker-compose up
3. postgresql not starting

gvm_1  | + pg_isready --username=postgres
gvm_1  | 2021-11-29 10:25:53,180 INFO exited: postgresql (exit status 1; not expected)
gvm_1  | + sleep 1
gvm_1  | 2021-11-29 10:25:53,192 INFO gave up: postgresql entered FATAL state, too many start retries too quickly

PostgreSQL logs

2021-11-29 10:25:53.180 GMT [716] FATAL:  database files are incompatible with server
2021-11-29 10:25:53.180 GMT [716] DETAIL:  The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 11.14 (Debian 11.14-0+deb10u1).

Expected behavior
PostgreSQL running

Screenshots
NA

Host Device:

• OS: Debian
• Version: 10

Image in use:
securecompliance/gvm:debian-master

[markdesilva]

PG is 11.14? Strange, since the last version of PG on the pre 21.4.0-v5 tags was 12.

Did you try to go into the 21.4.3-v1 container, stop gvm and do a pg_dumpall to get the sql dump file then try to import the sql dump file into the debian container?

[enterdv]

PG is 11.14? Strange, since the last version of PG on the pre 21.4.0-v5 tags was 12.

Did you try to go into the 21.4.3-v1 container, stop gvm and do a pg_dumpall to get the sql dump file then try to import the sql dump file into the debian container?

No, because postgresql won't start, so I can't do pg_restore. I try it with new running docker-compose without any data.

[enterdv]

OK, I start new empty docker-compose with debian, then stop all gvm services, and restore postgresql database from previous version. Start services. All tasks/reports/etc seems OK, but scan not working with error
libgvm util:WARNING:2021-11-29 15h19.41 UTC:1073: error decrypting credential: No secret key <GPGME> libgvm util: INFO:2021-11-29 15h19.41 UTC:1073: encrypted to keyid 6D74FF2A58FC30A9, algo=1: No secret key <GPGME> libgvm util:WARNING:2021-11-29 15h19.41 UTC:1073: error decrypting credential: No secret key <GPGME> libgvm util: INFO:2021-11-29 15h19.41 UTC:1073: encrypted to keyid 6D74FF2A58FC30A9, algo=1: No secret key <GPGME>

UPD: Copy /var/lib/gvm/gvmd/gnupg/ from previous GVM setup, now scan works fine

[markdesilva]

Glad it works!

Quick question, did you start the debian-master with the mounted volumes

--volume ./storage/postgres-db:/opt/database --volume ./storage/openvas-plugins:/var/lib/openvas/plugins --volume ./storage/gvm:/var/lib/gvm --volume ./storage/ssh:/etc/ssh

or using

--volume gvm-data:/

Thanks!

[enterdv]

I start like this

volumes:
  - /gvm_data/database:/opt/database
  - /gvm_data/lib:/var/lib/gvm
  - /gvm_data/plugins:/var/lib/openvas/plugins
  - /gvm_data/ssh:/etc/ssh

[markdesilva]

Thank you!

I start like this

volumes:
  - /gvm_data/database:/opt/database
  - /gvm_data/lib:/var/lib/gvm
  - /gvm_data/plugins:/var/lib/openvas/plugins
  - /gvm_data/ssh:/etc/ssh

[Dexus]

If you need to "downgrade" please read this: https://dba.stackexchange.com/questions/284260/how-to-prevent-pg-dump-from-generating-set-default-table-access-method-heap/284263#284263

There you get some Info how to do this - not very handy but works.

[enterdv]

If you need to "downgrade" please read this: https://dba.stackexchange.com/questions/284260/how-to-prevent-pg-dump-from-generating-set-default-table-access-method-heap/284263#284263

There you get some Info how to do this - not very handy but works.

Thanks for your reply! I build my own debian image based on yours, but with postgresql-13.

[Dexus]

Thanks for your reply! I build my own debian image based on yours, but with postgresql-13.

Thats also good!

[jonasdlm]

Is it possible to add the fix of @enterdv to install postgresql-13 instead of version 11? I'm having the same problem when updating.
Thanks!

[devMarc5598]

Is it possible to add the fix of @enterdv to install postgresql-13 instead of version 11? I'm having the same problem when updating. Thanks!

Any Solutions now?

[AjaCoder]

Same issue here:

2022-01-27 10:15:42,143 INFO exited: postgresql (exit status 1; not expected)
postgresql: ERROR (spawn error)
...
2022-01-27 10:16:38,976 INFO gave up: postgresql entered FATAL state, too many start retries too quickly
pg_isready --username=postgres
2022-01-27 10:16:38,955 INFO exited: postgresql (exit status 1; not expected)

[FiveOFive]

Not sure if this will help everyone or just me, but for postgresql (exist status 1; not expected) issues try checking the supervisor logs in the container. Steps:

1. Start a bash session in the container - docker exec -it <container name> /bin/bash
2. Dump logs - cat /var/log/supervisor/postgresql_err.log

In my case I found there were incorrect permissions for the data directory:

FATAL: data directory "/opt/database" has invalid permissions`
DETAIL: Permisions should be u=rwx (0700) or u=rwx,g=rx (0750).

Fixed by setting the permission chmod 0700 /opt/database and then restarting the container.

[jonasdlm]

Thanks for the suggestion, but my error has nothing to do with the permissions. The version of postgresql in the previous container was 13 and in the Debian version it is postgresql 11.
2022-02-25 15:50:48.187 CET [154] FATAL: database files are incompatible with server 2022-02-25 15:50:48.187 CET [154] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 11.14 (Debian 11.14-0+deb10u1). 2022-02-25 15:50:49.221 CET [155] FATAL: database files are incompatible with server 2022-02-25 15:50:49.221 CET [155] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 11.14 (Debian 11.14-0+deb10u1). 2022-02-25 15:50:51.328 CET [160] FATAL: database files are incompatible with server 2022-02-25 15:50:51.328 CET [160] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 11.14 (Debian 11.14-0+deb10u1).

[Dexus]

You can now use the https://hub.docker.com/r/deineagenturug/gvm images, which are using the latest gvmd and components releases.

It use POSTGRESQL 13.

What are meaning the tags:

docker pull deineagenturug/gvm:latest               # is an not initialized basic image, that first start its initialization on the first start (short: no db & no PDF report support)
docker pull deineagenturug/gvm:latest-full          # is an not initialized basic image, that first start its initialization on the first start (short: no db, PDF report is support)
docker pull deineagenturug/gvm:latest-data          # is an initialized basic image, that can run directly without extra VOLUMES added (short: included db, but no PDF report support)
docker pull deineagenturug/gvm:latest-data-full     # is an initialized basic image, that can run directly without extra VOLUMES added (short: included db & PDF report is support)

It can be used as direct replacement for the here provided image, with the same arguments on docker run.

---

⚠️ I have not yet tested any migration from Alpine images, but while nothing has changed, from the path and the PostgreSQL is now also version 13 it should work without any problems. Let me know if this help you.

[Dexus]

For further support, use https://github.com/DeineAgenturUG/greenbone-gvm-openvas-for-docker

[Dexus]

We have a "development" DB Upgrade tool for the migration from here to my deineagenturug/gvm.

Details you can read here: DeineAgenturUG/greenbone-gvm-openvas-for-docker#16 (comment)

Any feedback in the issue on my repo is welcome.