Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redeployment unable to startup again #166

Closed
andrewklau opened this issue Mar 16, 2017 · 16 comments
Closed

Redeployment unable to startup again #166

andrewklau opened this issue Mar 16, 2017 · 16 comments

Comments

@andrewklau
Copy link

Updated the resource limits for a postgresql-persistent 9.5 deployment

pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
 done
server started
ERROR:  tuple already updated by self

It seems the first pod did not shutdown cleanly and left the PID in /var/lib/pgsql/data/userdata/postmaster.pid volume thus preventing the container from starting up automatically without manual intervention

Perhaps an edge case as this is the first time seeing this with many other postgresql deployments

@pedro-dlfa
Copy link

pedro-dlfa commented May 25, 2018

Hello, here facing the same problem: container unable to start, with same output detailed above.
[Is there / does anyone have found] any solution or workaround for this problem? I quickly tried removing /var/lib/pgsql/data/userdata/postmaster.pid file but when starting the container I'm getting the same issue.

EDIT: I double checked, and in my case the output is:

 pg_ctl: another server might be running; trying to start server anyway
 waiting for server to start....LOG:  redirecting log output to logging collector process
 HINT:  Future log output will appear in directory "pg_log".
 ... done
 server started
 => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
 ERROR:  tuple already updated by self

@praiskup
Copy link
Contributor

Thanks for the report. Interesting, @pedro-dlfa so you manually dropped the pid file, and immediately after that the container again refused to start (because of the pid file)? Smells like pg_ctl stop isn't really doing what we expect.

@martin123218
Copy link

Hello, I am facing the same issue as @pedro-dlfa , I tried to delete the file manually and redeploy pod but with no success. My solution is to recreate pod again. I am using version 9.5 of postgresql.

@praiskup
Copy link
Contributor

If you are affected by this, can you confirm that deployment strategy is Recreate?

@martin123218
Copy link

martin123218 commented Oct 19, 2018

Hi,
My strategy is Rolling and I was affected last week again.

@pkubatrh
Copy link
Member

Hi @martin123218

The problem with the Rolling strategy is that it tells Openshift to first create a new pod with the same data volume as the old one and only shut down the original pod when the new pod is up and running. Since there are at one time two pods accessing (and presumably writing to) the same data volume you can run into this issue.

Please use the Recreate strategy instead. There will be some downtime since the new pod is only started after the old pod gets shut down but you should not run into this issue anymore.

@bkabrda
Copy link

bkabrda commented Nov 16, 2018

I've also just run into this issue. Is there a way to make this work with "Rolling" strategy to have zero downtime upgrades?

@praiskup
Copy link
Contributor

Not with this trivial layout. This problem is equivalent to non-container scenario where you do dnf update postgresql-server. You have to shut down the old server, and start a new one. I.e. you can not let two servers write into the same data directory.

Btw., PostgreSQL server has a guard against "multiple servers writing to the same data directory" situation, but unfortunately in container scenario - it has deterministic PID number (PID=1). So concurrent PostgreSQL server (in different container) checks the pid/lock file, compares the PID with it's own PID and assumes "I'm the PID=1, so the PID file is some leftover from previous run". So it removes the PID file and continues with data directory modifications. This has a disaster potential.

Our templates only support Recreate strategy. The fact Rolling "mostly" works is matter of luck that the old server is not under heavy load.

That said, zero downtime problem needs to be solved on higher logical layer.

@bkabrda
Copy link

bkabrda commented Nov 16, 2018

Ok, that makes sense, thanks. If I wanted to solve this on higher logical layer, how would I go about this? Do you have any good pointers?

@praiskup
Copy link
Contributor

praiskup commented Mar 4, 2019

At this point, you'd have to start thinking about pgpool or similar thing (I'd prefer to have a separate issue for such RFEs, to not go off-topic in this bug report).

@praiskup
Copy link
Contributor

This issue seems to be caused by concurrent run of multiple postgresql
containers against the same data directory (persistent VOLUME), e.g.
caused by Rolling strategy in OpenShift.

I've heard an idea that it could also happen if OpenShift happens to be
moving the container to idle state (because HA proxy decided so), while -
during that time - some traffic makes the container to be woken up (ie new
container is started even before it was successfully moved to idle state).
Anyone able to confirm that this could happen?

Anyways, I'd like to have opinions how to handle this situation properly;
how to protect against over-mounting the same storage - since detecting
this reasonably from within container seems to be close to hard problem.
The only way which comes to my mind is implementing a "best effort" guard
by some daemon implementing "leader election" mechanism. Any links to how
others do this?

We might delegate this to OpenShift operators, but I suspect that
templates will have to stay supported anyways - or at least that
postgresql-container should be also usable from (some)templates; and thus
the problem won't disappear from non-operator use-cases, or plain "docker"
and "podman" use-cases.

@flo-ryan
Copy link

flo-ryan commented Apr 9, 2020

Hi, facing the same issue, using the Recreate strategy. Deleting the postmaster.pid also did not help, as I got the same error at the next pod startup.
Any idea on how to fix or work around this?

@ShaunDave
Copy link

Had this problem after there was an issue with the underlying node that caused it to terminate very ungracefully. A new pod got spun up (as it is supposed to) on a new node but the container got stuck in a crash-back loop with this exact error message. Surely there needs to be an automated way to get around this problem? Especially because only a single replica is supported, there's not a lot of wiggle room for high-availability if the container can't start

@drobus
Copy link

drobus commented Sep 18, 2020

This is old issue, but just faced the same with Recreate strategy. Next article explains how to reanimate failing pod and it helped me.
https://pathfinder-faq-ocio-pathfinder-prod.pathfinder.gov.bc.ca/DB/PostgresqlCrashLoopTupleError.html
https://serverfault.com/questions/942743/postgres-crash-loop-caused-by-a-tuple-concurrently-updated-error

We use only one database pod, so I believe that maybe it will not solve high availability issue, but at least database will work with one pod. Maybe will be useful for somebody.

@hhorak
Copy link
Member

hhorak commented Sep 7, 2021

We've also had an off-line discussion with Daniel Messer from RH who've hit this problem in his team as well. After changing the strategy to Recreate, it problem seems to disappear, but there was a good point to start testing the crash scenario in the CI tests (run the OpenShift template, then kill the pod or postgres deamon directly). This seems like a good addition to our test coverage.

@phracek
Copy link
Member

phracek commented Feb 28, 2024

@drobus We changed the DeploymentConfig -> Deployment, here: https://github.com/sclorg/postgresql-container/blob/master/examples/postgresql-persistent-template.json. And strategy is mentioned 'Recrete'. So please closing this issue.

In case it is not yet fixed, Feel free to re-open it again.

@phracek phracek closed this as completed Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests