[Kamal 2]: Container Conflicts for Staging and Production Application Running on Same Host

[mike-weiner]

I've got a basic web application running on a DigitalOcean Droplet. I have both the production and staging instances of the application on the same host. I realize that's atypical. It's a non-critical, side project that I don't need redundancy for nor do I need to add complexity for 2 different hosts.

I'm hitting a weird edge case that I'm looking for some advice on.

When I've got a new build ready to go out to staging, I do a kamal deploy -d staging and it builds the container with the current commit (let's say that is 123), builds the image, and exposes port abcd for my web application. That deploys perfectly.

24 hours later I'll do a kamal deploy to push the changes out to production. Kamal re-builds the container with the same commit (123) but it exposes a different port, let's say efgh for production. That process then overwrites that image in my registry with a build that now exposes port efgh instead of abcd.

The production instance deploys fine, but if I ever need to restart the staging instance, it will go and re-pull the 123 container image which is now built to expose the incorrect port.

Is there a better way to achieve this kind of setup to avoid this? Or is this a limitation that I'm going to need to live with?

Thank you!

[tuladhar]

Are you using same service: name for both your staging and production?

[mike-weiner]

@djmb ^ Friendly bump on the above.

[djmb]

Hey @mike-weiner!

Kamal runs "docker stop" as part of the deploy, so if it is killing the staging container then that's when it would be happening. It shouldn't be doing this though, but obviously something is killing the container.

Could you repeat these steps and look for "docker stop" in the logs? The command will look something like:

docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop

If whatever you have in the docker container ls --all --filter name=^app-web-123$ --quiet matches your staging container then that's likely to be it.

For further debugging, there's also a docker events command that might track what's killing the container.

[mike-weiner]

@djmb - Found the problem.

kamal deploy -d staging:

...
Detect stale containers...
  INFO [ac667262] Running docker ps --filter label=service=upbeat --filter label=destination=staging --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#upbeat-web-staging-}; done on 159.203.151.191
  INFO [ac667262] Finished in 0.178 seconds with exit status 0 (successful).
...

kamal deploy:

...
Detect stale containers...
  INFO [fe423d05] Running docker ps --filter label=service=upbeat --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#upbeat-web-}; done on 159.203.151.191
  INFO [fe423d05] Finished in 0.195 seconds with exit status 0 (successful).
  INFO [2eadd5da] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=upbeat --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.hub.docker.com/michaelweiner/upbeat:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=upbeat --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#upbeat-web-}; done on 159.203.151.191
  INFO [2eadd5da] Finished in 0.256 seconds with exit status 0 (successful).
App Host: 159.203.151.191
Stopping stale container for role web with version staging-7d23aa9dd7985a3e592fa455f2fe8a1f4a4dfcf2

  INFO [05d3dee2] Running docker container ls --all --filter name=^upbeat-web-staging-7d23aa9dd7985a3e592fa455f2fe8a1f4a4dfcf2$ --quiet | xargs docker stop on 159.203.151.191
  INFO [05d3dee2] Finished in 1.281 seconds with exit status 0 (successful).
...

The key difference here is the docker ps --filter label=service=upbeat --filter label=role=web --format "{{.Names}}" when doing kamal deploy versus the docker ps --filter label=service=upbeat --filter label=destination=staging --filter label=role=web --format "{{.Names}}" when running kamal deploy -d staging.

I went and looked at the labels on my staging container and my production container:

~$ docker inspect --format '{{ json .Config.Labels }}' 721dc3f5a52f
{"destination":"staging","role":"web","service":"upbeat"}
~$ docker inspect --format '{{ json .Config.Labels }}' 3908b053db86
{"destination":"","role":"web","service":"upbeat"}

~$ docker ps --filter label=service=upbeat --filter label=role=web --format "{{.Names}}"
upbeat-web-staging-7d23aa9dd7985a3e592fa455f2fe8a1f4a4dfcf2
upbeat-web-7d23aa9dd7985a3e592fa455f2fe8a1f4a4dfcf2

So, that explains why doing kamal deploy is killing my "stale" staging container.

That begs another question for me. Why does kamal deploy not add the --filter label=destination=? I would expect that if I don't specify a destination that kamal wouldn't go and change the state of containers that do have a destination label.

~$ docker ps --filter label=service=upbeat --filter label=role=web --filter label=destination= --format "{{.Names}}"
upbeat-web-7d23aa9dd7985a3e592fa455f2fe8a1f4a4dfcf2

[djmb]

Ah! That's a bug then - I'll convert this to an issue and pick it up next week.

Thanks for digging into this 🙏

[mike-weiner]

Awesome. Thank you for helping dig into this!!

(For anyone else interested, here is the associated issue: #1184)