Migration procedure and some questions before going to production

[daks]

Hello everyone,

I have tested DMS during several weeks now, and all seems to work like I want. I have tuned the setup, prepared the migration procedure and tested most of it.
But before effective migration (for personal use) I would like to know if I missing anything important on this procedure, and have some unanswered questions. I would appreciate any comment from experienced users or devs.

My setup is the following:

• using Docker rootless with an unprivileged user
• using Docker compose
• binding on unprivileged (>1024) ports with nftables redirection rules to forward standard (25/465/993) ports on the host to the unprivileged ones
• LetsEncrypt runs on the host and I use a 'deploy' hook to copy them in the right directory
• fail2ban run on the host and watch for DMS logs using dovecot or postfix filters to block fraudulent IPs
• DMS is configured with SMTP, IMAP, rspamd+redis, fetchmail, managesieve, no amavis...

My migration procedure is:

• stop DMS with docker compose down
• RESET container with docker system prune
• RESET all data, keeping 'certs' and 'config'
• start DMS
• create all mailboxes with docker exec -ti mailserver setup email add [email protected]
• create aliases, for mailboxes or catchalls, with docker exec -ti mailserver setup alias add [email protected] [email protected]
• generate DKIM configurations for all domains with docker exec -it mailserver setup config dkim domain example.com
• insert DKIM keys into config/rspamd/override.d/dkim_signing.conf and add them to DNS
• sync mailboxes with mbsync (not perfect because IMAP client needs to suscribe again and re-download all files) or with a rsync between old server (running dovecot too) and DMS
• maybe restart container and check logs once migration is done
• change MX and SPF DNS entries to point to my host IP
• re-run sync once DNS propagation is complete
• I still need to decide how to add sieve configuration, either at mailbox creation or later copying file manually.

The questions I have are:

• is the RESET container+data steps needed? Are they right or should I do something else?
• I read this page about backups https://docker-mailserver.github.io/docker-mailserver/latest/faq/#what-about-backups, I understand that a backup of 'config', 'mail-data' and 'mail-state' (and 'mail-logs') is enough, am I wrong?
• what about containers restart? Using docker compose with restart: always, the container will be started automatically when server reboot. Is it safe? I saw https://github.com/orgs/docker-mailserver/discussions/4032 and bug: DMS v14 regression - Container restart support may fail due to unclean exit #4150 and I'm not sure how to understand them
• how to update the container? should I use something like watchcontainer for automatic updates? Or should I, from time to time, verify github changelog and upgrade manually? In my case it could mean several months or years between updates
• I just thought about Rspamd web UI but don't see any exposed port in the compose.yaml? do I need to declare it myself in the compose.yaml? or use another solution to access it?
• what about monitoring/alerting? For the processes but also mail queues or anything else involved in a working mail setup
• what about metrics? I use Munin, I found those plugins https://gallery.munin-monitoring.org/categories/postfix/, can I use them simply directing them to the right DMS files, or do I need another solution?

I hope I didn't forget something important. Please comment if you think about anything.

[polarathene]

Migration in the sense of the same system? Otherwise I'm not sure what your earlier steps are with "RESET". Is this like for migrating an image upgrade when we publish a new release, or getting a clean slate on the same system?

An OCI/Docker image runs a container like DMS which you can use volumes to persist your data changes outside of the container. Meanwhile a container has it's own internal state persisted until it's destroyed. So restarting your system or some crash that triggers restart: always which are like docker start dms / docker restart dms, those should still have the internal state preserved for the container. You'll get that with ctrl + c in a terminal when using docker compose up (without -d), unless you run docker compose down which properly stops the services and removes the container. Alternatively docker compose up --force-recreate doesn't need docker compose down, it forces starting a fresh container of the service instead.

Our startup scripts can be a bit naive and assume they have the original file state for config files in our image. Some are not as robust, so a container restart would in the past be more problematic because it tries to make a change (modify, add, delete) to some config file but instead of A => B, it's like B => C where C is broken/unexpected, thus weird bugs can occur. With v14 we tried to resolve that by only running the bulk of these scripts on a fresh container instance (we add a file to that internal container state and check if it's already there to know if this is not a fresh container but a restarted one), but as you link there was a mistake there which will be fixed for v15 (should be released before the end of this month AFAIK).

When your image tag (or the digest it points to) changes because you're updating to a new release, or you change some settings like the environment variables for the service container, it should result in creating a new image for you. You can't really run an old container that's associated to an older image of DMS with the newer image for example, so it must start with a fresh container.

Basically you can use docker compose up --force-recreate or the down + up sequence like our troubleshooting docs page mentions and you should be fine. With the upcoming v15 release it should be less problematic and you probably won't run into issues even when the container is restarted.

As for your own data persistence via volumes. If they're bind mount to host directory it's quite easy to observe the content and delete their content if you need to. Data volumes however aren't tied to a container, those have nothing to do with what I described earlier, clearing them is manual with docker volume rm, or via a shell into the container to access and manage their content.

For upgrading to a new release it's important to pay attention to our changelog, especially major versions which mention any breaking changes introduced (which if they affect you can require manual intervention).

---

• how to update the container? should I use something like watchcontainer for automatic updates? Or should I, from time to time, verify github changelog and upgrade manually? In my case it could mean several months or years between updates

We have a docs page on this for WatchTower advice: https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/maintenance/update-and-cleanup/

Note that watchtower hasn't been maintained for like a year (last commit Jan 2024). There are some alternatives, but if you're going to upgrade infrequently it makes more sense to just bump the version tag manually, you'll likely need to review our changelog anyway.

---

I just thought about Rspamd web UI but don't see any exposed port in the compose.yaml? do I need to declare it myself in the compose.yaml? or use another solution to access it?

I gave an example of how it can be configured here: #4022 (comment)

what about monitoring/alerting? For the processes but also mail queues or anything else involved in a working mail setup

Not my area of expertise, I can't assist there. I believe our docs touch on some log monitoring config/tools, but if you need something more than that you'd probably have to set that up yourself.

what about metrics? I use Munin, I found those plugins https://gallery.munin-monitoring.org/categories/postfix/, can I use them simply directing them to the right DMS files, or do I need another solution?

Likewise, that'll be something you'd need to investigate. We lack the time and resources as volunteers to cover everyones needs, but do appreciate contributions from users to share their knowledge in our docs to help others that come after them wanting the same thing.

For the most part DMS is glue and we try to make it easy to configure the services while still providing you control to manage config yourself (usually via our overrides and user-patches.sh support). So if you need to change configuration and DMS doesn't have any direct support officially, you should still be able to achieve that.

If you just need to monitor logs, you have the log volume and dockers own logging driver which you could probably leverage.

---

I realize this doesn't answer all your questions, but I hope it helps. I've not deployed DMS myself, I just help maintain and contribute.

[daks]

Migration in the sense of the same system? Otherwise I'm not sure what your earlier steps are with "RESET". Is this like for migrating an image upgrade when we publish a new release, or getting a clean slate on the same system?

I'm migrating from server A using Postfix/Dovecot/... to server B using DMS.
Server B has already some data I used for tests: test domain, test mailboxes...

the RESET steps are needed to get back to a 'bare' state to get rid of all tests I did.
I will proceed with doing it like I wrote above. (docker system prune)

An OCI/Docker image runs a container like DMS which you can use volumes to persist your data changes outside of the container. Meanwhile a container has it's own internal state persisted until it's destroyed. So restarting your system or some crash that triggers restart: always which are like docker start dms / docker restart dms, those should still have the internal state preserved for the container. You'll get that with ctrl + c in a terminal when using docker compose up (without -d), unless you run docker compose down which properly stops the services and removes the container. Alternatively docker compose up --force-recreate doesn't need docker compose down, it forces starting a fresh container of the service instead.

OK. I use volumes. And use "up -d" and "down"

Our startup scripts can be a bit naive and assume they have the original file state for config files in our image. Some are not as robust, so a container restart would in the past be more problematic because it tries to make a change (modify, add, delete) to some config file but instead of A => B, it's like B => C where C is broken/unexpected, thus weird bugs can occur. With v14 we tried to resolve that by only running the bulk of these scripts on a fresh container instance (we add a file to that internal container state and check if it's already there to know if this is not a fresh container but a restarted one), but as you link there was a mistake there which will be fixed for v15 (should be released before the end of this month AFAIK).

OK I see.
Instead of the 'lock file' a config management (like Ansible or Salt) would have been a perfect tool for that, ensuring the desired state of files. But it's another discussion :)

When your image tag (or the digest it points to) changes because you're updating to a new release, or you change some settings like the environment variables for the service container, it should result in creating a new image for you. You can't really run an old container that's associated to an older image of DMS with the newer image for example, so it must start with a fresh container.

Basically you can use docker compose up --force-recreate or the down + up sequence like our troubleshooting docs page mentions and you should be fine. With the upcoming v15 release it should be less problematic and you probably won't run into issues even when the container is restarted.

As for your own data persistence via volumes. If they're bind mount to host directory it's quite easy to observe the content and delete their content if you need to. Data volumes however aren't tied to a container, those have nothing to do with what I described earlier, clearing them is manual with docker volume rm, or via a shell into the container to access and manage their content.

I use volumes in compose.yaml and in fact I can read, write and delete files in there (assuming I take care of uid/gid).

For upgrading to a new release it's important to pay attention to our changelog, especially major versions which mention any breaking changes introduced (which if they affect you can require manual intervention).

• how to update the container? should I use something like watchcontainer for automatic updates? Or should I, from time to time, verify github changelog and upgrade manually? In my case it could mean several months or years between updates

We have a docs page on this for WatchTower advice: https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/maintenance/update-and-cleanup/

Note that watchtower hasn't been maintained for like a year (last commit Jan 2024). There are some alternatives, but if you're going to upgrade infrequently it makes more sense to just bump the version tag manually, you'll likely need to review our changelog anyway.

So that would mean not using image: ghcr.io/docker-mailserver/docker-mailserver:latest but something like image: ghcr.io/docker-mailserver/docker-mailserver:14?
I need to change that now then. Because maybe docker will update the container when starting it (in case of a server reboot for example)

I just thought about Rspamd web UI but don't see any exposed port in the compose.yaml? do I need to declare it myself in the compose.yaml? or use another solution to access it?

I gave an example of how it can be configured here: #4022 (comment)

👍

what about monitoring/alerting? For the processes but also mail queues or anything else involved in a working mail setup

Not my area of expertise, I can't assist there. I believe our docs touch on some log monitoring config/tools, but if you need something more than that you'd probably have to set that up yourself.

what about metrics? I use Munin, I found those plugins https://gallery.munin-monitoring.org/categories/postfix/, can I use them simply directing them to the right DMS files, or do I need another solution?

Likewise, that'll be something you'd need to investigate. We lack the time and resources as volunteers to cover everyones needs, but do appreciate contributions from users to share their knowledge in our docs to help others that come after them wanting the same thing.

For the most part DMS is glue and we try to make it easy to configure the services while still providing you control to manage config yourself (usually via our overrides and user-patches.sh support). So if you need to change configuration and DMS doesn't have any direct support officially, you should still be able to achieve that.

If you just need to monitor logs, you have the log volume and dockers own logging driver which you could probably leverage.

OK, I'll investigate by myself and eventually document it here or make a PR

I realize this doesn't answer all your questions, but I hope it helps. I've not deployed DMS myself, I just help maintain and contribute.

Oh... that's surprising.
Don't take it as a criticism but I always find it strange to contribute and maintain an open source software you don't use, except if you're doing it professionally :)
But YMMV :)

I hope some others can answer to some of my other questions. Thanks for your long answer!

[polarathene]

Instead of the 'lock file' a config management (like Ansible or Salt) would have been a perfect tool for that, ensuring the desired state of files. But it's another discussion :)

Yeah feel free to open a discussion about that if you want to propose an improvement. For the most part it's just a lack of time from current maintainers and DMS is built from years of contributors that have come and gone. Sometimes the decisions that were made to do something weren't thought out well, but replacing them with a better solution is time consuming (you usually need to understand the current approach / services / config well enough before you can introduce a better solution).

Getting the configuration / state management split out to be handled separately would be ideal, we've chipped away at cleaning up some of the messy contributions / approaches of the past, but the reliance on bash scripts is a friction point (there's been talk to port to something more appropriate but all maintainers need to be onboard otherwise we risk losing valued long-term maintainers).

---

I use volumes in compose.yaml and in fact I can read, write and delete files in there (assuming I take care of uid/gid).

Yes I meant to point out how volumes aren't directly associated to a specific container / service. Throwing away a container doesn't throw away the volume.

Docker Compose takes it a step further for anonymous volumes that should be discarded when a container is removed, but instead it persists it to the service which spans image upgrades IIRC (I had more of a problem with this when the image itself used the VOLUME instruction to enforce an anonymous volume which I couldn't easily target for removal if I wanted the container to be in a clean config state).

---

So that would mean not using image: ghcr.io/docker-mailserver/docker-mailserver:latest but something like image: ghcr.io/docker-mailserver/docker-mailserver:14?
I need to change that now then. Because maybe docker will update the container when starting it (in case of a server reboot for example)

Yeah you should not rely on :latest in production, that's an anti-pattern.

Usually it's the opposite problem. You start the :latest tag image thinking it's the latest container, but really it's still the same release you had 6 months ago because Docker recognizes it has the :latest tag of that image, so even when you have a new compose.yaml or docker run command, it doesn't pull the latest tag and uses the local pull from 6 months ago instead.

Eventually you may be checking something to troubleshoot with a separate compose or docker command like I mentioned and fix that by pulling the latest tag again to update it to the actual latest image available now. But that then has the production compose upgrade to that new :latest tag when it's eventually restarted which you might forget about 🤷‍♂️

Instead we provide semver tags 14, 14.0, 14.0.0, so that you can pin to major, minor, or patch tags. If we released v14.1, then the 14 tag will be updated to point to that, 14.0 will continue to point to the prior minor release. This can still get the :latest problem if we released a 14.0.1 tag after 14.1, because now (due to our CI being naive) will point 14 to 14.0.1 instead of the true semver latest 14.1.0. GH releases has this same issue with their latest URL alias.

You shouldn't have to worry much about the patch version, so preferring MAJOR.MINOR tag is advised. No unexpected surprises (like doing a migration between two systems when in-between we push out a new breaking release).

---

I've not deployed DMS myself, I just help maintain and contribute.

Oh... that's surprising. Don't take it as a criticism but I always find it strange to contribute and maintain an open source software you don't use, except if you're doing it professionally :) But YMMV :)

Ha yeah it wasn't intended, I was originally looking for a mail server image when I knew nothing about mail servers.

I chose DMS and made a contribution, then did a few more and it's been more of an opportunity for me to round out my skills in containers, documentation, OSS management and of course mail servers. I'll likely step away at some point but I enjoy supporting and growing the community we have here!

I realize this doesn't answer all your questions, but I hope it helps.
I hope some others can answer to some of my other questions. Thanks for your long answer!

You're welcome, here's hoping! 🙏