diff --git a/.github/ISSUE_TEMPLATE/server-migration.md b/.github/ISSUE_TEMPLATE/server-migration.md index c700c2fd6..34f62c200 100644 --- a/.github/ISSUE_TEMPLATE/server-migration.md +++ b/.github/ISSUE_TEMPLATE/server-migration.md @@ -9,6 +9,8 @@ assignees: '' Checklist based on general guide https://github.com/openfoodfoundation/ofn-install/wiki/Migrating-a-Production-Server +Tip: find/replace to set up most commands ready to go, eg: `x_prod` -> `ca_prod` + ## 1. Setting up the new server - [ ] Check old server config for any additional services to be aware of. Document any necessary steps for migration. Eg: - `ls /etc/nginx/sites-enabled` @@ -17,9 +19,10 @@ Checklist based on general guide https://github.com/openfoodfoundation/ofn-insta - [ ] DNS: add temporary domain (eg `prod2.openfoodnetwork.org`) ### config -- [ ] Add temporary name to `inventory/hosts` +- [ ] Add temporary name to `inventory/hosts` (suggest doing this on separate branch) - [ ] Review `host_vars/x/config.yml`, clean up if needed - Make a copy for the temp hostname, add temp domain to bottom of `certbot_domains` +- [ ] Review `group_vars/x.yml`, clean up if needed - [ ] Review `ofn-secrets:x_prod/secrets.yml`, clean up if needed - Change to shared bugsnag projects - Don't bother making a copy of this one @@ -41,7 +44,7 @@ Then setup new server. Ensure you have the correct secrets (current secrets are - [ ] Setup direct ssh access for `ofn-admin` and `openfoodnetwork` as per guide `ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/` -- [ ] `db_transfer.yml` +- [ ] `db_transfer.yml` && - [ ] `transfer_assets.yml` Make sure to clear cache so that instance settings are applied: @@ -60,44 +63,47 @@ Make sure to clear cache so that instance settings are applied: ## 3. Migration ### preparation -- [ ] **new server**: `bin/rake db:reset -e production` (important: make sure you're on the new server!) -- [ ] `deploy.yml -l x_prod2 -e "git_version=vX.Y.Z"` matching version with current prod +- [ ] Reset database on new server, to avoid any migration issues due to being out of sync + `bin/rake db:reset` (You will need to confirm. Make sure you're on the new server!) +- [ ] Update ansible_host IP in `inventory/hosts` and ensure provision works (this will update host in `.env.production`). + `ansible-playbook playbooks/provision.yml -l x_prod` +- [ ] `ansible-playbook playbooks/deploy.yml -l x_prod -e "git_version=vX.Y.Z"` matching version with current prod - [ ] old server: make a tiny data change to verify later (eg add `.` in meta description `/admin/general_settings/edit`) ### switchover: old server -- [ ] 🚧 `maintenance_mode.yml` +- [ ] 🚧 `ansible-playbook playbooks/maintenance_mode.yml -l x_prod` - [ ] `sudo systemctl stop sidekiq redis-jobs puma` +- [ ] `ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/db_transfer.yml &&` +- [ ] `ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/transfer_assets.yml` - [ ] Transfer `/var/lib/redis-jobs/dump.rdb` to new server (see guide) -- [ ] `db_transfer.yml` ~3min -- [ ] `sudo systemctl stop postgres` (ensure other integrations no longer touch it) -- [ ] `transfer_assets.yml` just in case +- [ ] `sudo systemctl stop postgresql` (ensure other integrations no longer touch it) ### switchover: new server - [ ] `sudo systemctl restart puma; sudo systemctl start sidekiq redis-jobs` -- [ ] `Rails.cache.clear` (or migrate redis-cache/dump.rdb also) -- [ ] ⏭️ `temporary_proxy.yml -e 'proxy_target='` redirect traffic to new prod +- [ ] `cd ~/apps/openfoodnetwork/current; bin/rails runner -e production "Rails.cache.clear"` (or migrate redis-cache/dump.rdb also) +- [ ] ⏭️ `ansible-playbook -l x_prod playbooks/temporary_proxy.yml -e 'proxy_target='` redirect traffic to new prod * Note: this doesn't include webservices, and doesn't handle images. So it's a very short-term fix if at all. * Use a `hosts` file entry to test a direct connection - Check there are no alarm bells, eg: - - [ ] `~/apps/openfoodnetwork/current/logs/production.log` and `sidekiq.log` - [ ] tiny data change is present. undo it. - [ ] shopfront and checkout looks good - [ ] upload a product image - - [ ] get confirmation from local team + - [ ] `~/apps/openfoodnetwork/current/log/production.log` and `sidekiq.log` - [ ] Update DNS to point to new server +- [ ] get confirmation from local team +- [ ] make sure the entries in ofn-install are up to date: set the new IP address and remove any temporary entry made for the migration +- Update documentation: + * [ ] https://github.com/openfoodfoundation/ofn-install/wiki/Current-servers + * [ ] This migration guide if necessary ## 4. Cleanup (after 48hrs) - [ ] check server access logs to verify no traffic -- [ ] shut down the old server, cancel old VPS +- [ ] shut down the old VPS +- [ ] delete old VPS (or rename for future deletion) - [ ] remove DNS for temporary subdomain -- [ ] make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address. -- [ ] validate that `provision.yml` still works. This will rename x-prod2 to x-prod - [ ] check metabase sync if required: https://data.openfoodnetwork.org/admin/databases/ - [ ] check n8n - [ ] check backups are functioning -- Update documentation: - * [ ] https://github.com/openfoodfoundation/ofn-install/wiki/Current-servers - * [ ] This migration guide if necessary ## Rollback plan diff --git a/README.md b/README.md index 2a9f4e717..afde9a3ca 100644 --- a/README.md +++ b/README.md @@ -59,15 +59,18 @@ $ bin/setup ## Secrets -Some tasks require host-specific secrets, and will show an error if they haven't been provided. These can change from time to time, so **always ensure you have the latest before provisioning**. +Some tasks (eg provision.yml) require host-specific secrets, and will show an error if they haven't been provided. Secrets can be added to the relevant `host_vars` subfolder (see [wiki](https://github.com/openfoodfoundation/ofn-install/wiki/Configuration#add-host_vars)) where they will be loaded automatically. -Secrets can be provided with a parameter like so: +Secrets can also be loaded from a different folder with a parameter like so: ```sh ansible-playbook playbooks/provision.yml --limit=au_staging -e "@../ofn-secrets/au_staging/secrets.yml" --ask-vault-pass ``` -If you have access to the `ofn-secrets` repository, you can fetch them with the `fetch_secrets.yml` playbook. The secrets for each host will be loaded into the relevant directory in `inventory/host_vars/`, then you can go ahead and provision. See the [readme](https://github.com/openfoodfoundation/ofn-secrets/#readme) for more tips on setup. +Many servers are managed by the OFN core team, so we have a copy of secrets in a shared repository which is considered the source of truth. Once your server is managed by the core team, ensure any config changes are sent to them. + +Core team members can fetch the latest with the `fetch_secrets.yml` playbook. The secrets for each host will be loaded into the relevant directory in `inventory/host_vars/`, then you can go ahead and provision. See the [ofn-secrets readme](https://github.com/openfoodfoundation/ofn-secrets/#readme) for more tips on setup. +These can change from time to time, so **always ensure you have the latest before provisioning**. ```sh ansible-playbook playbooks/fetch_secrets.yml && ansible-playbook playbooks/provision.yml