Skip to content

Restore mirrorbrain from a backup

Jump to bottom
benoit74 edited this page Dec 3, 2024 · 3 revisions

Download a backup

Backups are in borgbase. To download them you need the read-only credentials:

# those are all static values you need to enter
# those are all for the _slave_ (aka readonly) bitwarden account
export BW_CLIENTID=user.xxxxxxxxx
export BW_CLIENTSECRET=xxxxxxxxxxxx
export BW_PASSWORD=xxxxxxxxxxxx

Select a backup

docker run --rm -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name mirrorbrain --list

mirrorbrain is the name of the Borgbase repository in which we archive the Mirrorbrain DB backups.

Output would look like

List avaible archives ...
Remote: Warning: Permanently added the ECDSA host key for IP address '95.216.113.224' to the list of known hosts.
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] yes (from BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK)
mirrorbrain__backup__2022-12-31T03:02:25 Sat, 2022-12-31 03:02:28 [0938eae7985b017733dbe6bc44cce4cb79300bf5b79df7516784f9ba1c9cf557]
mirrorbrain__backup__2023-11-30T03:02:44 Thu, 2023-11-30 03:02:47 [76d3c8f57ac27d6a354bb1f02a616cf069f54b34339613f7c84dafe5389000fc]
mirrorbrain__backup__2023-12-31T03:02:59 Sun, 2023-12-31 03:03:03 [cd701be2db3555319ecda1acdaa62be55669674244008a98b1470af731c98a92]
mirrorbrain__backup__2024-01-31T03:03:02 Wed, 2024-01-31 03:03:06 [fab0aae3f5f0b64005a5813fcb55727a24b4c46bfb892db2e5d1c218f5f4180f]
mirrorbrain__backup__2024-02-29T03:02:59 Thu, 2024-02-29 03:03:03 [01b0b9f7a958652fdeb66642d756adc6c0d90ad54b3095f6637b66ea409b756a]
mirrorbrain__backup__2024-03-31T03:05:00 Sun, 2024-03-31 03:05:04 [0bdf2e2c32774a85bd06011435bc16e4a288e387d16a7d15fdfb78685e6cfd67]
mirrorbrain__backup__2024-04-30T03:02:30 Tue, 2024-04-30 03:02:33 [fa893eea0c83bf8e34d96bcdf1e314cf4a5d6c4dd80f7344f81764e90f53c68f]
mirrorbrain__backup__2024-05-31T03:03:16 Fri, 2024-05-31 03:03:22 [fa8d84633ad02efd290ba8bdeaead602439534e68960ff2db36d2895e916e08c]
mirrorbrain__backup__2024-06-30T03:02:52 Sun, 2024-06-30 03:02:55 [c11f3423e7bab97a5e4142579124d68a4f5af90b8303cc3a80fe877facd8d0d7]
mirrorbrain__backup__2024-07-31T03:02:36 Wed, 2024-07-31 03:02:39 [84701190d4e93583d615ddc6793b389909df0b9355157da3f3cfe7f4d67ac4db]
mirrorbrain__backup__2024-08-31T03:02:28 Sat, 2024-08-31 03:02:31 [14923cebce446efce335f27354274333ee5668d9019180326c8e04ef6e1b9020]
mirrorbrain__backup__2024-09-30T03:01:27 Mon, 2024-09-30 03:01:28 [169a3ba147dfd4cd35370926bb6ef8125648a5bb36d426293e094e096541b536]
mirrorbrain__backup__2024-10-20T03:01:24 Sun, 2024-10-20 03:01:25 [5998f8fbf073cb97e606e1590b499e4aabe5a1b09a5418fcc1f2975e750f0eb0]
mirrorbrain__backup__2024-10-27T03:01:24 Sun, 2024-10-27 03:01:25 [5b59f720e69f234bd4f0157c854edc9c9adff08df8f8c9ff7ebb003c341389c3]
mirrorbrain__backup__2024-10-31T03:01:20 Thu, 2024-10-31 03:01:22 [fd86d3c3f3519b84425be380ef49baa61ae0f473a2b149b86f6dbc4090dcaef6]
mirrorbrain__backup__2024-11-03T03:01:31 Sun, 2024-11-03 03:01:32 [bf3179a5e703e26b48685f07befb4455b0ac5fa20d28bd22f4ba51beed5a0bd5]
mirrorbrain__backup__2024-11-10T03:01:23 Sun, 2024-11-10 03:01:24 [80816d1b8523360f3d7babb6cae3012c27be4d843ffdd9455fd69b337cb921e0]
mirrorbrain__backup__2024-11-17T03:01:28 Sun, 2024-11-17 03:01:29 [252220f5f889e675f59e9f834416f2682951d0b001adaba9bf993ac51fa0dfdd]
mirrorbrain__backup__2024-11-23T03:01:27 Sat, 2024-11-23 03:01:28 [953f6597dae1d23c7d8eaab4615673c8535f3d2c41cba882f2939fbb0f71926a]
mirrorbrain__backup__2024-11-24T03:01:20 Sun, 2024-11-24 03:01:22 [18eba29aeba8dbeb5479690f7174d98cfb9b7628f747a4474182350b1949da2b]
mirrorbrain__backup__2024-11-25T03:01:21 Mon, 2024-11-25 03:01:22 [102b04d524e84057b9ebc205eda2bb9fb09c4751928d8a61b5c7a1cc15cdb88a]
mirrorbrain__backup__2024-11-26T03:01:24 Tue, 2024-11-26 03:01:25 [58ff4c06e23c5e24ebdcc559e03cbe716da46cfee7b9b0ccbbf82a2b66929a58]
mirrorbrain__backup__2024-11-27T03:01:27 Wed, 2024-11-27 03:01:28 [1b6c9088ecb56d517ad65f742c2a8f47a79aaf6233846dbb1f2099bffe037003]
mirrorbrain__backup__2024-11-28T03:01:24 Thu, 2024-11-28 03:01:25 [91e6265040414f4d701ebba2dd16e9fb2c5beb1dfb26705296733588f3ab756e]
mirrorbrain__backup__2024-11-29T03:01:28 Fri, 2024-11-29 03:01:29 [783bb502ba00b65292f58d33b7ca77b81f651d898a405195da785d3bf7a0286c]
mirrorbrain__backup__2024-11-30T03:01:28 Sat, 2024-11-30 03:01:30 [7b65aee1ff1cabfd8b550195f646766a6fa11478a90f0ca1bfa717e87546215c]

Choose one based on its date. Check the default backup periodicity in borg-backup tool and potential customization in k8s backup cronjob.

Note: the archive name is the first column (stops at first space). ex: mirrorbrain__backup__2022-12-31T03:02:25.

Extract a Backup file

With your selected archive name, download+extract it to your filesystem:

docker run -v /data/restore:/restore:rw -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name zimfarm-dispatcher-postgresdb --extract "zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41"

Zimfarm backup is a single Custom Database Dump file that would be extracted to /data/restore in this example. The file has no extension ; move it to a more practical location

mv /data/restore/root/.borgmatic/postgresql_databases/api-postgres-db-service/zimfarm /data/restore/zimfarm

Test the dump file

# start a new postrges server.
# note that this will create the `zimfarm` database and its credentials
docker run -v /data/restore:/data -it --name pg-tester --rm -e POSTGRES_DB=zimfarm -e POSTGRES_USER=zimfarm -e POSTGRES_PASSWORD=zimpass -p 5432:5432 postgres:15.2-bullseye
# import the dump in the same container
docker exec -it pg-tester pg_restore -U zimfarm -d zimfarm /data/zimfarm

Check the DB structure and data using any postgres client (DBeaver)

Production Restore Procedure

If the mirrorbrain is running

  • shutdown the API deployment by scaling it to 0.
  • shutdown the postgres sts by scaling it to 0.

Downloading dump into volume

In order to get the dump file into the volume, one needs to launch borg-backup into the cluster. This would be done with a temporary Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: borg-accessor
  namespace: zimfarm
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: borg-app
    spec:
      containers:
      - name: borg-backup
        image: ghcr.io/kiwix/borg-backup
        command: ["restore", "--name", "zimfarm-dispatcher-postgresdb", "--extract", "zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41"]
        imagePullPolicy: Always
        env:
        - name: BW_CLIENTID
          value: "xxxx"
        - name: BW_CLIENTSECRET
          value: "xxxx"
        - name: BW_PASSWORD
          value: "xxx"
        volumeMounts:
        - name: data-volume
          mountPath: "/restore"
          readOnly: false
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: zimfarm-api-postgres-db-pvc
      restartPolicy: Never
      nodeSelector:
        k8s.kiwix.org/role: "services"

Restore

  • Cleanup the volume on disk
  • Move the dump file to the volume folder /var/lib/postgres/data/dump.sql
  • Cleanup the dump
    • the problem is that the dump is created for PG16 while we still use PG11
    • there is some errors in CREATE DATABASE statements
    • search for LOCALE_PROVIDER or LOCALE arguments in CREATE DATABASE statements and remove them
    • e.g. CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8'; must be transformed into CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UTF8';

As of Dec. 2024

77:CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8';
177:CREATE DATABASE mirrorbrain WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8';
293017:CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8';

  • Start the postgres sts

    • comment out special command
    • comment out special initContainers
    • apply the STS

  • Open a shell on the postgres container

  • Restore the dump psql -f /var/lib/postgres/data/dump.sql

  • Start a PortForward and ensure the data is there

  • Check the DB structure and data using any postgres client

  • Restart the postgres STS without special commented-out lines (as in repo)

  • QQQ

Clone this wiki locally