Loop with stdin

[jasperjorna]

Hi,

I'm currently in the process of converting bash scripts to resticprofile. In my scripted version I'm currently looping through all existing MySQL databases and back them up via Restic with stdin. I know resticprofile has support for stdin, but can it also be used with a loop?

databases=`$MYSQL -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema|sys)"`

for db in $databases; do
  mysqldump --complete-insert --routines --triggers --single-transaction --no-tablespaces --databases $db | gzip | $RESTIC -r $REPO backup --stdin --stdin-filename $db.gz --tag $db
done

[creativeprojects]

Quick answer:

No see answer bellow

Long answer

Right now the configuration doesn't have any concept of loops and probably won't in the near future.
An interesting concept I could imagine here would be to generate dynamic profiles from variables.

Now, on my databases, I found out that using stdin could lock your table for a certain amount of time depending on the speed of your backup support. With the traditional country-side dialup connection we have in the UK it takes a few minutes to upload a table and the database is locked in a read-only transaction, which causes some issue with services I'm running at home.
So I dump the databases into temporary files before doing a backup of the temporary files. No more locking issues. You just need some spare space on a disk.

mysql:
  backup:
    tag:
      - mysql
    run-before:
      - rm -f /tmp/mysqldumpall.sql
      - mysqldump --all-databases > /tmp/mysqldumpall.sql
    source:
      - /tmp/mysqldumpall.sql
    run-after:
      - rm -f /tmp/mysqldumpall.sql
    schedule: 'daily'

Sorry if it's not the answer you were looking for 😉

[creativeprojects]

It's the group definition that is wrong for the version 1 of the configuration, it needs to be:

{{ $databases := .Env.databases | split "\n" }}

version: "1"

groups:
  database:
      {{ range $db := $databases }}
      - stdin-{{ $db }}
      {{ end }}

{{ range $db := $databases }}
stdin-{{ $db }}:
  inherit: base
  backup:
    repository: "${REPOSITORY}/database"
    tag:
      - "stdin-{{ $db }}"
{{ end }}

groups are only a list of profiles in v1, with no other option attached to them.

[jasperjorna]

Hi @creativeprojects,

Yep, that actually works! I just realised that this obviously has some impact on how other operations behave. It wants to have the list of databases for every operation – that makes it a bit tricky to use.

[creativeprojects]

It doesn't have to be, you can initialise an empty list when the environment variable is not available:

{{ $databases := list }}
{{ if .Env.databases }}
  {{ $databases = .Env.databases | split " " }}
{{ end }}

[2q2code]

Is there a version of the json schema that allows for templating in it? I use the schema as my guide for syntax in VSCode.

[2q2code]

Oh - never mind - the hashes were missing from the second example.... fiddling