feat: add vega datanode migrate-ipfs subcommand

[daniel1302]

Closes vegaprotocol/documentation#910 .

Changes

There is a tool designed to run the migration for the IPFS store to a newer version: https://github.com/ipfs/fs-repo-migrations/tree/master

I have incorporated this tool into the vega. I added a standalone command (go run main datanode migrate-ipfs), which is also fired when migration is needed before the data node is started (in the pre-start step).

High overview of how the IPFS migration tool is working:

1. It checks if the local node is working if yes, it looks for the required migration binaries
2. If the local node is not working it connects to the remote node(the node is hard coded in the Kubo library: https://pkg.go.dev/github.com/ipfs/[email protected]/repo/fsrepo/migrations#pkg-constants) and checks for the binaries to migrate the ipfs repo.
3. It downloads the binaries from the IPFS
4. It runs binaries to migrate the IPFS filesystem.

Testing

The command embedded into the datanode start

When the migration is needed

Downloading migration: fs-repo-14-to-15...
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-14-to-15/versions"
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-13-to-14/versions"
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-14-to-15/v1.0.1/fs-repo-14-to-15_v1.0.1_linux-amd64.tar.gz"
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-13-to-14/v1.0.0/fs-repo-13-to-14_v1.0.0_linux-amd64.tar.gz"
Downloaded and unpacked migration: /tmp/migrations1355099032/fs-repo-14-to-15 (v1.0.1)
Downloaded and unpacked migration: /tmp/migrations1355099032/fs-repo-13-to-14 (v1.0.0)
Running migration fs-repo-13-to-14 ...
  => Running: /tmp/migrations1355099032/fs-repo-13-to-14 -path=/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs -verbose=true
applying 13-to-14 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '13'
> Upgrading config to new format
updated version file
Migration 13 to 14 succeeded
Running migration fs-repo-14-to-15 ...
  => Running: /tmp/migrations1355099032/fs-repo-14-to-15 -path=/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs -verbose=true
applying 14-to-15 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '14'
> Upgrading config to new format
updated version file
Migration 14 to 15 succeeded
Success: fs-repo migrated to version 15.
2024-02-08T23:32:39.315+0100	INFO	datanode.start.persistentPre	start/node_pre.go:115	Migrating the IPFS storage finished
2024-02-08T23:32:42.153+0100	INFO	datanode.start.persistentPre	start/node_pre.go:123	Initializing Network History

...

When the migration is not needed

2024-02-08T23:29:35.190+0100	INFO	datanode.cfgwatcher	config/watcher.go:94	config watcher started successfully	{"config": "/home/daniel/vega_home2/config/data-node/config.toml"}
2024-02-08T23:29:35.191+0100	INFO	datanode.start.persistentPre	start/node_pre.go:70	Starting Vega Datanode	{"version": "v0.74.0-dev", "version-hash": ""}
2024-02-08T23:29:43.672+0100	INFO	datanode.start.persistentPre	start/node_pre.go:111	Migrating the IPFS storage to the latest version
2024-02-08T23:29:49.117+0100	INFO	datanode.start.persistentPre	ipfs/migration.go:52	The IPFS for the network-history is up to date. Migration not needed
2024-02-08T23:29:58.243+0100	INFO	datanode.start.persistentPre	start/node_pre.go:115	Migrating the IPFS storage finished
2024-02-08T23:30:00.183+0100	INFO	datanode.start.persistentPre	start/node_pre.go:123	Initializing Network History
...
...

The standalone migration command: go run main datanode migrate-ipfs

When migration is needed:

# go run main datanode migrate-ipfs --home /home/daniel/vega_home2

Looking for suitable migration binaries.
Need 2 migrations, downloading.
Downloading migration: fs-repo-13-to-14...
Downloading migration: fs-repo-14-to-15...
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-14-to-15/versions"
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-13-to-14/versions"
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-14-to-15/v1.0.1/fs-repo-14-to-15_v1.0.1_linux-amd64.tar.gz"
Error fetching: failed to create a ipfs shell migration: ipfs api address could not be found
Fetching with HTTP: "https://dweb.link/ipns/dist.ipfs.tech/fs-repo-13-to-14/v1.0.0/fs-repo-13-to-14_v1.0.0_linux-amd64.tar.gz"
Downloaded and unpacked migration: /tmp/migrations3323001421/fs-repo-13-to-14 (v1.0.0)
Downloaded and unpacked migration: /tmp/migrations3323001421/fs-repo-14-to-15 (v1.0.1)
Running migration fs-repo-13-to-14 ...
  => Running: /tmp/migrations3323001421/fs-repo-13-to-14 -path=/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs -verbose=true
applying 13-to-14 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '13'
> Upgrading config to new format

When the migration is not needed

# go run main datanode migrate-ipfs --home /home/daniel/vega_home2

2024-02-08T22:54:02.413+0100	INFO	root	ipfs/migration.go:47	The IPFS for the network-history is up to date. Migration not needed