Difference with openSUSE's immutability tooling

[Malix-Labs]

Hello folks !

Is there any documentation about a comparison between arkdep and openSUSE immutability tooling ?

Thanks !

These information would also help me maintain Awesome Atomic, thanks !

[dennis1248]

Not yet. But basically, the tech powering both is the same. Both do trickery with Btrfs subvolumes.

MicroOS makes a snapshot of root and then runs a traditional update on the snapshot. We instead work with subvolume exports which are build on the server and then downloaded in full by the user. Updates are a lot bigger, but the image provider is capable of making virtually any change to the image configuration.

[Malix-Labs]

Interesting

Have you discussed with openSUSE to share efforts on those tooling ?

[dennis1248]

I think the implementation is different enough that there is nothing to share.

[user1-github]

As someone who has previously used Aeon (OpenSUSE's immutable Gnome variant), I'd like to add that in the long run Arkdep is probably more efficient than btrfs snapshots in terms of disk space usage.

While technically each individual btrfs snapshot is more efficient (each new snapshot is just the difference from the previous snapshot), the downside of btrfs snapshot is that they add up over time and after some time they end up using many gigabytes of disk space. AFAIK, at some point old snapshots should supposedly be automatically removed, but to me it hasn't happened once after a few months of using Aeon and I gained over 20 snapshots which ended up using over 10gb of disk space. I've even seen some users of other OpenSUSE variants complaining that they ran out of disk space because of btrfs snapshots. I heard you can manually remove old snapshots, but it's very risky.

Arkdep on the other hand works more similarly to Fedora Silverblue - you only have a fixed amount of deployments (usually just 2 or 3), so most of the time system updates wouldn't add up disk space usage.

[dennis1248]

Aeon does not clear the old snapshots out automatically?

I actually thought Arkdep would be the less efficient one, we do not do any deduplication, which you do get for free when creating snapshots.

[user1-github]

Aeon supposedly should clear old snapshots automatically at some points, but as I said, it didn't happen during the few months I was using it, so I assume it just happens rarely. So most of the time you end up having dozens of snapshots and 99% chance you wouldn't need most of them.

As a result, even if snapshots are technically more efficient due to deduplication, dozens of them end up taking more disk space than just 2 or 3 Arkdep deployments without deduplication.

I'd say of all immutable systems/toolkits, Fedora Silverblue's rpm-ostree system is probably the most disk space efficient because it has deduplication and it keeps just 2 deployments by default. But rpm-ostree is also the most complex system and it's tied to Fedora. That's why I think Arkdep is the sweet spot of disk space efficiency and simplicity because it keeps only 2 or 3 deployments by default like rpm-ostree and it's more disk space efficient than btrfs snapshots and ABRoot.

[Malix-Labs]

That experience is not the normal behavior and would be valid to take as a neutral comparison to evaluate the storage efficiency between aeon and arkdep

ostree system is probably the most disk space efficient because it has deduplication and it keeps just 2 deployments by default

It is theoretically possible to set to keep an infinite amount of deployment with OSTree, and also set to keep only 2 deployments on Aeon

OSTree […] is more disk space efficient than btrfs snapshots and ABRoot

ABRoot is indeed not storage efficient because it keeps two slightly different Roots (A and B), and I think doesn't employ deduplication (yet, but might be improved)

Because of that, arkdep and ABRoot should be at the same level of magnitude about storage efficiency

[user1-github]

It is theoretically possible to set to keep an infinite amount of deployment with OSTree, and also set to keep only 2 deployments on Aeon

I don't know if it's possible to configure btrfs snapshots so that only the current and 1 or 2 previous snapshots will be kept. At least I tried searching about it and coupdn't find any information.

ABRoot is indeed not storage efficient because it keeps two slightly different Roots (A and B), and I think doesn't employ deduplication (yet, but might be improved). Because of that, arkdep and ABRoot should be at the same level of magnitude about storage efficiency

Last time VanillaOS team has said that they had 2 20gb root partitions, which is way less efficient than Arkdep. I just read that they reduced the size of the root partitions in VanillaOS 2, but it wasn't mentioned by how much.

[dennis1248]

I don't know if it's possible to configure btrfs snapshots so that only the current and 1 or 2 previous snapshots will be kept. At least I tried searching about it and coupdn't find any information.

That is something the wrappers scripts around it will handle.

Last time VanillaOS team has said that they had 2 20gb root partitions, which is way less efficient than Arkdep. I just read that they reduced the size of the root partitions in VanillaOS 2, but it wasn't mentioned by how much.

The new Vanilla OS works slightly different, it instead uses LVM for its volumes. And LVM probably has tricks to minimize this.

[cleanerspam]

downloaded in full by the user

Wouldn't using zync2 to do delta updates be more efficient and faster than full downloading of image ?
Zync2 reuses data that is reusable without compromising the integrity of downloads

[Malix-Labs]

Fedora is to implement zstd:chunked in the next release too btw

[dennis1248]

downloaded in full by the user

Wouldn't using zync2 to do delta updates be more efficient and faster than full downloading of image ? Zync2 reuses data that is reusable without compromising the integrity of downloads

This is something we extensively tested, we tried Zsync, Zsync2, Casync, Desync, etc.. and none worked well for this usecase.

Zsync2 and similar tools do not help much in reducing the image size and massively slowed down image deployment times. We can get similar gains and better performance by cranking the compression up to 11.

The delta updates ran in to issues with systems requiring massive specs (50GB RAM+, hours of CPU time) to generate diffs, and this would increase exponentially the larger the images.

Both solutions also complicate the repository management. Either you would have to store both delta and full images, so people can delta an existing one or download a full image if they do not have it yet. Or they need to download an old image and do multiple passes to upgrade it to the latest version. This complexity goes against what Arkdep is trying to be, stupid and simple.

Compared to rpm-ostree:

• It is a lot faster, at a 15MB/s download speed it will outperform rpm-ostree doing an update or rebase by ~5 minutes or more on reasonable hardware.
• It is much less efficient on bandwidth, rpm-ostree might typically do a 300-600MB download, we do a 1.4GB one.
• It is much lighter on the system and quicker to deploy, after download rpm-ostree would be blasting the CPU for 5 minutes to checkout a rootfs. Arkdep does it in 15-30 seconds on reasonable hardware.

And if we compare it to Android, Windows, Playstation, Xbox etc.. our update size is comparable or smaller. For some of these platforms 4-8GB updates are not uncommon.

Arkdep is really not doing too bad in terms of performance, as long as you have the bandwidth.

If efficiency is the goal then an immutable should be build around ostree or similar tech. At least until someone developes a tool which can efficiently a quickly create chunks or diff based on compressed disk images. I think systemd was looking in to building something for this usecase.

[user1-github]

downloaded in full by the user

Wouldn't using zync2 to do delta updates be more efficient and faster than full downloading of image ? Zync2 reuses data that is reusable without compromising the integrity of downloads

If you currently have Arkane Linux installed and want to save some disk space, you can change the amount of installed deployments from 3 (the default) to 2 in the arkdep config file. This way you'll only have the current and the previous deployment, each weighting around 4-5gb.

I think what could also save some disk space is reducing the size of the swap file. It's currently 6.4gb, which is a bit overkill imo. On Ubuntu for example, it's 2gb. But changing that would require manual intervention on existing installs..