Roadmap

File Hosting

Thoughts by siksik

First thing: the goal of the mirroring is not having faster response but higher availability and lower bandwidth, therefore a CDN is not what we're aiming for.

So, the idea was to distribute dockers with the following capabilities:

1. Will update their local mirror with rsync
2. Can be autoupdated if it needs changes
3. Will have configs
4. Is easy to install (as a consequence of being a docker)
5. might be the most important: the mirroring should be configurable both in size and bandwidth

Configuration of the bandwith is easy with rsync, here's an example that we could use: https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh

Size constraint is due to the fact that the mirrors that are being offered differ very much in size and most of them are unable to host the full bulk, therefore my idea was ordering the mods by traffic (hits/week?) and breaking it into chunks of, lets say 20GB.

• Having the docker we can control which parts are mirrored where
• Higher traffic bulks should have higher redundancy
• And we could update the ordering once a week or smthing like that, or even once a month

Having the docker, we can have some part of feedback on availability

In any case, I'd have a cron checking availability with a HEAD-only request I think it could be nice to have the servers divided by continental regions (so there's some broad close-to, but it not being a priority) and then would come the rules to check where to redirect, probably adding a rule to somewhat distribute load across regional mirrors

And the case of missing a file on a mirror: make the HEAD-request before redirecting, which would be ofc slower but safer

I believe the team can come with better ideas than mine on that point

It seems a bit overkill

But I think it's mostly getting the design right

Thoughts? Questions?