Host downloadable images on alternative geographic servers

[chrisjsewell]

It has been noted for a recent "Winter School workshop" for participants in Australia, that the original download link on https://quantum-mobile.readthedocs.io/en/latest/releases/desktop/20.03.1.html was taking many hours to download.

This link is hosted on CSCS in switzerland: https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/quantum_mobile_20.11.0a.ova, and so it might be improved if the image was also available on a geographically closer server.

To test this I have added two alternative links to that page:

1. On the EPFL google drive (hosted?): https://drive.google.com/file/d/1Xfqn2kVffS4X9717iDFDFpPIKbODv88n/view?usp=sharing
2. On the aiida.net AWS VM (zone: us-east-1a): https://www.aiida.net/wp-content/uploads/quantum-mobile/quantum_mobile_20.03.1.ova

I will get some people to try downloading these and report relative speeds.

To host the image on the aiida.net server, I had to increase the container storage from 8Gb -> 16Gb (for future reference heres how you increase the size: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html?icmpid=docs_ec2_console).
This should not cost too much more (the cost I think is 0.04 USD/Gb/month), but obviously if we wanted to start hosting multiple images on there (each 4-5 Gb) you would have to increase the size considerably.
Also this is probably not going to do the automatic backups (on https://theossrv6.epfl.ch) any favours

[chrisjsewell]

Here in switzerland I get:

CSCS ~10 Mb/s
google drive ~15 Mb/s
aiida.net ~3.6 Mb/s

[chrisjsewell]

Thanks to @AakashGfude using Australia National Uni WIFI (Canberra):

CSCS - 6.5 Mb/s
Google Drive - 8.5 Mb/s
aiida.net - 7.8 Mb/s

[chrisjsewell]

Thanks to @mmcky, also in Canberra, but at home:

CSCS - ~129kb/s
Google Drive - 5Mb/s
aiida.net - 1.3Mb/s

So definitely more of a difference there

[giovannipizzi]

I guess you're limited by your network speed, for CSCS I can download at 37MB/s (300Mb/s, my cable connection speed, so their bandwidth is larger). Indeed, however, it'll be slow on the other side of the world.

Rather than putting it on aiida.net I would put it directly on Amazon S3 at that point. Note however that in both cases we pay for data transfer out of them.
Probably, if Google Drive seems to work fast, I would suggest to keep the CSCS link as main link, and provide the alternate link as well (via bitly to track # of downloads, important for reporting).

[chrisjsewell]

Rather than putting it on aiida.net I would put it directly on Amazon S3 at that point.

Yeh I wasn't sure exactly how to do that off-hand: as noted above, the aiida.net addition is probably not the way to go, but added temporarily as an additional data point.
If google drive works then I agree that is the simplest/cheapest solution

via bitly to track # of downloads, important for reporting

yep on https://quantum-mobile.readthedocs.io/en/latest/releases/desktop/20.03.1.html they are already bitly links

[ltalirz]

Hey guys, just mentioning that we applied for a Chinese mirror following the last tutorial ustclug/mirrorrequest#273 (comment)

It's live at https://mirrors.ustc.edu.cn/quantum-mobile-images/

[ltalirz]

Can we get a decision on this?

The download rates from ANU are clearly fine and demonstrate that this is not an issue of there being too little bandwidth to Australia in general but perhaps the university traffic gets higher priority than the one of the internet service provider at home.
The traceroute from Bern to Canberra goes via the US, which might explain the surprisingly large difference in download speed (10x) between aiida.net (running in North Virginia, US) and CSCS.

One suggestion might be to try downloading the Quantum Mobile through the university network if possible.
Depending on the location, it might also be worth checking out the Chinese mirror https://mirrors.ustc.edu.cn/quantum-mobile-images/ (see also the map of submarine cables).

I'm not sure whether we need to add more mirrors at this stage...

[giovannipizzi]

I would vote to limit the number of mirrors of possible, or anyway ideally having mirrors that we don't have to maintain, or that are super-easy to maintain - otherwise they will always be out of sync, or we will forget to sync them.

Thinking better at it, I would avoid AWS as we pay data transfer, and for the same reason aiida.net.
Maybe we can just decide to keep

• main CSCS download URL
• Chinese mirror since it's already up (if not too complex to maintain).
• Google Drive link (as it's easy to maintain, we just drop the files in one of our shared Google Drives, and we let Google take care of mirroring since it seems to be working very well).

@chrisjsewell @ltalirz what do you think?
If you agree, @chrisjsewell go ahead with syncing the various mirrors. Also, in any case, let's not forget to document this in the instructions to make a new release, so whoever does the release, remembers to upload the file in all three locations (or even less, but I would avoid to do more).

[ltalirz]

Just to clarify: the chinese mirror is not maintained by us - they automatically pull in appropriately named images from the CSCS storage (i.e. it doesn't cost us anything).
I don't know how long it usually takes for images to appear there (you can test with the next release), so to keep our release process simple, I suggest that instead of linking directly to a file there you simply add a generic link to https://mirrors.ustc.edu.cn/quantum-mobile-images/ to the release notes.

As for the Google drive, I would consider it optional but that's up to you.

[yaoge123]

http://mirror.nju.edu.cn/quantum-mobile/
https://mirror.nju.edu.cn/quantum-mobile/
rsync://mirror.nju.edu.cn/quantum-mobile/

Synchronization frequency: every day
Location: Nanjing, China, Asia
Sponsor: eScience Center, Nanjing University
Sponsor URL: https://sci.nju.edu.cn