Investigate alternative hosting options for nightly builds

[J5lx]

Our nightly builds built via AppVeyor and Travis CI are currently hosted on Google Drive. This location was originally chosen because it was already being used for hand-crafted „nightly” builds before, however, as mentioned several times before, it is not a particularly great solution for hosting our builds. On one hand it’s a rather „closed” solution that doesn’t fit the Open Source nature of Pencil2D very well, but it’s also not exactly aimed at software distribution, it’s more of a personal storage space. Additionally, the usability of Google Drive is not the best as users are having difficulties finding the latest builds (Added Jan 11, see here). We should therefore try to find another solution that is a better fit for our use-case, or even distribute our builds on multiple services (added Jan 11).

These are some alternatives that have been brought up before and that I can think of:

Service	Storage	Traffic	Bandwidth	Upload	Statistics	Cost	Notes
SourceForge (our project page)	fair use	unlimited?	>=150Mbps, varies	SCP, rsync (over SSH), SFTP	Download counts over time, Countries, OSes, API	Free	They had some reputation issues related to their hosting of software releases in recent years, but I haven’t heard anything of that since the last change of owner. If we decide to use this option, we should probably also do some housekeeping on the project page since it currently still holds a bunch of content from the „other” Pencil2D which previously used that page.
Bintray Bintray is being shut down and replaced by JFrog Platform, which does not seem to fit our needs	10GB (Open Source Plan)	1TB/mo	?	proprietary HTTP API	Download counts over time, Location, API	Free (Open Source Plan)	I don’t know much about this but I have heard it mentioned here and there a few times and they have a free Open Source plan which should more than cover our needs. Look here for an example what this could look like for us (I haven’t investigated customization options etc. though)
GitHub releases	unlimited	unlimited	?	proprietary HTTP API	Download count (API only)	Free	These are problematic because they require tags on a GitHub repository. Creating these tags on the main Pencil2D repository would quickly pollute our releases page with dozens of unstable releases, and creating a separate repository just to hold the release assets is not a particularly elegant solution either.
[Updated Jan 14] Google Drive with revisions	15GB, 100 revisions	unknown limit?	?	proprietary HTTP API	None	Free	More details here. Possibly prevents general access to older builds.
[Updated Jan 14] Amazon	?	?	?	?	?	?	?
Self-hosted	Varies / flexible	Varies	Varies	Varies / flexible	Varies / flexible	Inexpensive	Potentially a little more effort to set up and some expenses (but not much, online storage is dead cheap and I already have a server where we can put some sort of overview page if needed) but we could basically do with it whatever we desire.

If you know other solutions which are worth checking out, put them in the comments and I’ll add them to the list above. Other than that, let’s hear what everyone thinks about this.

[J5lx]

Personally, my order of preference for the options above is 1) SF/self-hosted 2) Bintray 3) GitHub releases. Obviously self-hosted gives us the most control, the drawbacks I listed above aren’t that much of an issue to me. SF, while it has declined since the rise of GitHub, is still a big and recognisable name in the Open Source world and I like its upload options. Compared to the other options, Bintray is much more „closed”, but I still like it better than GitHub releases since I think the latter is pretty unsuitable for something like nightly builds.

[scribblemaniac]

On one hand it’s a rather „closed” solution that doesn’t fit the Open Source nature of Pencil2D very well, but it’s also not exactly aimed at software distribution, it’s more of a personal storage space.

I'm not really convinced that either of these are good enough reasons to justify switching. Let's not forget that SourceForge, Bintray, and yes even Github, are for-profit companies just like Google. SourceForge is a great example of how free open source does nto necessarily equate to trustworthiness. For those not familiar with the SourceForge incident (which to be fair was with a previous owner), see this article.

The way I see it, the bigger issue is not with where our files are being hosted, it's with how they are being presented to the users. Some of our users have been having difficulties finding the correct nightly builds with the current hosting solution. I see this as a better reasons for switching. I think that there is also value in mirroring our nightly builds across multiple services so this would be a reason to add an additional rather than a replacement hosting option.

Having said all of that, I would like to present a couple of options additional options:

• I brought this up very briefly when reviewing Check for updates & forum/discord links in Help menu #1126, but an option is to use the revisions feature of Google Drive. Here's the relevant quote:

  Google Drive does have a "revisions" feature which automatically deletes old revisions after a month or 100 revisions and keeps the same link so you could have a "latest nightly build" link for each OS which is pretty nice. Unfortunately it doesn't seem that you can access the revision list without OAuth, so if we're going to add a nightly build check with the API, it would not be a good idea to switch to this.

  The last part is actually incorrect, we could run a check by getting the lastModified date from the files query that we would currently use if we had a functional nightly build update check. It's also possible that we could use APIs from Github, Travis, or Appveyor to fetch the last successful nightly build date. The code for uploading builds as revisions has actually already been written as part of my workflow restructuring [WIP] Workflow Restructuring #1152 (Note: I haven't uploaded the actual script yet because of some unidentified issue with git, but it is finished and has worked in testing).
  The key takeaway from this option is that revisions can be used to create a fixed latest nightly build link for each OS, which is great.

• Custom interface for nightly builds. Regardless of which backend we use, it would be nice to have the links show up on our actual website. I am envisioning having a link that would always point to the download for the latest nightly build on the main download page, and then a separate page that would list all available nightly builds with their date, direct download links, mirrors, etc. For this to be possible we would need to either 1) Expose our hosting API keys and dynamically load the links with Javascript, 2) Use some server-side scripting to generate the page on a self-hosted server like the forum, or 3) push changes to the website repository with the nightly build CI. None of these solutions are ideal, but I think we need to implement at least one of them to perfect the user experience.

One feature to add to consideration is download statistics. Strangely I don't think Google Drive has download statistics, but it would be nice to know how frequently our nightly builds are being used. Support for this could easily be added with a self-hosting solution. SourceForge definitely has a basic download count. I do not know about Bintray.

There are probably other considerations such as download speed, storage and bandwidth limits, expenses, API ease-of-use and any other restrictions of each of the hosting options that should be weighed too before we make our decision.

The only solution from all of the ones that have been presented so far that I would be explicitly against is using the Github releases for nightly builds. As mentioned it would pollute the releases tab which currently makes it quite easy to find past stable releases. My current preference order for hosting solutions is:

1. Google Drive (I'm lazy alright 😛 )
2. Self-hosted
3. Bintray/SourceForge (still need to do more research on the former personally)
4. Amazon
5. Github releases

[J5lx]

I agree that Google Drive’s “closedness” might not be a particularly strong argument in favour of switching, but I’m under the impression that it’s still the argument that has been brought up the most (correct me if I’m wrong though).

It’s also true that all of the options I listed except self-hosting are from for-profit businesses, but I think the issue isn’t so much that these companies are trying to make money but rather how they deal with their content. Google has some strong support for Open Source (think GSOC, for instance), but this isn’t exactly exemplified by Google Drive in particular. SourceForge on the other hand is and has always been aimed at Open Source projects. As mentioned they’ve made some pretty bad mistakes in handling these under their previous owner (Dice), but they were sold three years ago (to Bizx) and personally I haven’t heard anything negative about the site ever since. In fact, most of the changes they made were received quite warmly as far as I’ve seen. (About Bintray I don’t actually know much.) That’s why I personally think that Google Drive is a worse option for us than, say, SF.

That said, what you mentioned about the usability of Drive is also an absolutely excellent point. I think it ties in with Drive being more of a personal storage space rather than a solution for software distribution. I also had a look at the revisions feature you mentioned, but I don’t really like it that much. Yes, it makes it easier to determine the latest version and being able to publish a fixed link is particularly great, but it makes it much harder to find older builds, should need be (personally I had to search for that particular feature to find it). Plus, it looks like older versions aren’t even accessible at all to non-owners, at least I couldn’t find a way to do it on a test file I created manually. As I see it, these revisions may solve one problem, but they create others. If anything, we’d have to find a way to let anonymous users download old versions and build our own overview site to make them accessible in order for this feature to be truly useful, but that’s just my opinion.

As for the custom interface, this is an idea I wholeheartedly support. It’s what I meant with the “overview page” for the self-hosted option, but creating such a page as a frontend to other hosting solutions is of course a great idea, I didn’t even think of that. Of the implementation options you listed I personally prefer the server-side solution – I really don’t like relying on JavaScript where it isn’t strictly necessary, and I don’t want to make our CI setup even bigger and more complicated than it already is.

I also second your thoughts on statistics and additional considerations. I’ll try to add some info to the initial post in regard to the latter soon.

Lastly, could you elaborate on Amazon a little? You only listed it in your preference order but didn’t really say anything about it. Unless there is a good reason I’d rather avoid it because I don’t really like Amazon as a company, at least here in Germany they have a somewhat questionable reputation in regard to the treatment of their warehouse workers and the way they use their market power. If you’re thinking of S3 we could also consider some of the independent providers offering S3-compatible storage.

tl;dr:

• I’m not sold on Google Drive revisions
• But fixed links for the latest nightly builds are a great idea
• So is a custom overview page
• I don’t like Amazon

[Jose-Moreno]

Hmm I agree that most of the issues with nightly builds are related to user interaction and presentation, however there is one problem that has been addressed indirectly and it's the size of the storage itself.

I personally would have liked previous nightly builds to be available to the public in order to have a historical overview of the builds as well, but if you've been attentive, chchwy has had to delete plenty of nightly builds from the past years and months, so that history has been lost to the devices of google drive, and his space storage is otherwise shared storage as J5lx noted, so even if he's generously lending Pencil2D that repository it is limited to his own use of the space. However that is just a quirk I have about google drive, other than that I agree it serves its purpose quite faithfully (except that one downtime where we couldn't access the builds at all)

For now maybe we could try to implement a front-end for the nightly builds, perhaps as an additional nightlies download page only accesible from the "downloads" section itself, that parses the current Google Drive repo and displays the links with the last modified order, because sometimes people make the mistake of getting the first one without organizing it so they download an actual older version and then complain about the same old bugs we've already fixed, which is why I countered that by giving direct links every time I interact with a user, since they have a hard time browsing the drive itself.

As for the other "back-end" options, well SF was proposed since it is mainly a unused repository that currently holds the original history of the project. There's also bitbucket where chchwy has a sourcecode mirror for the Pencil2D repository. Github was proposed because it's close, easy to parse and can be related to the same organization, what I saw Opentoonz do is they made a new repo strictly for nightlies. I don't know much about bintray I can't comment on that. Regarding amazon, I don't hate amazon, but if you guys don't like let's not try that out either.

One thing that worries me though and we have to be careful once we do this, is that if a file is downloaded too much it might lock down the file or the folder itself preventing people from downloading the nightlies for a period of time. As you can read here there are ways to fix that 😉 but of course realistically speaking we would probably need a lot of people downloading the same file on a short amount of time for that to happen, but still is something to take into account as well.

[scribblemaniac]

Plus, it looks like older versions aren’t even accessible at all to non-owners, at least I couldn’t find a way to do it on a test file I created manually.

I did not know that. If that is the case, then it definitely could be a deal-breaker for using revisions. The whole point of using revisions was with the intention of allowing access to a reasonable number of previous builds.

Lastly, could you elaborate on Amazon a little? You only listed it in your preference order but didn’t really say anything about it.

Well I did write a section where I brought up the possibility of using Amazon because I've see it used for similar purposes before, but I do not know to much about it. I ended up removing it, but forgot to remove Amazon from my final list, so you can ignore it unless someone comes up with a good reason to use it.

chchwy has had to delete plenty of nightly builds from the past years and months

Actually our build script automatically deletes anything older than 90 days. I don't think this is a big issue because we don't really want people to use our old unstable builds, and it will always be possible to rebuild them from source if necessary so nothing is lost forever.

[J5lx]

I just added some additional information (storage, traffic, bandwidth, etc.) to the list in the OP (unfortunately it became pretty cramped, though).

I personally would have liked previous nightly builds to be available to the public in order to have a historical overview of the builds as well

Agreed.

[chchwy’s] space storage is otherwise shared storage as J5lx noted, so even if he's generously lending Pencil2D that repository it is limited to his own use of the space.

Actually the nightly builds belong to a separate service account which doesn’t share its quota with any other account (see also #617 (comment) and #617 (comment)).

[MrStevns]

Whatever we choose, the host has to support direct file download. We don't want to redirect users to external pages if it can be avoided.

I agree that older versions should still be possible to look through so I see no reason why we couldn't keep google drive just for that.

That means we need a host for potentially just one file, so whatever is easiest to setup and gets the job done imo. should be fine.