[switch repo] Large packages like Retroarch take a few days before update is available

[jordaninthesky]

For the Switch NRO: The issue I am having is that sometimes updates for apps such as Atmosphere and Retroarch take too long to be reported. I will wait a couple days after a new release of Atmosphere, and see that the Homebrew Apps Store has yet to report on it... so I will update manually. Then after another couple days, the Homebrew App Store will tell me that I need to update it even though I am at the current version.

The most frustrating example I have is the most recent. Retroach was updated to 1.8.6. Then it was updated to 1.8.7. Even after 1.8.7 had been released for a few days, there was no change in the HB App store. So I went ahead and updated manually. So, when I open the app store, now it tells me that I need to update to 1.8.6 when I am already sitting on 1.8.7. The idea of updating would be so useful through the App Store if it weren't so counterproductive. I have to remember what versions I have and check it off manually, and as much as I would like to have an App like this, it seems like it is more work keeping track of what it falls short in doing.

[vgmoose]

There's a couple things going on in this issue, two are related to how the client tracks versions, and then one is a human/maintenance problem with the default provided repo.

I've written a lot here, so I'll preface this that most of a lot of this is RA-specific. Most apps in our repo update quickly and simply. Atmosphere is another one that has a few considerations (although we've been better at providing AMS updates recently).

Problem 1: human/maintenance problem

tldr: heart of this issue would've been solved if I had just updated it faster (and if I continue to update it quickly every future release)

When you updated Retroarch, you grabbed their several hundred MB zip off their website and unarchived the .7z file onto your SD card, using a 7z tool?

This process is not one that we have managed to automate very well. The situation with our Retroarch package that we host is that it's a subset of the official releases. We have stripped out some linux specific files, cores, and other large files/themes that we don't want in the app store package for taking up too much space.

This means that we have to manually go through after noticing there's been an update and pull out the files that we want to include into the package, and this process can take a while depending on what other things we're doing... In the case of RA I "just" did it for 1.8.6 and have been "putting off" doing it again for 1.8.7.

In the past this approach was more acceptable, but it seems RA is beginning to update more frequently. For me, it's hard to get motivated to continue updating using this method until we automate it better, and have a way to have a script do the process that I am manually going through.

On the RA side:, this is the script which packages build bot releases: https://github.com/libretro/libretro-super/blob/master/libretro-buildbot-recipe.sh#L1337-L1352

Which includes everything, including some .sh and .py files, and some potentially themes that don't apply to the Switch package (other platforms do decide to selectively strip out more)

On our side:, we have our spinarak project: https://gitlab.com/4TU/spinarak/-/tree/zipit which does not yet handle .7z files, but if it did would be a decent solution to this sourcing problem.

Ultimately, as time goes on, if RA continues to update at this pace, we will find a better automation solution. It just has not been enough of an issue until recently to continue looking into.

One more note: Although we provide the default repo, hb-appstore is configurable. If another entity wants to work with us to help us keep our repo up to date easier, or go their own path and host their own Retroarch libget repo for hb-appstore integration, they should feel free to do so. We can be contacted here, or libget's wiki goes into detail about how to host a repo.

We understand that our repo's Retroarch package is serving a common denominator of what users need. Most other packages are sync'd with Github and get releases right away, but RA has a broader set of concerns with cores, nightlies, configurations, themes, that our default repo (and the maintainers like myself) are not currently equipped to handle.

Problem 2: We don't use integer version codes

The current version check is string equality which is not the greatest, as you noted this can report "updates" that are actually downgrades. A fix for this is to use incrementing "version codes" in additional to the human readable version strings.

Android and iOS do something similar, which is track an only-increasing positive integer within their metadata file (so in our info.json). This is touched upon in fortheusers/libget#21 but not outrightly stated.

I created a new issue here, just for tracking the version codes: https://gitlab.com/4TU/libget/-/issues/7

Problem 3: We don't properly identify existing homebrew apps

When you updated it manually, I am surprised to hear that hb-appstore did detect an "update", as right now we only track apps that come from the store. In other words, if you manually replace the files but don't touch the libget metadata, it should still think that your "1.8.7" is a "1.8.6" install, and then only show "Update" when we actually do get around to updating it.

The issue is that the NRO version string is not a very reliable method of detecting what version of an app is installed (Although, in RA's case it would work pretty well). On Wii U, we can use meta.xml files to track the version number, but on Switch we've been talking about other methods that can get more creative.

One solution: hint files: https://gitlab.com/4TU/libget/-/issues/1
Much more on this issue: #49

So what now
I'm going to leave this issue open until we have a better solution for automatically updating our package from RA's build bot (Problem 1). It's one of those things that in theory isn't too difficult, but in practice we've spent too long trying to work on a "good" solution, and have been focusing on other things instead.

We must have like three different (old/halfworking) versions of a 7z-extracting scripts lying around that sync with our repo, and on at least two occasions I know we've brought people on to specifically be Retroarch-maintainers but they have disappeared over time. Not updating RA quickly enough has been plaguing us for a while.

I'll also finally go as far to suggest that, to me personally, a few days or even a week's lag on RA's package or other large packages in the grand scheme of things is not the end of the world, or enough for me to consider hb-appstore's offerings as "counterproductive". (Hence why it's easier for me to "put off" updating it)

I'd imagine that most consumers of the RA package just want to play a few games and aren't too focused on it being immediately up to date (also RA's built-in updater takes some responsibility away from us too). It's nice to have it up-to-date ASAP of course, but for most people I'm not sure they mind that much.

In other words, I'm suggesting it's a "power-user" thing to be aware of a new RA update early enough to the point that you're manually copying the files over, and that most of our audience isn't that plugged in, or even has the willingness/devices to manually copy them over.

THAT BEING SAID of course, we shouldn't have delays where we can avoid them, and should work to address them. Especially if RA keeps updating at this pace.

The other two technical problems (2 & 3) can be tracked in the issues linked out for them.

[vgmoose]

Currently, our hosted switch RA package subset is as follows:

• delete all /retroarch/cores except: snes9x, gambatte, mgba, nestopia
• delete all assets from /retroarch/assets/xmb, except: dot-art, flatui, monochrome, pixel
• delete all folders from /retroarch/, except: assets, autoconfig, cores, info
• delete /retroarch/autoconfig/android (probably other autoconfigs can be removed as well...)

Which results in a final package compressed size of ~50MB, (~95MB, 2756 files uncompressed). Compare at 217MB compressed, 1.62GB decompressed from the full buildbot package (280MB, 8356 files if you only include the same 4 cores).

I didn't have this list on-hand, just have constructed it for the first time here. Every time we update, I usually visually diff the release against our previous package. (I tested a long time ago a bunch of different subsets to see what I was willing to cut/remove and still provide a good user-experience). Hopefully this list helps going forwards, or any automation efforts.

Due to the visual diff-ing, sometimes I'd forget a directory, and end up uploading thousands of android/linux files into the package which results in a bad time for everyone. Probably adjusting the buildbot .sh script is enough to address a lot of these frustrations, but I'm not well-versed enough in Retroarch to know which specific files should be cut or kept.

Any input is welcome (from anyone who may be reading this)! For instance: Is this a good idea? Are these good cores? Any other essential files we need? Any files that we definitely don't? Which ones should be removed from the buildbot scripts, if any?

[jordaninthesky]

Thank you for your very informative post. I was unaware that you were gutting the zip file of redundancy and baggage--I had thought it was downloading and extracting the whole package. You have obviously put in a lot of work into this, and I do not intend to be dismissive. The counterproductive comment was from an End User who falls to an extremity of your Target Audience. Of course, it would be ideal that everything could be automated, but now I understand why the human element is difficult to reproduce.

To clear something up: "I am surprised to hear that hb-appstore did detect an 'update'"--it did not detect an update in comparison to my current install, but in comparison to a past install done through the App Store, as it intended. I was trying to express the irony of having upgraded two versions ahead (manually) only to have an upgrade available reported by the App Store, to which would actually be a downgrade. I botched up the explanation, but you still expressed understanding of what happened.

All-in-all, it is not the end of the world, as you say. I like the app and hope to see it continue in development. The RA solution is not something I feel like I can help with, but I do hope something works out. A workaround to those (like me) do not want to see the red circle for an app that they have manually updated to "skip this version". It's a thought that might not have value in the larger scope of this project and I don't want to divert attention away from places better spent. Thank you for the time that you have put into this project and your explanation.