Binary patching and FOMOD Presets

[halgari]

This all ended up being both harder and easier than I expected.

Binary patching - it's implemented and works on the unit tests I included in the code. It's a bit harder than one would expect as often binary patching is paired with replicate that uses a MD5 hash to identify files to install to specific locations. Here's the catch, the MD5 in this case is the MD5 after the patch, and no one ever thought to provide a link between MD5 hashes and patches. And the patch's hash is a CRC32, because why not?

This all means that you kindof have to hash everything in the mod, then apply all patches, then hash any patched files and then you have what you need to install the files. In our case we store the patched files and then link them up by xxHash like any other file.

FOMOD Presets - this ended up being harder than I thought thanks to the utterly broken fomod-installer project. In the end I gave up on trying to pass the options into the installer. Instead we implement a new IGuidedInstaller and emit UserChoice objects based on the presets. The resulting solution is quite clean, just took me a good 4 hours to figure it out.

TL;DR - two of the biggest blockers with collection installs are now solved.

[github-actions]

This PR conflicts with main. You need to rebase the PR before it can be merged.

[github-actions]

This PR doesn't conflict with main anymore. It can be merged after all status checks have passed and it has been reviewed.

[Pickysaurus]

Related #1667

[Sewer56]

Do we know if the names are case sensitive? I imagine the preset choices aren't set by humans, but if they can be modified by humans, a case insensitive comparison may be better here.

---

Edit:

Do we also know if the group names are supposed to be unique?
This code assumes that there may be multiple groups with the same name.

i.e. The input does not terminate when installGroup.Name == srcGroup.name, instead, it continues to process all other possible pairs of groups.

[halgari]

We need to talk with the Vortex devs and figure out these questions, it's not entirely clear to me what the interpretation should be. This works in the cases I tested, but we should find out more

[Sewer56]

The other feedback is mainly up to whatever you want to do with it; however this piece is blocking.

NexusMods.App/src/Abstractions/NexusMods.Abstractions.IO/IFileStore.cs

Lines 79 to 89 in 6444ae4

	/// <summary>
	/// Completely loads the file into memory
	/// </summary>
	async Task<Memory<byte>> Load(Hash hash, CancellationToken token = default)
	{
	await using var stream = await GetFileStream(hash, token);
	var memory = new Memory<byte>(GC.AllocateUninitializedArray<byte>((int)stream.Length));
	await stream.ReadExactlyAsync(memory, token);
	return memory;
	}

I have a small number of issues with this:

• Every chunked size read goes through the entire caching mechanism, which involves a O(N) lookup in LightweightLRUCache for every ChunkSize (usually 1MiB).

  • Because we're always reading forward (ReadToEnd), every read is a cache miss, so we sequentially iterate 16 keys every 1MB read, none of the keys hit, and we rent a new set of bytes from the pool each time.
  • LightweightLRUCache (capacity 16) always rents from the MemoryPool when we add a new cached chunk.
  • Because we have 16 * NumThreads active rentals, number of rentals from global is going to eat a lot of memory, and likely to run out on high core count processors.
  • In fact they will run out. In Shared ArrayPool from runtime, the rentals are per core, (see runtime code), we probably shouldn't use more than 8 at once if using the global pool otherwise every pool rental is a full allocation. (i.e. Reduce LightweightLRUCache capacity to 7, or to 8, and reuse rentals)
  • Because every read is a cache miss, every rental (1MB read) will become a full allocation.
  • tl;dr every 1MB read after 8MB of file is a full allocation; and every read is a cache miss that iterates over 16 keys.

• This will always run on a single thread, when larger files could be extracted in parallel.

  • If we need control over thread count, that can be a parameter.

I feel very uneasy throwing this specific part to the main branch. At least as part of the IFileStore interface.
Just make an ExtractFile method and strip down the existing code from ExtractFiles to single file; it should end up fairly short in terms of LoC. Thread count is part of Nx' UnpackerSettings

---

Misc Note:

Bsdiff patches can also technically be bigger than 2GiB in file size, but that point is very moot; most code implementations limit files to 2GiB even if format supports it; and nobody makes patches for files this big because memory usage is astronomical (17 times the input size). Technically with diff algorithms you should be using streams in general; however in this case a full read to RAM is unavoidable since we need to read the file twice to hash it. Without very custom code.

---

Edit: I also checked GC stuff; no issues there.

[Sewer56]

Ah, actually, before you merge, you may want to have a way to adjust thread count via parameter. Nx will default to number of physical cores.

Since you're doing the work for bsdiff in parallel already, you may want to limit thread count when calling Load here, so you don't wind up making NumCores^2 threads.

Edit: Either here (Nx via parameter) or on the other end where the patch apply happens.