Index files from Steam #2385

halgari · 2024-12-12T17:03:18Z

Adds a CLI verb for indexing game files for use in the game-hashes repo. This PR also does the following

Adds support for logging into steam, and accessing product info
Adds support for getting manfiest info from the steam content network
Adds support for reading files from Steam's CDN
Updates the Chunked stream reader so that it can handle blocks of any size (previously all blocks but the last had to be the same size)
Adds a MultiHasher for hashing a stream effiently into all the various hash formats we would need
Adds value types for SHA1, MD5, and CRC32.
Adds hashing logic for the minimal hash
The CLI verb creates all these variants for every file it hashes, along with xxHash3, and xxHash64
Adds value types for all the Steam values

How to Test:

There's a test called CanDownloadProuctInfo by default this test is set to skip.
Enable the test (remove the skip reason)
Run the test
The logs for the test will present a QR code, use the steam app on your phone to scan this QR code
The test will download and the data for SDV

Alternatively:

Run the index-steam-app verb
It will ask you to log in via a Steam QR code
It will download all the current versions of SDV and index them (not saving any of the game files, it indexes purely in a streaming method)

…d in

…ced a .Hashes abstraction library for hash value types. Moved Chunked streams into the IO abstractions library

erri120 · 2024-12-16T10:14:32Z

Abstractions/NexusMods.Abstractions.Steam/DTOs/Depot.cs

+    /// <summary>
+    /// The OSes that the depot is available on.
+    /// </summary>
+    public required string[] OsList { get; init; }


If we have the OS list, we could also get the languages of a depot.

erri120 · 2024-12-16T10:24:11Z

Abstractions/NexusMods.Abstractions.Steam/DTOs/ManifestInfo.cs

+    /// <summary>
+    /// The size of the downloaded files
+    /// </summary>
+    public required Size Size { get; init; }
+
+    /// <summary>
+    /// The size of the files on the CDN
+    /// </summary>
+    public required Size DownloadSize { get; init; }


Why are these two sizes different?

Presumably one is compressed, other is decompressed.

DownloadSize is compressed, Size is size-on-disk (uncompressed).

To be consistent, we should name these properties CompressedSize and DecompressedSize.

I'm going to veto this one, we're going to process this data and import it into a MnemonicDB database before pulling it into the app, so the names here aren't the ones we'll be using anywhere in the app, instead they're used when we're talking directly to SteamAPIs, and it's important that we don't manipulate this data too much or try to adapt it to some other format.

erri120 · 2024-12-16T10:25:22Z

Abstractions/NexusMods.Abstractions.Steam/DTOs/Manifest.cs

+        /// <summary>
+        /// The size of the file, compressed
+        /// </summary>
+        public Size Size { get; init; }


The other types have both compressed and decompressed sizes, for consistency this should be named CompressedSize and we should add DecompressedSize if we have it.

I'm matching what SteamKit provides here, I don't mind changing it, but then we'd be out-of-sync with the "official" source

I'd be happy if we have something that is consistent internally.

See the comment on the other section

Abstractions/NexusMods.Abstractions.Steam/DTOs/ProductInfo.cs

Abstractions/NexusMods.Abstractions.Steam/DTOs/Manifest.cs

src/Networking/NexusMods.Networking.Steam/Session.cs

src/Networking/NexusMods.Networking.Steam/NexusMods.Networking.Steam.csproj

erri120 · 2024-12-16T11:20:57Z

src/Networking/NexusMods.Networking.Steam/NexusMods.Networking.Steam.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+


Might be missing an import of directory.props.

src/Networking/NexusMods.Networking.Steam/ManifestParser.cs

src/Networking/NexusMods.Networking.Steam/ProductInfoParser.cs

Co-authored-by: erri120 <[email protected]>

….Steam.csproj Co-authored-by: erri120 <[email protected]>

Co-authored-by: erri120 <[email protected]>

…er.cs Co-authored-by: erri120 <[email protected]>

….App into steam-login

halgari added 8 commits December 11, 2024 10:12

Basic structure of the projects, can login via a test, and stay logge…

47595e6

…d in

Implemented the rest of the Steam flow for getting manifests. Introdu…

669d794

…ced a .Hashes abstraction library for hash value types. Moved Chunked streams into the IO abstractions library

Boom, can stream files off of Steam

f52f6a9

Can download files, serialize manifests, all that good stuff

c6d7703

Fix the chunked stream so that it can use arbitrarily sized chunks

49b3d9c

Delete the old Store project

c90d121

Disable the test that has to be run with human interaction

eb32afd

Few more comments and cleanup

ed8f39f

halgari marked this pull request as ready for review December 12, 2024 17:06

halgari requested a review from a team December 12, 2024 17:06

erri120 self-requested a review December 13, 2024 16:08

erri120 assigned halgari Dec 13, 2024

erri120 requested changes Dec 16, 2024

View reviewed changes

halgari and others added 15 commits December 16, 2024 06:26

Update Abstractions/NexusMods.Abstractions.Steam/DTOs/ProductInfo.cs

d64c215

Co-authored-by: erri120 <[email protected]>

Update src/Abstractions/NexusMods.Abstractions.Hashes/Crc32.cs

f734e43

Co-authored-by: erri120 <[email protected]>

Update src/Networking/NexusMods.Networking.Steam/NexusMods.Networking…

b58d349

….Steam.csproj Co-authored-by: erri120 <[email protected]>

Update src/Networking/NexusMods.Networking.Steam/Session.cs

05e5f85

Co-authored-by: erri120 <[email protected]>

Update Abstractions/NexusMods.Abstractions.Steam/DTOs/Manifest.cs

0c20eb3

Co-authored-by: erri120 <[email protected]>

Update src/Networking/NexusMods.Networking.Steam/ProductInfoParser.cs

fd26dc2

Co-authored-by: erri120 <[email protected]>

Update src/Abstractions/NexusMods.Abstractions.Hashes/MultiHasher.cs

869a959

Co-authored-by: erri120 <[email protected]>

Update src/Networking/NexusMods.Networking.Steam/Session.cs

752486b

Co-authored-by: erri120 <[email protected]>

sizeof(ulong) vs size(long)

6815ce4

Update doc strings to better explain the value sizes

f6e59ce

Update src/Networking/NexusMods.Networking.Steam/ManifestParser.cs

782c74f

Co-authored-by: erri120 <[email protected]>

Update src/Abstractions/NexusMods.Abstractions.Hashes/HashJsonConvert…

a5dd382

…er.cs Co-authored-by: erri120 <[email protected]>

Fix the .Steam project

6cca217

Merge branch 'steam-login' of https://github.com/Nexus-Mods/NexusMods…

71c769a

….App into steam-login

fix compile errors introduced during code review

f2b3e6b

halgari merged commit f79dd35 into main Dec 17, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index files from Steam #2385

Index files from Steam #2385

halgari commented Dec 12, 2024 •

edited

Loading

erri120 Dec 16, 2024

erri120 Dec 16, 2024

Sewer56 Dec 16, 2024

halgari Dec 16, 2024

erri120 Dec 16, 2024 •

edited

Loading

halgari Dec 17, 2024

erri120 Dec 16, 2024

halgari Dec 16, 2024

erri120 Dec 16, 2024

halgari Dec 17, 2024

erri120 Dec 16, 2024

		<Project Sdk="Microsoft.NET.Sdk">

Index files from Steam #2385

Index files from Steam #2385

Conversation

halgari commented Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erri120 Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halgari commented Dec 12, 2024 •

edited

Loading

erri120 Dec 16, 2024 •

edited

Loading