From 78ace0fa9c91831cfeb505a21a0488161b1b6865 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 19 Sep 2024 02:39:51 +0100 Subject: [PATCH 01/15] WIP: Documentation for Update Detection --- .../decisions/explanations/01-update-logic.md | 360 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 361 insertions(+) create mode 100644 docs/developers/decisions/explanations/01-update-logic.md diff --git a/docs/developers/decisions/explanations/01-update-logic.md b/docs/developers/decisions/explanations/01-update-logic.md new file mode 100644 index 0000000000..117b5ed91c --- /dev/null +++ b/docs/developers/decisions/explanations/01-update-logic.md @@ -0,0 +1,360 @@ +!!! info "Supporting ***generic*** mod updates on the Nexus today is a tricky subject." + +## Problem Statement + +!!! info "In other words, what is the context we're working with." + + As of 18th of September, 2024. + +### No Standardization + +!!! info "There is no 'standardized' way to support mod updates on Nexus." + +Users typically: + +- Upload some loose files +- Provide install instructions in descriptions +- Mod Managers are expected to make sense of these files + +*Automation on the website side is not viable.* + +Some modding frameworks like SMAPI or Reloaded have unique identifiers per mod; which +can be used to support reliable, robust update systems. + +However, the vast majority of mods on the site use a varied amount of modding frameworks. +Many of these are not suitable to support mod updates natively. + +Example: How do you update a DLL file with no additional context? + +In other words, there is little 'business case' to request a more robust update system +on the website side of Nexus. + +### Limited Update Metadata + +!!! warning "Nexus has limited on-site support for mod updates (as of today)." + +A generic, limited 'update' system is supported on the Nexus. + +When a mod author uploads a new file to the site, they can specify a previous file +for which this file is an update for. + +However, this system has limitations: + +- You set the 'previous version' on upload. +- The 'previous version' cannot be overwritten once set. +- The `previous version` -> `next version` is a one-to-many relationship. + - A user is allowed to set the same `previous version` on many files. + +## Existing Implementations + +!!! info "Some existing implementations of ***generic*** mod updates over Nexus exist." + +The most commonly used implementations are in [Mod Organizer 2] and [Vortex]. +Below is a rough simplified breakdown on how they both operate. + +### 1. Determine Updated Mod Pages + +!!! info "First we must determine which mod pages have been updated." + +Both [Mod Organizer 2] and [Vortex] rely on the V1 API for this. + +We query `/v1/games/{game_domain_name}/mods/updated.json` to obtain the most recently +updated mods for a given game. This gives us mods updated in ***the last month*** (max API limit). + +We can then check the `latest_file_update` timestamps. + +Example response (truncated): + +```json +[ + { + "mod_id": 167, + "latest_file_update": 1725035688, + "latest_mod_activity": 1725035918 + }, + { + "mod_id": 198, + "latest_file_update": 1715015733, + "latest_mod_activity": 1725386934 + }, + { + "mod_id": 266, + "latest_file_update": 1721451025, + "latest_mod_activity": 1725213667 + } +] +``` + +We match the response against our locally installed mods. + +If any of our mods is in the response, and the `latest_file_update` timestamp is older +or equal to our locally saved cached response; we don't query the mod. + +Otherwise, if the mod hasn't been checked in over 1 month (max API limit), or its +`mod_id` is not present in the response, we will have to +[query the files for an update](#2-querying-mod-files) individually + +### 2. Querying Mod Files + +!!! info "Traverse the `previous version` -> `next version` mappings defined by the user." + +We query the mod page for its files using the V1 API. +`/v1/games/{game_domain_name}/mods/{mod_id}/files.json` + +Example payload: + +```json +"files": [ + { + "file_id": 207864, + "name": "Unofficial Skyrim Special Edition Patch", + "version": "4.2.5a", + "category_name": null, + "file_name": "Unofficial Skyrim Special Edition Patch-266-4-2-5a-1623034459.7z", + "uploaded_timestamp": 1623034459, + "uploaded_time": "2021-06-07T02:54:19.000+00:00", + "mod_version": "4.2.5a", + "description": "" + }, + { + "file_id": 522942, + "name": "Unofficial Skyrim Special Edition Patch", + "version": "4.3.2", + "category_name": "MAIN", + "file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-2-1721451025.7z", + "uploaded_timestamp": 1721451025, + "uploaded_time": "2024-07-20T04:50:25.000+00:00", + "mod_version": "4.3.2", + "description": "Meant for use ONLY with SSE v1.6.1130+." + }], + "file_updates": [ + { + "old_file_id": 449151, + "new_file_id": 449719, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0-1701835628.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0a-1702019266.7z", + "uploaded_timestamp": 1702019266, + "uploaded_time": "2023-12-08T07:07:46.000+00:00" + }, + { + "old_file_id": 449719, + "new_file_id": 491975, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0a-1702019266.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-1-1713394824.7z", + "uploaded_timestamp": 1713394827, + "uploaded_time": "2024-04-17T23:00:27.000+00:00" + }, + { + "old_file_id": 491975, + "new_file_id": 522942, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-1-1713394824.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-2-1721451025.7z", + "uploaded_timestamp": 1721451025, + "uploaded_time": "2024-07-20T04:50:25.000+00:00" + } +] +``` + +(Query: `https://api.nexusmods.com/v1/games/skyrimspecialedition/mods/266/files.json`) +Some fields removed and results trimmed. + +The `previous version` -> `next version` mappings are defined in the `file_updates` +array. Suppose we downloaded a mod before which has an `old_file_id` of `449719`. + +To find mod updates, we need to traverse the `file_updates`, find the entry with +`old_file_id` of `449719`, and save the `new_file_id` of that entry. Repeat this process +until there is no entry with the current `old_file_id`. + +### Reference Source Code + +- [Mod Organizer 2 Update Source][mo2-update-source] +- [Vortex Update Source][vortex-update-source] + +## Technical Problems + +!!! info "Some edge cases need to be worked around when handling updates." + +### Archived Mods + +!!! info "Mods on Nexus can be 'archived' which hides them from search results." + +Recall the earlier example, there was an entry with `449151`, but +there is no item with `file_id` of `449151` in the `files` array. + +This was not an omission from the original response when trimming in the example. +The response does not contain 'archived' files. + +Since the introduction of 'collections' at Nexus Mods, mods can no longer be removed, +but they can be 'archived' to hide them. + + +!!! tip "We must manually query the individual files with the V1 API." + + `/v1/games/{game_domain_name}/mods/{mod_id}/files/{file_id}.json` + +```json +{ + "file_id": 449151, + "name": "Unofficial Skyrim Special Edition Patch", + "version": "4.3.0", + "category_name": "ARCHIVED", + "file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0-1701835628.7z", + "uploaded_timestamp": 1701835628, + "uploaded_time": "2023-12-06T04:07:08.000+00:00", + "mod_version": "4.3.0", + "description": "" +} +``` + +This gets us the info for hidden entries. Notice the `category_name` field says `ARCHIVED`. + +!!! warning "It's possible that the downloaded mod has been 'archived'" + + For this reason, we should also support backtracking the Mod IDs, then forward + tracking to find available versions. + +### Rate Limits + +!!! tip "Nexus API has generous API rate limits, but we must handle them nonetheless" + +As of 19th of September 2024, the limits are: + +``` +hourly_limit: 500 +daily_limit: 10000 +``` + +Note that the hourly limit is 500. What this means is that ***we can only check 500 +mods for updates per hour.*** + +If the user has a huge amount of mods, we may have to maintain a 'queue' of mods, +checking the ones we have not checked in the longest time first. + +### One to Many Relationships + +!!! info "The `previous version` -> `next version` relationships are one-to-many." + +Note the previous API response for +`https://api.nexusmods.com/v1/games/skyrimspecialedition/mods/266/files.json` + +```json +"file_updates": [ + { + "old_file_id": 449151, + "new_file_id": 449719, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0-1701835628.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0a-1702019266.7z", + "uploaded_timestamp": 1702019266, + "uploaded_time": "2023-12-08T07:07:46.000+00:00" + }, + { + "old_file_id": 449719, + "new_file_id": 491975, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-0a-1702019266.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-1-1713394824.7z", + "uploaded_timestamp": 1713394827, + "uploaded_time": "2024-04-17T23:00:27.000+00:00" + }, + { + "old_file_id": 491975, + "new_file_id": 522940, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-1-1713394824.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-2-1721450806.7z", + "uploaded_timestamp": 1721450806, + "uploaded_time": "2024-07-20T04:46:46.000+00:00" + }, + { + "old_file_id": 491975, + "new_file_id": 522942, + "old_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-1-1713394824.7z", + "new_file_name": "Unofficial Skyrim Special Edition Patch-266-4-3-2-1721451025.7z", + "uploaded_timestamp": 1721451025, + "uploaded_time": "2024-07-20T04:50:25.000+00:00" + } +] +``` + +You may notice that there are 2 entries with `old_file_id` of `491975`. +This is because the first upload for new version (`522940`) was archived due to being a broken upload. + +To put this in a graph, it would look like this: + +```mermaid +graph LR + A[449151: v4.3.0] --> B[449719: v4.3.0a] + B --> C[491975: v4.3.1] + C --> D[522940: v4.3.2 ARCHIVED] + C --> E[522942: v4.3.2] + style D fill:#793939,stroke:#333 +``` + +It is important that when traversing the graph, we ignore the `ARCHIVED` entry +if a non-archived entry exists. + +A 'cheap' quick way to do this is to check the `files` array. If an ID is not +present in that array, the entry was 'archived', 'deleted' or otherwise. + +!!! note "Some implementations manually query the files by ID in the V1 API" + + This is an optimization over that. + +## Our Approach to Updates + +For [1. Determining Updated Mod Pages](#1-determine-updated-mod-pages), we can improve +upon the existing implementation with [Multi Query Pages](#multi-query-pages) optimization. + +For [2. Querying Mod Files](#2-querying-mod-files), we should use the V1 API that is used by the existing +implementation (`/v1/games/{game_domain_name}/mods/{mod_id}/files.json`); as the V2 API +does not contain the necessary information. + +### Multi-Query Pages + +!!! info "The V2 API allows us to query multiple mod pages at once" + +This is a more efficient way to check if mods were updated. + +The V2 API has an `modsByUid` endpoint that can be used to query multiple mods at once. + +Example query: + +```graphql +query ModsByUid { + modsByUid(uids: ["7318624401981"]) { + nodes { + updatedAt + } + } +} +``` + +Example result: + +```json +{ + "data": { + "modsByUid": { + "nodes": [ + { + "updatedAt": "2024-09-18T23:01:09Z" + } + ] + } + } +} +``` + +Using the `uids` parameter we can query multiple mods at once. +Then we can compare the `updatedAt` timestamps. + +!!! note "A `uid` is a tuple of `modId` and `gameId` at Nexus." + + First 4 bytes are modId, last 4 bytes are gameId; using little endian. + +!!! note "There isn't currently a field for last time the files were updated." + + We can ask backend about this. + +[Mod Organizer 2]: https://github.com/ModOrganizer2/modorganizer +[Vortex]: https://github.com/Nexus-Mods/Vortex +[mo2-update-source]: https://github.com/ModOrganizer2/modorganizer/blob/9c130cbf2fc7225fb2916e46419af50671772aa0/src/modinfo.cpp#L299 +[vortex-update-source]: https://github.com/Nexus-Mods/Vortex/blob/85880b9f54df1cc4c1e29e0008755bda575573b0/src/extensions/nexus_integration/util/checkModsVersion.ts#L125 diff --git a/mkdocs.yml b/mkdocs.yml index 58d4b96244..92ed7b3810 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -127,6 +127,7 @@ nav: - '00: ADR Template': developers/decisions/templates/adr-template.md - Explanations: - '00: Explanation of Library Item Deletion': developers/decisions/explanations/00-library-item-removal.md + - '01: Update Logic': developers/decisions/explanations/01-update-logic.md - Concepts: - '00: Immutable Modlists': developers/concepts/0000-immutable-modlists.md - '01: DAG Sorting': developers/concepts/0001-dag-sorting.md From 796c123246207a7284dd06ec115746505b0182b0 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Mon, 23 Sep 2024 15:30:10 +0100 Subject: [PATCH 02/15] Added: Additional Edge Cases to Update Logic Docs --- .../decisions/explanations/01-update-logic.md | 70 +++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/docs/developers/decisions/explanations/01-update-logic.md b/docs/developers/decisions/explanations/01-update-logic.md index 117b5ed91c..de071996df 100644 --- a/docs/developers/decisions/explanations/01-update-logic.md +++ b/docs/developers/decisions/explanations/01-update-logic.md @@ -307,6 +307,8 @@ For [2. Querying Mod Files](#2-querying-mod-files), we should use the V1 API tha implementation (`/v1/games/{game_domain_name}/mods/{mod_id}/files.json`); as the V2 API does not contain the necessary information. +We represent the data as a *Directed Acyclic Graph (DAG)*. + ### Multi-Query Pages !!! info "The V2 API allows us to query multiple mod pages at once" @@ -354,6 +356,74 @@ Then we can compare the `updatedAt` timestamps. We can ask backend about this. +## Edge Cases + +The following edge cases should be considered. + +### Matching Version Names with one that has been Archived + +!!! note "(ARCHIVED) is category name, they were both submitted with same version." + +```mermaid +graph LR + A[v1.0.0] --> B[v1.0.1] + B --> D["v1.0.2 (ARCHIVED)"] + B --> E[v1.0.2] + style D fill:#793939,stroke:#333 +``` + +!!! question "Should we offer the user an update?" + + We may want to offer the user to download `v4.3.2` if they are on + `v4.3.2 (ARCHIVED)`. + +In this scenario, when the user 'queries' for an update, the archived version should +be ignored; there should be 1 result. + +### Multiple Updates for Single File + +!!! info "There are two 'branches' of the same mod, both are valid, and contain non-archived files." + +```mermaid +graph LR + A[v1.0.0] --> B[v1.0.1] + B --> D[v1.0.2] + D --> E[v1.0.3] + + B --> F[v1.0.2-beta] + F --> G[v1.0.3-beta0] + G --> H[v1.0.3-beta1] +``` + +One of the branches can contain 'beta' versions, while another branch can contain 'stable' versions. + +If the user has `v1.0.0` or `v1.0.1` installed, then both `v1.0.3` and `v1.0.3-beta1` are technically +the 'latest' version of the mod. + +In the case of `SemVer`, this is easy to resolve. +However, Nexus allows arbitrary 'version' values, so the versions could be +`banana` and `orange`. + +API for querying updates should return all the 'latest versions'. + +### Multiple Updates for Single File With Archived Branch End + +!!! info "There are two 'branches' of the same mod, one contains an archived file at end." + +```mermaid +graph LR + A[v1.0.0] --> B[v1.0.1] + B --> D[v1.0.2] + D --> E[v1.0.3] + + B --> F[v1.0.2-beta] + F --> G[v1.0.3-beta0] + G --> H["v1.0.3-beta1 (ARCHIVED)"] + style H fill:#793939,stroke:#333 +``` + +In this case the `Latest Version` is both `v1.0.3` and `v1.0.3-beta0`. + [Mod Organizer 2]: https://github.com/ModOrganizer2/modorganizer [Vortex]: https://github.com/Nexus-Mods/Vortex [mo2-update-source]: https://github.com/ModOrganizer2/modorganizer/blob/9c130cbf2fc7225fb2916e46419af50671772aa0/src/modinfo.cpp#L299 From 0cfb3927d15cc8ff7f98609e81a17d742fad83fa Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Mon, 23 Sep 2024 15:43:27 +0100 Subject: [PATCH 03/15] Added: Extra case of `Archived in the Middle` --- .../decisions/explanations/01-update-logic.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/developers/decisions/explanations/01-update-logic.md b/docs/developers/decisions/explanations/01-update-logic.md index de071996df..194ad29706 100644 --- a/docs/developers/decisions/explanations/01-update-logic.md +++ b/docs/developers/decisions/explanations/01-update-logic.md @@ -372,7 +372,7 @@ graph LR style D fill:#793939,stroke:#333 ``` -!!! question "Should we offer the user an update?" +!!! tip "Should we offer the user an update?" We may want to offer the user to download `v4.3.2` if they are on `v4.3.2 (ARCHIVED)`. @@ -424,6 +424,21 @@ graph LR In this case the `Latest Version` is both `v1.0.3` and `v1.0.3-beta0`. +### Archived in the Middle + +!!! info "There is an 'ARCHIVED' version in the middle of the version list." + + In this case, we should use the non-archived version. + +```mermaid +graph LR + A[v1.0.0] --> B["v1.0.1 (ARCHIVED)"] + B --> C["v1.0.2"] + style B fill:#793939,stroke:#333 +``` + +In this case `v1.0.2` should be the latest version of the mod. + [Mod Organizer 2]: https://github.com/ModOrganizer2/modorganizer [Vortex]: https://github.com/Nexus-Mods/Vortex [mo2-update-source]: https://github.com/ModOrganizer2/modorganizer/blob/9c130cbf2fc7225fb2916e46419af50671772aa0/src/modinfo.cpp#L299 From 6498415c9eca48d50cb907118d23691152080792 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Mon, 23 Sep 2024 15:52:26 +0100 Subject: [PATCH 04/15] Fixed: Indentation for `file_updates` field. --- docs/developers/decisions/explanations/01-update-logic.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/decisions/explanations/01-update-logic.md b/docs/developers/decisions/explanations/01-update-logic.md index 194ad29706..a51af4a9f1 100644 --- a/docs/developers/decisions/explanations/01-update-logic.md +++ b/docs/developers/decisions/explanations/01-update-logic.md @@ -127,7 +127,7 @@ Example payload: "mod_version": "4.3.2", "description": "Meant for use ONLY with SSE v1.6.1130+." }], - "file_updates": [ +"file_updates": [ { "old_file_id": 449151, "new_file_id": 449719, From 600ec0be87c94bf75981db30e5d7e88ce9791d03 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Tue, 24 Sep 2024 04:12:52 +0100 Subject: [PATCH 05/15] Finalized 'Updating Mods' doc with simplified Implementation requested. --- .../decisions/backend/0019-updating-mods.md | 40 +++++ .../00-update-implementation-research.md} | 163 +++++++++++------- mkdocs.yml | 3 +- 3 files changed, 141 insertions(+), 65 deletions(-) create mode 100644 docs/developers/decisions/backend/0019-updating-mods.md rename docs/developers/{decisions/explanations/01-update-logic.md => misc/research/00-update-implementation-research.md} (87%) diff --git a/docs/developers/decisions/backend/0019-updating-mods.md b/docs/developers/decisions/backend/0019-updating-mods.md new file mode 100644 index 0000000000..f145874808 --- /dev/null +++ b/docs/developers/decisions/backend/0019-updating-mods.md @@ -0,0 +1,40 @@ +# Updating Mods + +!!! info "This is a design document detailing the steps taken to update mods." + +A corresponding research document (original design) can be found on a [separate page][research-doc]. + +## General Approach + +!!! tip "First read the [Problem Statement] in the [Research Document]" + +The requested approach (from business) has been to maximize the use of the V2 API, +as opposed to programming against the legacy V1 API. + +To achieve this, we will [NOT use the `file_updates` array from V1 API's Querying Mod Files][querying-mod-files]; +instead choosing to opt to wait until backend decides their future plans with +respect to 'Mods 2.0' project, and how mod updates will be handled in V2 API in the future. + +For now, we will: + +- [1. Determine Updated Mod Pages], to update our local cache. +- [2. Multi Query Pages], for update mod pages with a 'cache miss'. + +## Displaying Mod Updates + +!!! info "We display all files on a given mod page that are more recent (file upload time) than the user's file." + +Although uncommon this may include: + +- Files for other mods on same mod page. +- Older files (if uploaded out of order). + +We will for now rely on *users' common sense* to identify whether a file is an +update to a previous file or not. Until site decides on future plans. + +[Problem Statement]: ../../misc/research/00-update-implementation-research.md#problem-statement +[1. Determine Updated Mod Pages]: ../../misc/research/00-update-implementation-research.md#1-determine-updated-mod-pages +[2. Multi Query Pages]: ../../misc/research/00-update-implementation-research.md#multi-query-pages +[querying-mod-files]: ../../misc/research/00-update-implementation-research.md#2-querying-mod-files +[Research Document]: ../../misc/research/00-update-implementation-research.md +[research-doc]: ../../misc/research/00-update-implementation-research.md diff --git a/docs/developers/decisions/explanations/01-update-logic.md b/docs/developers/misc/research/00-update-implementation-research.md similarity index 87% rename from docs/developers/decisions/explanations/01-update-logic.md rename to docs/developers/misc/research/00-update-implementation-research.md index a51af4a9f1..fac290d108 100644 --- a/docs/developers/decisions/explanations/01-update-logic.md +++ b/docs/developers/misc/research/00-update-implementation-research.md @@ -1,3 +1,10 @@ +!!! note "This document is the *former* design document on how to handle updates via Nexus Mods." + + By request, the approach used was changed from the steps in [Our Approach to Updates](#our-approach-to-updates) + to using a [much simpler design][current-adr] during implementation. + + This document remains as a research document. + !!! info "Supporting ***generic*** mod updates on the Nexus today is a tricky subject." ## Problem Statement @@ -45,6 +52,15 @@ However, this system has limitations: - The `previous version` -> `next version` is a one-to-many relationship. - A user is allowed to set the same `previous version` on many files. +### No Plans for V2 API + +!!! info "Plans for V2 API in regard to 'mod updates' are unclear." + +More specifically, there are no plans yet in regard to what will happen with the +`file_updates` field (see below). The V2 API does not contain this field; it is unclear +whether V2 API will support this field, or whether the whole system of marking mod updates +will be replaced with a new on-site implementation. + ## Existing Implementations !!! info "Some existing implementations of ***generic*** mod updates over Nexus exist." @@ -85,14 +101,32 @@ Example response (truncated): ] ``` -We match the response against our locally installed mods. +We match the response against our locally installed mods; +performing the following actions: + +#### Updating our Cached Data + +If any of our mods: -If any of our mods is in the response, and the `latest_file_update` timestamp is older -or equal to our locally saved cached response; we don't query the mod. +- Are in the response and have a `latest_file_update` timestamp than our local cached one. +- Have not been checked in over 1 month (max API limit) -Otherwise, if the mod hasn't been checked in over 1 month (max API limit), or its -`mod_id` is not present in the response, we will have to -[query the files for an update](#2-querying-mod-files) individually +We [query the mod files for an update](#2-querying-mod-files). +We then update the locally cached data and set timestamp to current time. + +#### Updating Cached Timestamp + +For all other mods; including: + +- Those sooner than 1 month old but with older or equal `latest_file_update` timestamp. +- Those sooner than 1 month old but not in the response. + +We simply update the 'last checked timestamp' to the current time. + +!!! tip "This means that on repeated calls to the `updated.json` API, we will incur a 'local cache hit'." + + Logically, a mod whose update you checked for sooner than 1 month ago, that has not been + updated in 1 month could not have changed. ### 2. Querying Mod Files @@ -298,64 +332,6 @@ present in that array, the entry was 'archived', 'deleted' or otherwise. This is an optimization over that. -## Our Approach to Updates - -For [1. Determining Updated Mod Pages](#1-determine-updated-mod-pages), we can improve -upon the existing implementation with [Multi Query Pages](#multi-query-pages) optimization. - -For [2. Querying Mod Files](#2-querying-mod-files), we should use the V1 API that is used by the existing -implementation (`/v1/games/{game_domain_name}/mods/{mod_id}/files.json`); as the V2 API -does not contain the necessary information. - -We represent the data as a *Directed Acyclic Graph (DAG)*. - -### Multi-Query Pages - -!!! info "The V2 API allows us to query multiple mod pages at once" - -This is a more efficient way to check if mods were updated. - -The V2 API has an `modsByUid` endpoint that can be used to query multiple mods at once. - -Example query: - -```graphql -query ModsByUid { - modsByUid(uids: ["7318624401981"]) { - nodes { - updatedAt - } - } -} -``` - -Example result: - -```json -{ - "data": { - "modsByUid": { - "nodes": [ - { - "updatedAt": "2024-09-18T23:01:09Z" - } - ] - } - } -} -``` - -Using the `uids` parameter we can query multiple mods at once. -Then we can compare the `updatedAt` timestamps. - -!!! note "A `uid` is a tuple of `modId` and `gameId` at Nexus." - - First 4 bytes are modId, last 4 bytes are gameId; using little endian. - -!!! note "There isn't currently a field for last time the files were updated." - - We can ask backend about this. - ## Edge Cases The following edge cases should be considered. @@ -439,7 +415,66 @@ graph LR In this case `v1.0.2` should be the latest version of the mod. +## Our Approach to Updates + +!!! note "This was replaced by a [simpler design][current-adr] by request" + + This is the original design. + +For [1. Determining Updated Mod Pages](#1-determine-updated-mod-pages), we can improve +upon the existing implementation with [Multi Query Pages](#multi-query-pages) optimization. + +For [2. Querying Mod Files](#2-querying-mod-files), we should use the V1 API that is used by the existing +implementation (`/v1/games/{game_domain_name}/mods/{mod_id}/files.json`); as the V2 API +does not contain the necessary information. + +We represent the data as a *Directed Acyclic Graph (DAG)*. + +### Multi-Query Pages + +!!! info "The V2 API allows us to query multiple mod pages at once" + +This is a more efficient way to check if mods were updated. + +The V2 API has an `modsByUid` endpoint that can be used to query multiple mods at once. + +Example query: + +```graphql +query ModsByUid { + modsByUid(uids: ["7318624401981"]) { + nodes { + updatedAt + } + } +} +``` + +Example result: + +```json +{ + "data": { + "modsByUid": { + "nodes": [ + { + "updatedAt": "2024-09-18T23:01:09Z" + } + ] + } + } +} +``` + +Using the `uids` parameter we can query multiple mods at once. +Then we can compare the `updatedAt` timestamps. + +!!! note "A `uid` is a tuple of `modId` and `gameId` at Nexus." + + First 4 bytes are modId, last 4 bytes are gameId; using little endian. + [Mod Organizer 2]: https://github.com/ModOrganizer2/modorganizer [Vortex]: https://github.com/Nexus-Mods/Vortex [mo2-update-source]: https://github.com/ModOrganizer2/modorganizer/blob/9c130cbf2fc7225fb2916e46419af50671772aa0/src/modinfo.cpp#L299 [vortex-update-source]: https://github.com/Nexus-Mods/Vortex/blob/85880b9f54df1cc4c1e29e0008755bda575573b0/src/extensions/nexus_integration/util/checkModsVersion.ts#L125 +[current-adr]: ../../decisions/backend/0019-updating-mods.md diff --git a/mkdocs.yml b/mkdocs.yml index 92ed7b3810..f2daefd9fb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -127,7 +127,6 @@ nav: - '00: ADR Template': developers/decisions/templates/adr-template.md - Explanations: - '00: Explanation of Library Item Deletion': developers/decisions/explanations/00-library-item-removal.md - - '01: Update Logic': developers/decisions/explanations/01-update-logic.md - Concepts: - '00: Immutable Modlists': developers/concepts/0000-immutable-modlists.md - '01: DAG Sorting': developers/concepts/0001-dag-sorting.md @@ -140,6 +139,8 @@ nav: - About Fomod: developers/misc/AboutFomod.md - Comparison of File Management Systems: developers/misc/ComparisonOfFileManagementSystems.md - Drawbacks of Nexus App Approach: developers/misc/DrawbacksOfNexusApproach.md + - Research: + - '00: Update Implementation': developers/misc/research/00-update-implementation-research.md - Contributing to Docs: Nexus/Pages/contributing.md validation: From 2155750f7c54f9682cccae908a07c554fc3dacf9 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Tue, 24 Sep 2024 04:24:43 +0100 Subject: [PATCH 06/15] Fixed: Minor Notes from older Research Doc --- .../00-update-implementation-research.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/developers/misc/research/00-update-implementation-research.md b/docs/developers/misc/research/00-update-implementation-research.md index fac290d108..0b05c1cdfe 100644 --- a/docs/developers/misc/research/00-update-implementation-research.md +++ b/docs/developers/misc/research/00-update-implementation-research.md @@ -1,11 +1,6 @@ -!!! note "This document is the *former* design document on how to handle updates via Nexus Mods." +!!! info "This is a research document, listing the various caveats, edge cases and existing solutions for generic updates on *Nexus Mods*" - By request, the approach used was changed from the steps in [Our Approach to Updates](#our-approach-to-updates) - to using a [much simpler design][current-adr] during implementation. - - This document remains as a research document. - -!!! info "Supporting ***generic*** mod updates on the Nexus today is a tricky subject." + Supporting ***generic*** mod updates on the Nexus today is a tricky subject. ## Problem Statement @@ -415,11 +410,14 @@ graph LR In this case `v1.0.2` should be the latest version of the mod. -## Our Approach to Updates +## Our Improved Approach to Updates [Legacy] -!!! note "This was replaced by a [simpler design][current-adr] by request" +!!! warning "This was replaced by a [simpler design][current-adr] by request" - This is the original design. + This is the original design; which involved the use of the V1 API. + Because we are moving away from V1 API, project lead requested to not + proceed with this design. But [Multi Query Pages](#multi-query-pages) optimization + will be reused. For [1. Determining Updated Mod Pages](#1-determine-updated-mod-pages), we can improve upon the existing implementation with [Multi Query Pages](#multi-query-pages) optimization. From b54efacf315bf286f0edac07d1ac4d99ce00c9a0 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Tue, 24 Sep 2024 18:07:48 +0100 Subject: [PATCH 07/15] Fixed: Added Missing 'Updating Mods' mkdocs sidebar item. --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index b1eca3edd4..cf57357d75 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -111,6 +111,7 @@ nav: - '10: Disk State Storage': developers/decisions/backend/0016-disk-state-storage.md - '11: Garbage Collector Design': developers/decisions/backend/0017-garbage-collector-design.md - '12: Wine and Proton Detection Design': developers/decisions/backend/0018-running-tools-on-linux.md + - '13: Updating Mods': developers/decisions/backend/0019-updating-mods.md - Frontend: - '01: UI Panel Layout': developers/decisions/frontend/0001-UI-Panel-Layout.md - '02: Workspaces': developers/decisions/frontend/0002-Workspaces.md From 71a131b273ca1600c23f6abc2c66f1df4d392cf0 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Wed, 25 Sep 2024 16:53:37 +0100 Subject: [PATCH 08/15] [Working WIP] Added: Initial Implementation of Generic Page Caching System used by Mod Pages --- NexusMods.App.sln | 14 ++ .../MultiFeedCacheUpdater.cs | 110 +++++++++ .../NexusMods.Networking.ModUpdates.csproj | 16 ++ .../PerFeedCacheUpdater.cs | 155 +++++++++++++ .../PerFeedCacheUpdaterResult.cs | 44 ++++ .../Private/CacheUpdaterAction.cs | 24 ++ .../Structures/GameId.cs | 8 + .../Structures/ModId.cs | 10 + .../Structures/Uid.cs | 33 +++ .../Traits/ICanGetLastUpdatedTimestamp.cs | 12 + .../Traits/ICanGetUid.cs | 17 ++ .../GlobalUsings.cs | 1 + .../Helpers/TestItem.cs | 25 ++ .../MultiFeedCacheUpdaterTests.cs | 214 ++++++++++++++++++ ...xusMods.Networking.ModUpdates.Tests.csproj | 9 + .../PerFeedCacheUpdaterTests.cs | 180 +++++++++++++++ .../Startup.cs | 11 + .../Tests.cs | 7 + 18 files changed, 890 insertions(+) create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Private/CacheUpdaterAction.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetLastUpdatedTimestamp.cs create mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/GlobalUsings.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/NexusMods.Networking.ModUpdates.Tests.csproj create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/Startup.cs create mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs diff --git a/NexusMods.App.sln b/NexusMods.App.sln index e5cf3acc10..1ac2c8326a 100644 --- a/NexusMods.App.sln +++ b/NexusMods.App.sln @@ -258,6 +258,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "NexusMods.Abstractions.Medi EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "NexusMods.Games.Larian", "src\Games\NexusMods.Games.Larian\NexusMods.Games.Larian.csproj", "{2A35EBB5-1CA6-4F5D-8CE8-352146C82C28}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "NexusMods.Networking.ModUpdates", "src\Networking\NexusMods.Networking.ModUpdates\NexusMods.Networking.ModUpdates.csproj", "{8B246C04-F372-47F6-9397-F658915429A8}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "NexusMods.Networking.ModUpdates.Tests", "tests\Networking\NexusMods.Networking.ModUpdates.Tests\NexusMods.Networking.ModUpdates.Tests.csproj", "{CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -668,6 +672,14 @@ Global {2A35EBB5-1CA6-4F5D-8CE8-352146C82C28}.Debug|Any CPU.Build.0 = Debug|Any CPU {2A35EBB5-1CA6-4F5D-8CE8-352146C82C28}.Release|Any CPU.ActiveCfg = Release|Any CPU {2A35EBB5-1CA6-4F5D-8CE8-352146C82C28}.Release|Any CPU.Build.0 = Release|Any CPU + {8B246C04-F372-47F6-9397-F658915429A8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8B246C04-F372-47F6-9397-F658915429A8}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8B246C04-F372-47F6-9397-F658915429A8}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8B246C04-F372-47F6-9397-F658915429A8}.Release|Any CPU.Build.0 = Release|Any CPU + {CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF}.Debug|Any CPU.Build.0 = Debug|Any CPU + {CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF}.Release|Any CPU.ActiveCfg = Release|Any CPU + {CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -787,6 +799,8 @@ Global {8C817874-7A88-450E-B216-851A1B03684C} = {52AF9D62-7D5B-4AD0-BA12-86F2AA67428B} {5CB6D02C-07D0-4C0D-BF5C-4E2E958A0612} = {0CB73565-1207-4A56-A79F-6A8E9BBD795C} {2A35EBB5-1CA6-4F5D-8CE8-352146C82C28} = {70D38D24-79AE-4600-8E83-17F3C11BA81F} + {8B246C04-F372-47F6-9397-F658915429A8} = {D7E9D8F5-8AC8-4ADA-B219-C549084AD84C} + {CDA2C52B-A9A7-446B-9D2F-D7B75C1905EF} = {897C4198-884F-448A-B0B0-C2A6D971EAE0} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {9F9F8352-34DD-42C0-8564-EE9AF34A3501} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs new file mode 100644 index 0000000000..29c1bedf4e --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs @@ -0,0 +1,110 @@ +using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Networking.ModUpdates.Traits; +namespace NexusMods.Networking.ModUpdates; + +/// +/// This is a helper struct which combines multiple +/// instances into a single interface. This allows for a cache update operation across +/// multiple mod feeds (games). +/// +/// For usage instructions, see ; the API and concepts +/// here are similar, except for the difference that this class' public API allows +/// you to use mods and API responses which are sourced from multiple feeds (games), +/// as opposed to a single feed. +/// +public class MultiFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +{ + private readonly Dictionary> _updaters; + + /// + /// Creates a cache updater from a given list of items for which we want to + /// check for updates. + /// + /// The items to check for updates. + /// + /// Maximum age before an item has to be re-checked for updates. + /// The max for Nexus Mods API is 1 month. + /// + /// + /// In order to ensure accuracy, the age field should include the lifespan of + /// the as well as the cache + /// time on the server's end. That should prevent any technical possible + /// eventual consistency errors due to race conditions. Although + /// + public MultiFeedCacheUpdater(TUpdateableItem[] items, TimeSpan expiry) + { + _updaters = new Dictionary>(); + + // First group the items by their GameId + // We will use a list of groups, the assumption being that the number + // of feeds is low (usually less than 5) + var groupedList = new List<(GameId, List)>(); + foreach (var item in items) + { + var gameId = item.GetUniqueId().GameId; + + // Get or Update List for this GameId. + var found = false; + foreach (var (key, value) in groupedList) + { + if (key != gameId) + continue; + + value.Add(item); + found = true; + break; + } + + if (!found) + groupedList.Add((gameId, [item])); + } + + // Create a PerFeedCacheUpdater for each group + foreach (var (key, value) in groupedList) + _updaters[key] = new PerFeedCacheUpdater(value.ToArray(), expiry); + } + + /// + /// Updates the internal state of the + /// provided the results of the 'most recently updated mods for game' endpoint. + /// + /// + /// The items returned by the 'most recently updated mods' endpoint. This can + /// include items corresponding to multiple feeds (games); the feed source + /// is automatically detected. + /// + /// Wrap elements in a struct that implements + /// and if necessary. + /// + public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + { + foreach (var item in items) + { + // Determine feed + var feed = item.GetUniqueId().GameId; + + // The result may contain items from feeds which we are not tracking. + // For instance, results for other games. This is not an error, we + // just need to filter the items out. + if (!_updaters.TryGetValue(feed, out var updater)) + continue; + + updater.UpdateSingleItem(item); + } + } + + /// + /// Determines the actions needed to taken on the items in the ; + /// returning the items whose actions have to be taken grouped by the action that needs performed. + /// + /// The results of multiple feeds are flattened here; everything is returned as a single result. + /// + public PerFeedCacheUpdaterResult BuildFlattened() + { + var result = new PerFeedCacheUpdaterResult(); + foreach (var updater in _updaters) + result.AddFrom(updater.Value.Build()); + + return result; + } +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj b/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj new file mode 100644 index 0000000000..91abef7d7e --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + diff --git a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs new file mode 100644 index 0000000000..3189f1ee7c --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs @@ -0,0 +1,155 @@ +using System.Diagnostics; +using NexusMods.Networking.ModUpdates.Private; +using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Networking.ModUpdates.Traits; +namespace NexusMods.Networking.ModUpdates; + +/// +/// This is a helper struct which helps us make use of the 'most recently updated mods for game' +/// API endpoint to use as a cache. (Where 'game' is a 'feed') +/// For full info on internals, see the NexusMods.App project documentation. +/// +/// This API consists of the following: +/// +/// 1. Input [Constructor]: A set of items with a 'last update time' (see ) +/// and a 'unique id' (see ) that are relevant to the current 'feed' (game). +/// +/// 2. Update [Method]: Submit results from API endpoint returning 'most recently updated mods for game'. +/// This updates the internal state of the . +/// +/// 3. Output [Info]: The outputs items with 2 categories: +/// - Up-to-date mods. These should have their timestamp updated. +/// - Out of date mods. These require re-querying the data from external source. +/// +/// +/// Within the Nexus Mods App: +/// +/// - 'Input' is our set of locally cached mod pages. +/// - 'Update' is our results of `updated.json` for a given game domain. +/// - 'Output' are the pages we need to update. +/// +/// The 'Feed' in the context of the Nexus App is the individual game's 'updated.json' endpoint; +/// i.e. a 'Game Mod Feed' +/// +public class PerFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +{ + private readonly TUpdateableItem[] _items; + private readonly Dictionary _itemToIndex; + private readonly CacheUpdaterAction[] _actions; + + /// + /// Creates a (per-feed) cache updater from a given list of items for which + /// we want to check for updates. + /// + /// The items to check for updates. + /// + /// Maximum age before an item has to be re-checked for updates. + /// The max for Nexus Mods API is 1 month. + /// + /// + /// In order to ensure accuracy, the age field should include the lifespan of + /// the as well as the cache + /// time on the server's end. That should prevent any technical possible + /// eventual consistency errors due to race conditions. Although + /// + public PerFeedCacheUpdater(TUpdateableItem[] items, TimeSpan expiry) + { + _items = items; + DebugVerifyAllItemsAreFromSameGame(); + + _actions = new CacheUpdaterAction[items.Length]; + _itemToIndex = new Dictionary(items.Length); + for (var x = 0; x < _items.Length; x++) + _itemToIndex[_items[x].GetUniqueId().ModId] = x; + + // Set the action to refresh cache for any mods which exceed max age. + var utcNow = DateTime.UtcNow; + var minCachedDate = utcNow - expiry; + for (var x = 0; x < _items.Length; x++) + { + if (_items[x].GetLastUpdatedDate() < minCachedDate) + _actions[x] = CacheUpdaterAction.NeedsUpdate; + } + } + + /// + /// Updates the internal state of the + /// provided the results of the 'most recently updated mods for game' endpoint. + /// + /// + /// The items returned by the 'most recently updated mods for game' endpoint. + /// Wrap elements in a struct that implements + /// and if necessary. + /// + public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + { + foreach (var item in items) + UpdateSingleItem(item); + } + + internal void UpdateSingleItem(T item) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + { + // Try to get index of the item. + // Not all the items from the update feed are locally stored, thus we need to + // make sure we actually have this item. + if (!_itemToIndex.TryGetValue(item.GetUniqueId().ModId, out var index)) + return; + + var existingItem = _items[index]; + + // If the file timestamp is newer than our cached copy, the item needs updating. + if (item.GetLastUpdatedDate() > existingItem.GetLastUpdatedDate()) + _actions[index] = CacheUpdaterAction.NeedsUpdate; + else + _actions[index] = CacheUpdaterAction.UpdateLastCheckedTimestamp; + } + + /// + /// Determines the actions needed to taken on the items in the ; + /// returning the items whose actions have to be taken grouped by the action that needs performed. + /// + public PerFeedCacheUpdaterResult Build() + { + // We now have files in 3 categories: + // - Up-to-date mods. (Determined in `Update` method) a.k.a. CacheUpdaterAction.UpdateLastCheckedTimestamp + // - Out of date mods. (Determined in `Update` method and constructor) a.k.a. CacheUpdaterAction.NeedsUpdate + // - Undetermined Mods. (Mods with CacheUpdaterAction.Default) + // - This is the case for mods that are not in the `updated.json` payload + // which for some reason are in our cache and are not out of date (not greater than expiry). + // - Alternatively, they are in our cache but not in the `updated.json` payload. + // - This can be the case if the expiry field is changed between calls, or the maxAge of the + // request whose results we feed to `Update` is inconsistent due to programmer error. + // - We return these in a separate category, but the consumer should treat them as 'Out of Date' + var result = new PerFeedCacheUpdaterResult(); + for (var x = 0; x < _actions.Length; x++) + { + switch (_actions[x]) + { + case CacheUpdaterAction.Default: + result.UndeterminedItems.Add(_items[x]); + break; + case CacheUpdaterAction.NeedsUpdate: + result.OutOfDateItems.Add(_items[x]); + break; + case CacheUpdaterAction.UpdateLastCheckedTimestamp: + result.UpToDateItems.Add(_items[x]); + break; + default: + throw new ArgumentOutOfRangeException(); + } + } + + return result; + } + + [Conditional("DEBUG")] + private void DebugVerifyAllItemsAreFromSameGame() + { + if (_items.Length == 0) return; + + var firstGameId = _items[0].GetUniqueId().GameId; + var allSame = _items.All(x => x.GetUniqueId().GameId == firstGameId); + if (!allSame) + throw new ArgumentException("All items must have the same game id", nameof(_items)); + } +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs new file mode 100644 index 0000000000..1f14548a1c --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs @@ -0,0 +1,44 @@ +using NexusMods.Networking.ModUpdates.Traits; +namespace NexusMods.Networking.ModUpdates; + +/// +/// Stores the result of updating the 'mod page' cache for a given feed or sets +/// of feeds. +/// +/// Wrapper for item supported by the cache updater. +public class PerFeedCacheUpdaterResult where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +{ + /// + /// This is a list of items that is 'out of date'. + /// For these items, we need to fetch updated info from the Nexus Servers and update the timestamp. + /// + public List OutOfDateItems { get; init; } = new(); + + /// + /// These are the items that are 'up-to-date'. + /// Just update the timestamp on these items and you're good. + /// + public List UpToDateItems { get; init; } = new(); + + /// + /// These are the items that are 'undetermined'. + /// + /// These should be treated the same as the items in ; + /// however having items here is indicative of a possible programmer error. + /// (Due to inconsistent expiry parameter between Nexus API call and cache updater). + /// + /// Consider logging these items. + /// + public List UndeterminedItems { get; init; } = new(); + + /// + /// Adds items from another + /// into the current one. + /// + public void AddFrom(PerFeedCacheUpdaterResult other) + { + OutOfDateItems.AddRange(other.OutOfDateItems); + UpToDateItems.AddRange(other.UpToDateItems); + UndeterminedItems.AddRange(other.UndeterminedItems); + } +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Private/CacheUpdaterAction.cs b/src/Networking/NexusMods.Networking.ModUpdates/Private/CacheUpdaterAction.cs new file mode 100644 index 0000000000..7026bff0fe --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Private/CacheUpdaterAction.cs @@ -0,0 +1,24 @@ +namespace NexusMods.Networking.ModUpdates.Private; + +/// +/// Defines the actions that need to be taken on all elements submitted to the . +/// +internal enum CacheUpdaterAction : byte +{ + /// + /// This defaults to . + /// Either the entry is missing from the remote, or a programmer error has occurred. + /// + Default = 0, + + /// + /// The item needs to be updated in the local cache. + /// + NeedsUpdate = 1, + + /// + /// The item's 'last checked timestamp' needs to be updated. + /// (The item is already up to date) + /// + UpdateLastCheckedTimestamp = 2, +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs new file mode 100644 index 0000000000..8dfbbd207e --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs @@ -0,0 +1,8 @@ +using TransparentValueObjects; +namespace NexusMods.Networking.ModUpdates.Structures; + +/// +/// Represents the unique ID of an individual game in the Nexus Backend. +/// +[ValueObject] // Do not modify. Unsafe code relies on this. +public readonly partial struct GameId { } diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs new file mode 100644 index 0000000000..4b2a82b295 --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs @@ -0,0 +1,10 @@ +using TransparentValueObjects; +namespace NexusMods.Networking.ModUpdates.Structures; + +/// +/// Represents the unique ID of an individual mod in the Nexus Backend. +/// This is specific to the it belongs to; +/// forming a composite key. +/// +[ValueObject] // Do not modify. Unsafe code relies on this. +public readonly partial struct ModId { } diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs new file mode 100644 index 0000000000..b72c404b44 --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs @@ -0,0 +1,33 @@ +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +namespace NexusMods.Networking.ModUpdates.Structures; + +/// +/// This represents a unique ID of an individual mod page as stored on Nexus Mods. +/// +/// +/// Not tested on Big Endian architectures. +/// +[StructLayout(LayoutKind.Sequential, Pack = 1)] +public struct Uid +{ + /// + /// Unique identifier for the mod, within the specific . + /// + public ModId ModId; + + /// + /// Unique identifier for the game. + /// + public GameId GameId; + + /// + /// Reinterprets the current as a single . + /// + public ulong AsUlong => Unsafe.As(ref this); + + /// + /// Reinterprets a given into a . + /// + public static Uid FromUlong(ulong value) => Unsafe.As(ref value); +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetLastUpdatedTimestamp.cs b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetLastUpdatedTimestamp.cs new file mode 100644 index 0000000000..37fc2c1737 --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetLastUpdatedTimestamp.cs @@ -0,0 +1,12 @@ +namespace NexusMods.Networking.ModUpdates.Traits; + +/// +/// This interface marks an item which has the time it was last updated. +/// +public interface ICanGetLastUpdatedTimestamp +{ + /// + /// Retrieves the time the item was last updated. + /// + public DateTime GetLastUpdatedDate(); +} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs new file mode 100644 index 0000000000..b7e341ebe3 --- /dev/null +++ b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs @@ -0,0 +1,17 @@ +using NexusMods.Networking.ModUpdates.Structures; +namespace NexusMods.Networking.ModUpdates.Traits; + +/// +/// A trait representing an item which has a unique ID. +/// In this case, a 'unique ID' refers to a NexusMods Mod Page ID. +/// +/// This ID must be truly unique and belong to a specific item of its type. +/// +public interface ICanGetUid +{ + /// + /// Returns a unique identifier for the given item, based on the ID format + /// used in the NexusMods V2 API. + /// + public Uid GetUniqueId(); +} diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/GlobalUsings.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/GlobalUsings.cs new file mode 100644 index 0000000000..c802f4480b --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/GlobalUsings.cs @@ -0,0 +1 @@ +global using Xunit; diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs new file mode 100644 index 0000000000..1536926193 --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs @@ -0,0 +1,25 @@ +using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Networking.ModUpdates.Traits; +namespace NexusMods.Networking.ModUpdates.Tests; + +// Helper class to simulate updateable items +public class TestItem : ICanGetLastUpdatedTimestamp, ICanGetUid +{ + public DateTime LastUpdated { get; set; } + public Uid Uid { get; set; } + + public DateTime GetLastUpdatedDate() => LastUpdated; + public Uid GetUniqueId() => Uid; + + // Helper method to create a test item + public static TestItem Create(uint gameId, uint modId, DateTime lastUpdated) + { + return new TestItem + { + Uid = new Uid { GameId = GameId.From(gameId), ModId = ModId.From(modId) }, + LastUpdated = lastUpdated, + }; + } +} + + diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs new file mode 100644 index 0000000000..3f578e0d6e --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs @@ -0,0 +1,214 @@ +using FluentAssertions; +using static NexusMods.Networking.ModUpdates.Tests.TestItem; + +namespace NexusMods.Networking.ModUpdates.Tests; + +public class MultiFeedCacheUpdaterTests +{ + [Fact] + public void Constructor_WithEmptyItems_ShouldNotThrow() + { + // Arrange & Act + Action act = () => new MultiFeedCacheUpdater([], TimeSpan.FromDays(30)); + + // Assert + act.Should().NotThrow(); + } + + [Fact] + public void Constructor_ShouldSetOldItemsToNeedUpdate() + { + // Items which have a 'last checked date' older than expiry + // should be marked as 'Out of Date' across multiple feeds. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-40)), + Create(1, 2, now.AddDays(-20)), + Create(2, 1, now.AddDays(-35)), + Create(2, 2, now.AddDays(-25)), + }; + + // Act + var updater = new MultiFeedCacheUpdater(items, TimeSpan.FromDays(30)); + var result = updater.BuildFlattened(); + + // Assert + // items [0] and [2] should be out of date + result.OutOfDateItems.Should().HaveCount(2); + result.OutOfDateItems.Should().Contain(items[0]); + result.OutOfDateItems.Should().Contain(items[2]); + } + + [Fact] + public void Update_ShouldMarkMissingItemsAsUndetermined() + { + // Items that are missing from the 'updated' payload + // may be 'inaccessible', such as deleted or taken down + // due to a DMCA. We should notice the mods which fall + // under this category across multiple feeds. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(2, 1, now.AddDays(-15)), + Create(2, 2, now.AddDays(-8)), + }; + + var updater = new MultiFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + var updateItems = new[] + { + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + Create(2, 1, now.AddDays(-13)), // Newer, needs update + }; + + // Act + updater.Update(updateItems); + var result = updater.BuildFlattened(); + + // Assert + // item[3] is not in 'update' feed/result, but is within + // the expiry date. This means the mod page may have been archived + // or taken down due to a DMCA. + result.UndeterminedItems.Should().ContainSingle(); + result.UndeterminedItems.Should().Contain(items[3]); + } + + [Fact] + public void Update_ShouldMarkItemsAsUpToDateOrNeedingUpdateAcrossMultipleFeeds() + { + // The MultiFeedCacheUpdater correctly compares the 'lastUpdated' field + // of the update entry with the 'lastUpdated' field of the cached item + // across multiple feeds. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(2, 1, now.AddDays(-12)), + Create(2, 2, now.AddDays(-7)), + }; + + var updater = new MultiFeedCacheUpdater(items, TimeSpan.FromDays(30)); + var updateItems = new[] + { + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + Create(2, 1, now.AddDays(-10)), // Newer, needs update + Create(2, 2, now.AddDays(-9)), // Older, up-to-date + }; + + // Act + updater.Update(updateItems); + var result = updater.BuildFlattened(); + + // Assert + // items[0] and items[2] are out of date, items[1] and items[3] are up-to-date + result.OutOfDateItems.Should().HaveCount(2); + result.OutOfDateItems.Should().Contain(items[0]); + result.OutOfDateItems.Should().Contain(items[2]); + + result.UpToDateItems.Should().HaveCount(2); + result.UpToDateItems.Should().Contain(items[1]); + result.UpToDateItems.Should().Contain(items[3]); + } + + [Fact] + public void Update_HavingExtraItemsInUpdatePayloadHasNoSideEffects() + { + // Copy of Update_ShouldMarkItemsAsUpToDateOrNeedingUpdateAcrossMultipleFeeds, + // but with extra item(s) in update payload. These items are ones we don't have cached; + // therefore they should be ignored without side effects. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(2, 1, now.AddDays(-12)), + Create(2, 2, now.AddDays(-7)), + }; + + var updater = new MultiFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + var updateItems = new[] + { + Create(1, 5, now), // New item, should be ignored + Create(2, 6, now), // New item, should be ignored + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + Create(2, 1, now.AddDays(-10)), // Newer, needs update + Create(2, 2, now.AddDays(-9)), // Older, up-to-date + Create(1, 7, now), // New item, should be ignored + Create(2, 8, now), // New item, should be ignored + }; + + // Act + updater.Update(updateItems); + var result = updater.BuildFlattened(); + + // Assert + // items[0] and items[2] are out of date, items[1] and items[3] are up-to-date + result.OutOfDateItems.Should().HaveCount(2); + result.OutOfDateItems.Should().Contain(items[0]); + result.OutOfDateItems.Should().Contain(items[2]); + + result.UpToDateItems.Should().HaveCount(2); + result.UpToDateItems.Should().Contain(items[1]); + result.UpToDateItems.Should().Contain(items[3]); + } + + [Fact] + public void Update_ShouldHandleUpdatesFromDifferentFeeds() + { + // Ensure that the MultiFeedCacheUpdater correctly handles updates + // from different feeds separately. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(2, 1, now.AddDays(-12)), + Create(2, 2, now.AddDays(-7)), + Create(3, 1, now.AddDays(-15)), + }; + + var updater = new MultiFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + var updateItems = new[] + { + Create(1, 1, now.AddDays(-8)), // Feed 1: Newer, needs update + Create(1, 2, now.AddDays(-7)), // Feed 1: Older, up-to-date + Create(2, 1, now.AddDays(-10)), // Feed 2: Newer, needs update + Create(3, 1, now.AddDays(-16)), // Feed 3: Older, up-to-date + }; + + // Act + updater.Update(updateItems); + var result = updater.BuildFlattened(); + + // Assert + result.OutOfDateItems.Should().HaveCount(2); + result.OutOfDateItems.Should().Contain(items[0]); + result.OutOfDateItems.Should().Contain(items[2]); + + result.UpToDateItems.Should().HaveCount(2); + result.UpToDateItems.Should().Contain(items[1]); + result.UpToDateItems.Should().Contain(items[4]); + + result.UndeterminedItems.Should().ContainSingle(); + result.UndeterminedItems.Should().Contain(items[3]); + } +} diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/NexusMods.Networking.ModUpdates.Tests.csproj b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/NexusMods.Networking.ModUpdates.Tests.csproj new file mode 100644 index 0000000000..ca2fea394b --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/NexusMods.Networking.ModUpdates.Tests.csproj @@ -0,0 +1,9 @@ + + + NexusMods.Networking.ModUpdates.Tests + + + + + + diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs new file mode 100644 index 0000000000..e6acd44711 --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs @@ -0,0 +1,180 @@ +using System.Diagnostics; +using FluentAssertions; +using static NexusMods.Networking.ModUpdates.Tests.TestItem; + +namespace NexusMods.Networking.ModUpdates.Tests; + +public class PerFeedCacheUpdaterTests +{ + [Fact] + public void Constructor_WithEmptyItems_ShouldNotThrow() + { + // Arrange & Act + Action act = () => new PerFeedCacheUpdater([], TimeSpan.FromDays(30)); + + // Assert + act.Should().NotThrow(); + } + + [Fact] + [Conditional("DEBUG")] + public void Constructor_WithItemsFromDifferentGames_ShouldThrowArgumentException_InDebug() + { + // Arrange + var items = new[] + { + Create(1, 1, DateTime.UtcNow), + Create(2, 1, DateTime.UtcNow), + }; + + // Act + // ReSharper disable once HeapView.ObjectAllocation.Evident + // ReSharper disable once ObjectCreationAsStatement + Action act = () => new PerFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + // Assert + act.Should().Throw(); + } + + [Fact] + public void Constructor_ShouldSetOldItemsToNeedUpdate() + { + // Items which have a 'last checked date' older than expiry + // should be marked as 'Out of Date'. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-40)), + Create(1, 2, now.AddDays(-20)), + Create(1, 3, now.AddDays(-35)), + }; + + // Act + var updater = new PerFeedCacheUpdater(items, TimeSpan.FromDays(30)); + var result = updater.Build(); + + // Assert + // input items [0] and [2] should be out of date + result.OutOfDateItems.Should().HaveCount(2); + result.OutOfDateItems.Should().Contain(items[0]); + result.OutOfDateItems.Should().Contain(items[2]); + } + + [Fact] + public void Update_ShouldMarkMissingItemsAsUndetermined() + { + // Items that are missing from the 'updated' payload + // may be 'inaccessible', such as deleted or taken down + // due to a DMCA. We should notice the mods which fall + // under this category. The external code doing the actual + // mod page querying can determine what to do with these based + // on extra info from the API. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(1, 3, now.AddDays(-15)), + }; + + var updater = new PerFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + var updateItems = new[] + { + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + }; + + // Act + updater.Update(updateItems); + var result = updater.Build(); + + // Assert + // item[2] is not in 'update' feed/result, but is within + // the expiry date. This means the mod page may have been archived + // or taken down due to a DMCA. + result.UndeterminedItems.Should().ContainSingle(); + result.UndeterminedItems.Should().Contain(items[2]); + } + + [Fact] + public void Update_ShouldMarkItemsAsUpToDateOrNeedingUpdate() + { + // The PerFeedCacheUpdater correctly compares the 'lastUpdated' field + // of the update entry with the 'lastUpdated' field of the cached item. + // Here we test whether the update function correctly detects if an item + // is older or newer. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + }; + + var updater = new PerFeedCacheUpdater(items, TimeSpan.FromDays(30)); + var updateItems = new[] + { + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + }; + + // Act + updater.Update(updateItems); + var result = updater.Build(); + + // Assert + // item[0] is out of date, item[1] is up-to-date + result.OutOfDateItems.Should().ContainSingle(); + result.OutOfDateItems.Should().Contain(items[0]); + + result.UpToDateItems.Should().ContainSingle(); + result.UpToDateItems.Should().Contain(items[1]); + } + + [Fact] + public void Update_HavingExtraItemsInUpdatePayloadHasNoSideEffects() + { + // Copy of Update_ShouldMarkItemsAsUpToDateOrNeedingUpdate, but with extra + // item(s) in update payload. These items are ones we don't have cached; + // therefore they should be ignored without side effects. + + // Arrange + var now = DateTime.UtcNow; + var items = new[] + { + Create(1, 1, now.AddDays(-10)), + Create(1, 2, now.AddDays(-5)), + Create(1, 3, now.AddDays(-15)), + }; + + var updater = new PerFeedCacheUpdater(items, TimeSpan.FromDays(30)); + + var updateItems = new[] + { + Create(1, 5, now), // New item, should be ignored + Create(1, 6, now), // New item, should be ignored + Create(1, 1, now.AddDays(-8)), // Newer, needs update + Create(1, 2, now.AddDays(-7)), // Older, up-to-date + Create(1, 4, now), // New item, should be ignored + Create(1, 7, now), // New item, should be ignored + }; + + // Act + updater.Update(updateItems); + var result = updater.Build(); + + // Assert + // item[0] is out of date, item[1] is up-to-date + result.OutOfDateItems.Should().ContainSingle(); + result.OutOfDateItems.Should().Contain(items[0]); + + result.UpToDateItems.Should().ContainSingle(); + result.UpToDateItems.Should().Contain(items[1]); + } +} diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Startup.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Startup.cs new file mode 100644 index 0000000000..b8a5a63a3a --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Startup.cs @@ -0,0 +1,11 @@ +using Microsoft.Extensions.DependencyInjection; +namespace NexusMods.Networking.ModUpdates.Tests; + +public class Startup +{ + public void ConfigureServices(IServiceCollection container) + { + + } +} + diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs new file mode 100644 index 0000000000..d49bc2a468 --- /dev/null +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs @@ -0,0 +1,7 @@ +namespace NexusMods.Networking.ModUpdates.Tests; + +public class Tests +{ + [Fact] + public void Test1() { } +} From 7afd32d1447c56a65808c5dd7a100b0350178295 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 26 Sep 2024 05:24:44 +0100 Subject: [PATCH 09/15] Tech Debt Reduction: Add additional V2 GraphQL Types and Correct Sizes of Existing Types --- .../Json/ModSource.cs | 1 + .../NexusModsFileMetadata.cs | 1 + .../NexusModsModPageMetadata.cs | 1 + .../DTOs/GameInfo.cs | 4 +- .../DTOs/ModFile.cs | 5 ++- .../DTOs/ModUpdate.cs | 4 +- .../INexusApiClient.cs | 2 + .../NexusMods.Abstractions.NexusWebApi.csproj | 4 ++ .../NexusModsArchiveMetadata.cs | 1 + .../Types/NXMModUrl.cs | 8 ++-- .../Types/{ => V2}/FileId.cs | 14 +++---- .../Types/{ => V2}/GameId.cs | 7 ++-- .../Types/{ => V2}/ModId.cs | 15 ++++--- .../Types/V2/Uid/UidForFile.cs | 37 +++++++++++++++++ .../Types/V2/Uid/UidForMod.cs | 40 +++++++++++++++++++ .../Types/V2/Uid/UidHelpers.cs | 9 +++++ .../Cyberpunk2077/Emitters/Pattern.cs | 1 + .../Emitters/PatternDefinitions.cs | 1 + .../Verbs/StressTest.cs | 4 +- .../MultiFeedCacheUpdater.cs | 2 +- .../NexusMods.Networking.ModUpdates.csproj | 4 ++ .../PerFeedCacheUpdater.cs | 2 +- .../Structures/GameId.cs | 8 ---- .../Structures/ModId.cs | 10 ----- .../Structures/Uid.cs | 33 --------------- .../Traits/ICanGetUid.cs | 4 +- .../Extensions/ClientExtensions.cs | 1 + .../NexusApiClient.cs | 1 + .../NexusApiVerbs.cs | 1 + .../NexusModsLibrary.cs | 5 ++- .../Downloader/NexusModMetadata.cs | 1 + .../Helpers/TestItem.cs | 11 ++--- .../MultiFeedCacheUpdaterTests.cs | 3 +- .../PerFeedCacheUpdaterTests.cs | 3 +- 34 files changed, 152 insertions(+), 96 deletions(-) rename src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/{ => V2}/FileId.cs (68%) rename src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/{ => V2}/GameId.cs (56%) rename src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/{ => V2}/ModId.cs (55%) create mode 100644 src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs create mode 100644 src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs create mode 100644 src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs delete mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs delete mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs delete mode 100644 src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs diff --git a/src/Abstractions/NexusMods.Abstractions.Collections/Json/ModSource.cs b/src/Abstractions/NexusMods.Abstractions.Collections/Json/ModSource.cs index b1fc29669f..886f8e7c04 100644 --- a/src/Abstractions/NexusMods.Abstractions.Collections/Json/ModSource.cs +++ b/src/Abstractions/NexusMods.Abstractions.Collections/Json/ModSource.cs @@ -1,5 +1,6 @@ using System.Text.Json.Serialization; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Paths; namespace NexusMods.Abstractions.Collections.Json; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsFileMetadata.cs b/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsFileMetadata.cs index 404c0cb288..49771dfbaf 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsFileMetadata.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsFileMetadata.cs @@ -1,6 +1,7 @@ using JetBrains.Annotations; using NexusMods.Abstractions.MnemonicDB.Attributes; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.MnemonicDB.Abstractions.Attributes; using NexusMods.MnemonicDB.Abstractions.Models; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsModPageMetadata.cs b/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsModPageMetadata.cs index 73f293adb6..bdfa2ada14 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsModPageMetadata.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusModsLibrary/NexusModsModPageMetadata.cs @@ -1,6 +1,7 @@ using JetBrains.Annotations; using NexusMods.Abstractions.MnemonicDB.Attributes; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Abstractions.Telemetry; using NexusMods.MnemonicDB.Abstractions.Attributes; using NexusMods.MnemonicDB.Abstractions.Models; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/GameInfo.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/GameInfo.cs index 3bae052e78..4c8f9c8c17 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/GameInfo.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/GameInfo.cs @@ -1,7 +1,7 @@ using System.Text.Json.Serialization; using System.Text.Json.Serialization.Metadata; using NexusMods.Abstractions.NexusWebApi.DTOs.Interfaces; -using GameId = NexusMods.Abstractions.NexusWebApi.Types.GameId; +using GameId = NexusMods.Abstractions.NexusWebApi.Types.V2.GameId; // ReSharper disable MemberCanBePrivate.Global // ReSharper disable InconsistentNaming @@ -27,7 +27,7 @@ public class GameInfo : IJsonArraySerializable /// This field is for deserialization only. /// [JsonPropertyName("id")] - public int _Id { get; set; } + public uint _Id { get; set; } /// /// Returns the ID as typed ValueObject . diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModFile.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModFile.cs index 2faa6b7488..1167260354 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModFile.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModFile.cs @@ -2,7 +2,8 @@ using System.Text.Json.Serialization.Metadata; using NexusMods.Abstractions.NexusWebApi.DTOs.Interfaces; using NexusMods.Abstractions.NexusWebApi.Types; -using FileId = NexusMods.Abstractions.NexusWebApi.Types.FileId; +using NexusMods.Abstractions.NexusWebApi.Types.V2; +using FileId = NexusMods.Abstractions.NexusWebApi.Types.V2.FileId; // 👇 Suppress uninitialised variables. Currently Nexus has mostly read-only API and we expect server to return the data. #pragma warning disable CS8618 @@ -47,7 +48,7 @@ public class ModFile : IJsonSerializable /// This ID is unique within the context of the game. /// i.e. This ID might be used for another mod if you search for mods for another game. /// - public FileId FileId => FileId.From(_FileId); + public FileId FileId => FileId.From((uint)_FileId); /// /// Name (title) of the mod file as seen on the `Files` section of the mod page. diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModUpdate.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModUpdate.cs index bb71b4a136..89c9b9ff1f 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModUpdate.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/DTOs/ModUpdate.cs @@ -1,7 +1,7 @@ using System.Text.Json.Serialization; using System.Text.Json.Serialization.Metadata; using NexusMods.Abstractions.NexusWebApi.DTOs.Interfaces; -using ModId = NexusMods.Abstractions.NexusWebApi.Types.ModId; +using ModId = NexusMods.Abstractions.NexusWebApi.Types.V2.ModId; // 👇 Suppress uninitialised variables. Currently Nexus has mostly read-only API and we expect server to return the data. #pragma warning disable CS8618 @@ -25,7 +25,7 @@ public class ModUpdate : IJsonArraySerializable /// /// An individual mod ID that is unique for this game. /// - public ModId ModId => ModId.From(_ModId); + public ModId ModId => ModId.From((uint)_ModId); /// /// The last time a file on the mod page was updated. diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/INexusApiClient.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/INexusApiClient.cs index bed70e56ae..b922853450 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/INexusApiClient.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/INexusApiClient.cs @@ -1,6 +1,8 @@ using NexusMods.Abstractions.NexusWebApi.DTOs; using NexusMods.Abstractions.NexusWebApi.DTOs.OAuth; using NexusMods.Abstractions.NexusWebApi.Types; +using FileId = NexusMods.Abstractions.NexusWebApi.Types.V2.FileId; +using ModId = NexusMods.Abstractions.NexusWebApi.Types.V2.ModId; namespace NexusMods.Abstractions.NexusWebApi; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusMods.Abstractions.NexusWebApi.csproj b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusMods.Abstractions.NexusWebApi.csproj index 3823713dcc..2d62ced743 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusMods.Abstractions.NexusWebApi.csproj +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusMods.Abstractions.NexusWebApi.csproj @@ -12,4 +12,8 @@ + + + + diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusModsArchiveMetadata.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusModsArchiveMetadata.cs index f5bf9d569c..c2f44012c7 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusModsArchiveMetadata.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/NexusModsArchiveMetadata.cs @@ -1,4 +1,5 @@ using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.MnemonicDB.Abstractions.Attributes; using NexusMods.MnemonicDB.Abstractions.Models; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/NXMModUrl.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/NXMModUrl.cs index eaeeff56b0..c445738df7 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/NXMModUrl.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/NXMModUrl.cs @@ -9,12 +9,12 @@ public class NXMModUrl : NXMUrl /// /// id of the mod page /// - public ModId ModId { get; set; } + public V2.ModId ModId { get; set; } /// /// id of the file (within that game domain) /// - public FileId FileId { get; set; } + public V2.FileId FileId { get; set; } /// /// game domain (name of the game within the Nexus Mods page) @@ -36,8 +36,8 @@ public NXMModUrl(Uri uri) Game = uri.Host; try { - ModId = ModId.From(ulong.Parse(uri.Segments[2].TrimEnd('/'))); - FileId = FileId.From(ulong.Parse(uri.Segments[4].TrimEnd('/'))); + ModId = V2.ModId.From(uint.Parse(uri.Segments[2].TrimEnd('/'))); + FileId = V2.FileId.From(uint.Parse(uri.Segments[4].TrimEnd('/'))); } catch (FormatException) { diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/FileId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs similarity index 68% rename from src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/FileId.cs rename to src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs index ed8eecce6b..c9509bd83d 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/FileId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs @@ -2,8 +2,7 @@ using NexusMods.MnemonicDB.Abstractions.Attributes; using NexusMods.MnemonicDB.Abstractions.ElementComparers; using TransparentValueObjects; - -namespace NexusMods.Abstractions.NexusWebApi.Types; +namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// /// Unique ID for a game file hosted on a mod page. @@ -11,23 +10,22 @@ namespace NexusMods.Abstractions.NexusWebApi.Types; /// This ID is unique within the context of the game. /// i.e. This ID might be used for another mod if you search for mods for another game. /// -[ValueObject] +[ValueObject] // Matches backend. Do not change. public readonly partial struct FileId : IAugmentWith, IAugmentWith { /// - public static FileId DefaultValue => From(default); + public static FileId DefaultValue => From(default(uint)); } - /// /// File ID attribute, for NexusMods API file IDs. /// public class FileIdAttribute(string ns, string name) : - ScalarAttribute(ValueTags.UInt64, ns, name) + ScalarAttribute(ValueTags.UInt64, ns, name) { /// - protected override ulong ToLowLevel(FileId value) => value.Value; + protected override uint ToLowLevel(FileId value) => value.Value; /// - protected override FileId FromLowLevel(ulong value, ValueTags tags, RegistryId registryId) => FileId.From(value); + protected override FileId FromLowLevel(ulong value, ValueTags tags, RegistryId registryId) => FileId.From((uint)value); } diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/GameId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/GameId.cs similarity index 56% rename from src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/GameId.cs rename to src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/GameId.cs index e33aa6f99e..a798ab4adb 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/GameId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/GameId.cs @@ -1,13 +1,12 @@ using TransparentValueObjects; - -namespace NexusMods.Abstractions.NexusWebApi.Types; +namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// /// Unique identifier for an individual game hosted on Nexus. /// -[ValueObject] +[ValueObject] // Matches backend. Do not change. public readonly partial struct GameId : IAugmentWith { /// - public static GameId DefaultValue => From(default); + public static GameId DefaultValue => From(default(uint)); } diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/ModId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs similarity index 55% rename from src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/ModId.cs rename to src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs index 7350226ea6..fa0e2456b6 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/ModId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs @@ -2,18 +2,17 @@ using NexusMods.MnemonicDB.Abstractions.Attributes; using NexusMods.MnemonicDB.Abstractions.ElementComparers; using TransparentValueObjects; - -namespace NexusMods.Abstractions.NexusWebApi.Types; +namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// -/// An individual mod ID. Unique per game. +/// An individual mod ID. Unique per . /// i.e. Each game has its own set of IDs and starts with 0. /// -[ValueObject] +[ValueObject] // Matches backend. Do not change. public readonly partial struct ModId : IAugmentWith, IAugmentWith { /// - public static ModId DefaultValue => From(default); + public static ModId DefaultValue => From(default(uint)); } @@ -21,11 +20,11 @@ namespace NexusMods.Abstractions.NexusWebApi.Types; /// Mod ID attribute, for NexusMods API mod IDs. /// public class ModIdAttribute(string ns, string name) - : ScalarAttribute(ValueTags.UInt64, ns, name) + : ScalarAttribute(ValueTags.UInt64, ns, name) { /// - protected override ulong ToLowLevel(ModId value) => value.Value; + protected override uint ToLowLevel(ModId value) => value.Value; /// - protected override ModId FromLowLevel(ulong value, ValueTags tags, RegistryId registryId) => ModId.From(value); + protected override ModId FromLowLevel(uint value, ValueTags tags, RegistryId registryId) => ModId.From(value); } diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs new file mode 100644 index 0000000000..05a888dc34 --- /dev/null +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs @@ -0,0 +1,37 @@ +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +namespace NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; + +/// +/// This represents a unique ID of an individual file as stored on Nexus Mods. +/// +/// This is a composite key of and , where +/// the upper 4 bytes represent the and the lower 4 bytes represent +/// the . +/// +/// This is consistent with how the Nexus Mods backend produces the UID and is not +/// expected to change. +/// +[StructLayout(LayoutKind.Sequential, Pack = 1)] +public struct UidForFile +{ + /// + /// Unique identifier for the file, within the specific . + /// + public FileId FileId; + + /// + /// Unique identifier for the game. + /// + public GameId GameId; + + /// + /// Reinterprets the current as a single . + /// + public ulong AsUlong => Unsafe.As(ref this); + + /// + /// Reinterprets a given into a . + /// + public static UidForFile FromUlong(ulong value) => Unsafe.As(ref value); +} diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs new file mode 100644 index 0000000000..b4d49c8001 --- /dev/null +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs @@ -0,0 +1,40 @@ +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +namespace NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; + +/// +/// This represents a unique ID of an individual mod page as stored on Nexus Mods. +/// +/// This is a composite key of and , where +/// the upper 4 bytes represent the and the lower 4 bytes represent +/// the . Values are stored in little endian byte order. +/// +/// When transferred over the wire via the API, the resulting `ulong` is converted into +/// a string. +/// +/// This is consistent with how the Nexus Mods backend produces the UID and is not +/// expected to change. +/// +[StructLayout(LayoutKind.Sequential, Pack = 1)] +public struct UidForMod +{ + /// + /// Unique identifier for the mod, within the specific . + /// + public ModId ModId; + + /// + /// Unique identifier for the game. + /// + public GameId GameId; + + /// + /// Reinterprets the current as a single . + /// + public ulong AsUlong => Unsafe.As(ref this); + + /// + /// Reinterprets a given into a . + /// + public static UidForMod FromUlong(ulong value) => Unsafe.As(ref value); +} diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs new file mode 100644 index 0000000000..e0d25662d1 --- /dev/null +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs @@ -0,0 +1,9 @@ +namespace NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; + +/// +/// These are utility methods for working with various UID implementations such as and +/// +internal static class UidHelpers +{ + +} diff --git a/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/Pattern.cs b/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/Pattern.cs index efc561b535..5651746ce0 100644 --- a/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/Pattern.cs +++ b/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/Pattern.cs @@ -2,6 +2,7 @@ using DynamicData.Kernel; using NexusMods.Abstractions.GameLocators; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Paths; namespace NexusMods.Games.RedEngine.Cyberpunk2077.Emitters; diff --git a/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/PatternDefinitions.cs b/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/PatternDefinitions.cs index 9937083dfb..37429ebde1 100644 --- a/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/PatternDefinitions.cs +++ b/src/Games/NexusMods.Games.RedEngine/Cyberpunk2077/Emitters/PatternDefinitions.cs @@ -1,6 +1,7 @@ using System.Text.RegularExpressions; using NexusMods.Abstractions.GameLocators; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Paths; namespace NexusMods.Games.RedEngine.Cyberpunk2077.Emitters; diff --git a/src/Games/NexusMods.Games.TestHarness/Verbs/StressTest.cs b/src/Games/NexusMods.Games.TestHarness/Verbs/StressTest.cs index 693bd3b37d..05ceba0625 100644 --- a/src/Games/NexusMods.Games.TestHarness/Verbs/StressTest.cs +++ b/src/Games/NexusMods.Games.TestHarness/Verbs/StressTest.cs @@ -13,7 +13,7 @@ using NexusMods.ProxyConsole.Abstractions; using NexusMods.ProxyConsole.Abstractions.VerbDefinitions; using NexusMods.StandardGameLocators; -using ModId = NexusMods.Abstractions.NexusWebApi.Types.ModId; +using ModId = NexusMods.Abstractions.NexusWebApi.Types.V2.ModId; namespace NexusMods.Games.TestHarness.Verbs; @@ -38,7 +38,7 @@ internal static async Task RunStressTest( AdvancedManualInstallerUI.Headless = true; var mods = await nexusApiClient.ModUpdatesAsync(game.Domain.Value, PastTime.Day, token); - var results = new List<(string FileName, ModId ModId, Abstractions.NexusWebApi.Types.FileId FileId, Hash Hash, bool Passed, Exception? exception)>(); + var results = new List<(string FileName, ModId ModId, Abstractions.NexusWebApi.Types.V2.FileId FileId, Hash Hash, bool Passed, Exception? exception)>(); await using var gameFolder = temporaryFileManager.CreateFolder(); var (manualId, install) = await manualLocator.Add(game, new Version(1, 0), gameFolder); diff --git a/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs index 29c1bedf4e..c9770addad 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs @@ -1,4 +1,4 @@ -using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Networking.ModUpdates.Traits; namespace NexusMods.Networking.ModUpdates; diff --git a/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj b/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj index 91abef7d7e..adbddb4057 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj +++ b/src/Networking/NexusMods.Networking.ModUpdates/NexusMods.Networking.ModUpdates.csproj @@ -13,4 +13,8 @@ + + + + diff --git a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs index 3189f1ee7c..fac7981bac 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs @@ -1,6 +1,6 @@ using System.Diagnostics; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Networking.ModUpdates.Private; -using NexusMods.Networking.ModUpdates.Structures; using NexusMods.Networking.ModUpdates.Traits; namespace NexusMods.Networking.ModUpdates; diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs deleted file mode 100644 index 8dfbbd207e..0000000000 --- a/src/Networking/NexusMods.Networking.ModUpdates/Structures/GameId.cs +++ /dev/null @@ -1,8 +0,0 @@ -using TransparentValueObjects; -namespace NexusMods.Networking.ModUpdates.Structures; - -/// -/// Represents the unique ID of an individual game in the Nexus Backend. -/// -[ValueObject] // Do not modify. Unsafe code relies on this. -public readonly partial struct GameId { } diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs deleted file mode 100644 index 4b2a82b295..0000000000 --- a/src/Networking/NexusMods.Networking.ModUpdates/Structures/ModId.cs +++ /dev/null @@ -1,10 +0,0 @@ -using TransparentValueObjects; -namespace NexusMods.Networking.ModUpdates.Structures; - -/// -/// Represents the unique ID of an individual mod in the Nexus Backend. -/// This is specific to the it belongs to; -/// forming a composite key. -/// -[ValueObject] // Do not modify. Unsafe code relies on this. -public readonly partial struct ModId { } diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs b/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs deleted file mode 100644 index b72c404b44..0000000000 --- a/src/Networking/NexusMods.Networking.ModUpdates/Structures/Uid.cs +++ /dev/null @@ -1,33 +0,0 @@ -using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -namespace NexusMods.Networking.ModUpdates.Structures; - -/// -/// This represents a unique ID of an individual mod page as stored on Nexus Mods. -/// -/// -/// Not tested on Big Endian architectures. -/// -[StructLayout(LayoutKind.Sequential, Pack = 1)] -public struct Uid -{ - /// - /// Unique identifier for the mod, within the specific . - /// - public ModId ModId; - - /// - /// Unique identifier for the game. - /// - public GameId GameId; - - /// - /// Reinterprets the current as a single . - /// - public ulong AsUlong => Unsafe.As(ref this); - - /// - /// Reinterprets a given into a . - /// - public static Uid FromUlong(ulong value) => Unsafe.As(ref value); -} diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs index b7e341ebe3..5113324876 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs @@ -1,4 +1,4 @@ -using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; namespace NexusMods.Networking.ModUpdates.Traits; /// @@ -13,5 +13,5 @@ public interface ICanGetUid /// Returns a unique identifier for the given item, based on the ID format /// used in the NexusMods V2 API. /// - public Uid GetUniqueId(); + public UidForMod GetUniqueId(); } diff --git a/src/Networking/NexusMods.Networking.NexusWebApi/Extensions/ClientExtensions.cs b/src/Networking/NexusMods.Networking.NexusWebApi/Extensions/ClientExtensions.cs index f5f056a701..0cc5f71c38 100644 --- a/src/Networking/NexusMods.Networking.NexusWebApi/Extensions/ClientExtensions.cs +++ b/src/Networking/NexusMods.Networking.NexusWebApi/Extensions/ClientExtensions.cs @@ -1,6 +1,7 @@ using NexusMods.Abstractions.Games.DTO; using NexusMods.Abstractions.NexusWebApi.DTOs; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; namespace NexusMods.Networking.NexusWebApi.Extensions; diff --git a/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiClient.cs b/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiClient.cs index 2910c37369..fd14a94eba 100644 --- a/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiClient.cs +++ b/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiClient.cs @@ -6,6 +6,7 @@ using NexusMods.Abstractions.NexusWebApi.DTOs.Interfaces; using NexusMods.Abstractions.NexusWebApi.DTOs.OAuth; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; namespace NexusMods.Networking.NexusWebApi; diff --git a/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiVerbs.cs b/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiVerbs.cs index 31bfc9a778..aed7d9a13e 100644 --- a/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiVerbs.cs +++ b/src/Networking/NexusMods.Networking.NexusWebApi/NexusApiVerbs.cs @@ -2,6 +2,7 @@ using NexusMods.Abstractions.Cli; using NexusMods.Abstractions.NexusWebApi; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.ProxyConsole.Abstractions; using NexusMods.ProxyConsole.Abstractions.VerbDefinitions; diff --git a/src/Networking/NexusMods.Networking.NexusWebApi/NexusModsLibrary.cs b/src/Networking/NexusMods.Networking.NexusWebApi/NexusModsLibrary.cs index 08880a2093..f4a5304bbd 100644 --- a/src/Networking/NexusMods.Networking.NexusWebApi/NexusModsLibrary.cs +++ b/src/Networking/NexusMods.Networking.NexusWebApi/NexusModsLibrary.cs @@ -9,6 +9,7 @@ using NexusMods.Abstractions.NexusWebApi; using NexusMods.Abstractions.NexusWebApi.DTOs; using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Extensions.BCL; using NexusMods.MnemonicDB.Abstractions; using NexusMods.Networking.HttpDownloader; @@ -122,7 +123,7 @@ public NexusModsLibrary(IServiceProvider serviceProvider) { var fileInfo = file.File!; var modInfo = fileInfo.Mod; - var nexusModResolver = GraphQLResolver.Create(db, tx, NexusModsModPageMetadata.ModId, ModId.From((ulong)fileInfo.ModId)); + var nexusModResolver = GraphQLResolver.Create(db, tx, NexusModsModPageMetadata.ModId, ModId.From((uint)fileInfo.ModId)); nexusModResolver.Add(NexusModsModPageMetadata.Name, modInfo.Name); nexusModResolver.Add(NexusModsModPageMetadata.GameDomain, GameDomain.From(modInfo.Game.DomainName)); @@ -133,7 +134,7 @@ public NexusModsLibrary(IServiceProvider serviceProvider) nexusModResolver.Add(NexusModsModPageMetadata.ThumbnailUri, thumbnailUri); - var nexusFileResolver = GraphQLResolver.Create(db, tx, (NexusModsFileMetadata.FileId, FileId.From((ulong)fileInfo.FileId)), (NexusModsFileMetadata.ModPageId, nexusModResolver.Id)); + var nexusFileResolver = GraphQLResolver.Create(db, tx, (NexusModsFileMetadata.FileId, FileId.From((uint)fileInfo.FileId)), (NexusModsFileMetadata.ModPageId, nexusModResolver.Id)); nexusFileResolver.Add(NexusModsFileMetadata.ModPageId, nexusModResolver.Id); nexusFileResolver.Add(NexusModsFileMetadata.Name, fileInfo.Name); nexusFileResolver.Add(NexusModsFileMetadata.Version, fileInfo.Version); diff --git a/tests/Games/NexusMods.Games.TestFramework/Downloader/NexusModMetadata.cs b/tests/Games/NexusMods.Games.TestFramework/Downloader/NexusModMetadata.cs index 648b332451..a7e84b60b6 100644 --- a/tests/Games/NexusMods.Games.TestFramework/Downloader/NexusModMetadata.cs +++ b/tests/Games/NexusMods.Games.TestFramework/Downloader/NexusModMetadata.cs @@ -1,4 +1,5 @@ using NexusMods.Abstractions.NexusWebApi.Types; +using NexusMods.Abstractions.NexusWebApi.Types.V2; using NexusMods.Hashing.xxHash64; namespace NexusMods.Games.TestFramework.Downloader; diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs index 1536926193..b5e6c4cb54 100644 --- a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs @@ -1,22 +1,23 @@ -using NexusMods.Networking.ModUpdates.Structures; +using NexusMods.Abstractions.NexusWebApi.Types.V2; +using NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; using NexusMods.Networking.ModUpdates.Traits; -namespace NexusMods.Networking.ModUpdates.Tests; +namespace NexusMods.Networking.ModUpdates.Tests.Helpers; // Helper class to simulate updateable items public class TestItem : ICanGetLastUpdatedTimestamp, ICanGetUid { public DateTime LastUpdated { get; set; } - public Uid Uid { get; set; } + public UidForMod Uid { get; set; } public DateTime GetLastUpdatedDate() => LastUpdated; - public Uid GetUniqueId() => Uid; + public UidForMod GetUniqueId() => Uid; // Helper method to create a test item public static TestItem Create(uint gameId, uint modId, DateTime lastUpdated) { return new TestItem { - Uid = new Uid { GameId = GameId.From(gameId), ModId = ModId.From(modId) }, + Uid = new UidForMod { GameId = GameId.From(gameId), ModId = ModId.From(modId) }, LastUpdated = lastUpdated, }; } diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs index 3f578e0d6e..63a8b7749b 100644 --- a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/MultiFeedCacheUpdaterTests.cs @@ -1,5 +1,6 @@ using FluentAssertions; -using static NexusMods.Networking.ModUpdates.Tests.TestItem; +using NexusMods.Networking.ModUpdates.Tests.Helpers; +using static NexusMods.Networking.ModUpdates.Tests.Helpers.TestItem; namespace NexusMods.Networking.ModUpdates.Tests; diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs index e6acd44711..7abcd9ad4a 100644 --- a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/PerFeedCacheUpdaterTests.cs @@ -1,6 +1,7 @@ using System.Diagnostics; using FluentAssertions; -using static NexusMods.Networking.ModUpdates.Tests.TestItem; +using NexusMods.Networking.ModUpdates.Tests.Helpers; +using static NexusMods.Networking.ModUpdates.Tests.Helpers.TestItem; namespace NexusMods.Networking.ModUpdates.Tests; From 1173046bc453c723f04f507b1c189cb9a2b7b9ee Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 26 Sep 2024 06:12:51 +0100 Subject: [PATCH 10/15] Added: Missing 'UInt32' types in attribute definitions --- .../NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs | 2 +- .../NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs index c9509bd83d..ef5e4adf74 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs @@ -21,7 +21,7 @@ namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// File ID attribute, for NexusMods API file IDs. /// public class FileIdAttribute(string ns, string name) : - ScalarAttribute(ValueTags.UInt64, ns, name) + ScalarAttribute(ValueTags.UInt32, ns, name) { /// protected override uint ToLowLevel(FileId value) => value.Value; diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs index fa0e2456b6..f31a1fd7d0 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/ModId.cs @@ -20,7 +20,7 @@ namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// Mod ID attribute, for NexusMods API mod IDs. /// public class ModIdAttribute(string ns, string name) - : ScalarAttribute(ValueTags.UInt64, ns, name) + : ScalarAttribute(ValueTags.UInt32, ns, name) { /// protected override uint ToLowLevel(ModId value) => value.Value; From 3eb2986e547969c45bca078a417a704a7b071c2b Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 26 Sep 2024 06:13:21 +0100 Subject: [PATCH 11/15] Improved: Accuracy of documentation for FileId struct. --- .../NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs index ef5e4adf74..f6b0314245 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/FileId.cs @@ -5,10 +5,8 @@ namespace NexusMods.Abstractions.NexusWebApi.Types.V2; /// -/// Unique ID for a game file hosted on a mod page. -/// -/// This ID is unique within the context of the game. -/// i.e. This ID might be used for another mod if you search for mods for another game. +/// Unique ID for a mod file associated with a game (). +/// Querying mod pages returns items of this type. /// [ValueObject] // Matches backend. Do not change. public readonly partial struct FileId : IAugmentWith, IAugmentWith From 2de875adcb76cc335954889c0ac51e9737d8a259 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 26 Sep 2024 07:17:49 +0100 Subject: [PATCH 12/15] Added: Method for constructing UidForMod and UidForFile from GraphQL API Results --- .../Types/V2/Uid/UidForFile.cs | 9 +++++++++ .../Types/V2/Uid/UidForMod.cs | 9 +++++++++ .../Types/V2/Uid/UidHelpers.cs | 9 --------- 3 files changed, 18 insertions(+), 9 deletions(-) delete mode 100644 src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs index 05a888dc34..f644dab2b5 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForFile.cs @@ -25,6 +25,15 @@ public struct UidForFile /// public GameId GameId; + /// + /// Decodes a Nexus Mods API result which contains an 'uid' field into a . + /// + /// The 'uid' field of a GraphQL API query. This should be an 8 byte number represented as a string. + /// + /// This throws if is not a valid number. + /// + public static UidForFile FromV2Api(string uid) => FromUlong(ulong.Parse(uid)); + /// /// Reinterprets the current as a single . /// diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs index b4d49c8001..1e5f8987d7 100644 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs +++ b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidForMod.cs @@ -28,6 +28,15 @@ public struct UidForMod /// public GameId GameId; + /// + /// Decodes a Nexus Mods API result which contains an 'uid' field into a . + /// + /// The 'uid' field of a GraphQL API query. This should be an 8 byte number represented as a string. + /// + /// This throws if is not a valid number. + /// + public static UidForMod FromV2Api(string uid) => FromUlong(ulong.Parse(uid)); + /// /// Reinterprets the current as a single . /// diff --git a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs b/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs deleted file mode 100644 index e0d25662d1..0000000000 --- a/src/Abstractions/NexusMods.Abstractions.NexusWebApi/Types/V2/Uid/UidHelpers.cs +++ /dev/null @@ -1,9 +0,0 @@ -namespace NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; - -/// -/// These are utility methods for working with various UID implementations such as and -/// -internal static class UidHelpers -{ - -} From a0d80d48710d0d368158b210ead167281e4352d3 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 26 Sep 2024 14:29:35 +0100 Subject: [PATCH 13/15] Rename: ICanGetUid to ICanGetUidForMod --- .../MultiFeedCacheUpdater.cs | 6 +++--- .../PerFeedCacheUpdater.cs | 10 +++++----- .../PerFeedCacheUpdaterResult.cs | 2 +- .../Traits/{ICanGetUid.cs => ICanGetUidForMod.cs} | 2 +- .../Helpers/TestItem.cs | 2 +- 5 files changed, 11 insertions(+), 11 deletions(-) rename src/Networking/NexusMods.Networking.ModUpdates/Traits/{ICanGetUid.cs => ICanGetUidForMod.cs} (94%) diff --git a/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs index c9770addad..166d21efd3 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/MultiFeedCacheUpdater.cs @@ -12,7 +12,7 @@ namespace NexusMods.Networking.ModUpdates; /// you to use mods and API responses which are sourced from multiple feeds (games), /// as opposed to a single feed. /// -public class MultiFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +public class MultiFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { private readonly Dictionary> _updaters; @@ -74,9 +74,9 @@ public MultiFeedCacheUpdater(TUpdateableItem[] items, TimeSpan expiry) /// is automatically detected. /// /// Wrap elements in a struct that implements - /// and if necessary. + /// and if necessary. /// - public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { foreach (var item in items) { diff --git a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs index fac7981bac..f60a075621 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdater.cs @@ -12,7 +12,7 @@ namespace NexusMods.Networking.ModUpdates; /// This API consists of the following: /// /// 1. Input [Constructor]: A set of items with a 'last update time' (see ) -/// and a 'unique id' (see ) that are relevant to the current 'feed' (game). +/// and a 'unique id' (see ) that are relevant to the current 'feed' (game). /// /// 2. Update [Method]: Submit results from API endpoint returning 'most recently updated mods for game'. /// This updates the internal state of the . @@ -31,7 +31,7 @@ namespace NexusMods.Networking.ModUpdates; /// The 'Feed' in the context of the Nexus App is the individual game's 'updated.json' endpoint; /// i.e. a 'Game Mod Feed' /// -public class PerFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +public class PerFeedCacheUpdater where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { private readonly TUpdateableItem[] _items; private readonly Dictionary _itemToIndex; @@ -79,15 +79,15 @@ public PerFeedCacheUpdater(TUpdateableItem[] items, TimeSpan expiry) /// /// The items returned by the 'most recently updated mods for game' endpoint. /// Wrap elements in a struct that implements - /// and if necessary. + /// and if necessary. /// - public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + public void Update(IEnumerable items) where T : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { foreach (var item in items) UpdateSingleItem(item); } - internal void UpdateSingleItem(T item) where T : ICanGetLastUpdatedTimestamp, ICanGetUid + internal void UpdateSingleItem(T item) where T : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { // Try to get index of the item. // Not all the items from the update feed are locally stored, thus we need to diff --git a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs index 1f14548a1c..78287340c4 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/PerFeedCacheUpdaterResult.cs @@ -6,7 +6,7 @@ namespace NexusMods.Networking.ModUpdates; /// of feeds. /// /// Wrapper for item supported by the cache updater. -public class PerFeedCacheUpdaterResult where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUid +public class PerFeedCacheUpdaterResult where TUpdateableItem : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { /// /// This is a list of items that is 'out of date'. diff --git a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUidForMod.cs similarity index 94% rename from src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs rename to src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUidForMod.cs index 5113324876..06d417eb05 100644 --- a/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUid.cs +++ b/src/Networking/NexusMods.Networking.ModUpdates/Traits/ICanGetUidForMod.cs @@ -7,7 +7,7 @@ namespace NexusMods.Networking.ModUpdates.Traits; /// /// This ID must be truly unique and belong to a specific item of its type. /// -public interface ICanGetUid +public interface ICanGetUidForMod { /// /// Returns a unique identifier for the given item, based on the ID format diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs index b5e6c4cb54..cee5ced92a 100644 --- a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs +++ b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Helpers/TestItem.cs @@ -4,7 +4,7 @@ namespace NexusMods.Networking.ModUpdates.Tests.Helpers; // Helper class to simulate updateable items -public class TestItem : ICanGetLastUpdatedTimestamp, ICanGetUid +public class TestItem : ICanGetLastUpdatedTimestamp, ICanGetUidForMod { public DateTime LastUpdated { get; set; } public UidForMod Uid { get; set; } From e8e09e3213447437f232977646393573ca0cdf77 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Fri, 27 Sep 2024 02:02:42 +0100 Subject: [PATCH 14/15] Added: Tests for UidForModTests and UidForFileTests --- .../Types/V2/UidForFileTests.cs | 312 ++++++++++++++ .../Types/V2/UidForModTests.cs | 400 ++++++++++++++++++ 2 files changed, 712 insertions(+) create mode 100644 tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForFileTests.cs create mode 100644 tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForModTests.cs diff --git a/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForFileTests.cs b/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForFileTests.cs new file mode 100644 index 0000000000..521101cca1 --- /dev/null +++ b/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForFileTests.cs @@ -0,0 +1,312 @@ +using FluentAssertions; +using NexusMods.Abstractions.NexusWebApi.Types.V2; +using NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; +namespace NexusMods.Networking.NexusWebApi.Tests.Types.V2; + +public class UidForFileTests +{ + [Fact] + public void UidForFile_IsCorrectSize() + { + // Arrange + unsafe + { + // UidForFile does an unsafe cast in FromUlong and AsUlong + // This test ensures nobody tampers with the size of the struct + // or its components; ensuring those unsafe casts are safe. + sizeof(UidForFile).Should().Be(8); + sizeof(FileId).Should().Be(4); + sizeof(GameId).Should().Be(4); + } + } + + [Theory] + [InlineData(1704U, 405U, "7318624272789")] + [InlineData(1704U, 407U, "7318624272791")] + [InlineData(1704U, 406U, "7318624272790")] + [InlineData(1704U, 5564U, "7318624277948")] + [InlineData(1704U, 5565U, "7318624277949")] + [InlineData(1704U, 163337U, "7318624435721")] + [InlineData(1704U, 163338U, "7318624435722")] + [InlineData(1704U, 296267U, "7318624568651")] + [InlineData(1704U, 296268U, "7318624568652")] + [InlineData(3333U, 1U, "14315125997569")] + [InlineData(3333U, 2U, "14315125997570")] + [InlineData(3333U, 41002U, "14315126038570")] + [InlineData(3333U, 4U, "14315125997572")] + public void FromV2Api_ValidInput_ReturnsCorrectUidForFile(uint expectedGameId, uint expectedFileId, string uidString) + { + // Act + var result = UidForFile.FromV2Api(uidString); + + // Assert + result.GameId.Should().Be((GameId)expectedGameId); + result.FileId.Should().Be((FileId)expectedFileId); + } + + [Fact] + public void FromV2Api_InvalidInput_ThrowsFormatException() + { + // Arrange + var invalidUid = "not a number"; + + // Act & Assert + Action act = () => UidForFile.FromV2Api(invalidUid); + act.Should().Throw(); + } + + [Theory] + [InlineData(1704U, 405U, 7318624272789UL)] + [InlineData(1704U, 407U, 7318624272791UL)] + [InlineData(1704U, 406U, 7318624272790UL)] + [InlineData(3333U, 1U, 14315125997569UL)] + [InlineData(3333U, 2U, 14315125997570UL)] + public void AsUlong_ReturnsCorrectValue(uint gameId, uint fileId, ulong expectedUlong) + { + // Arrange + var uidForFile = new UidForFile { GameId = (GameId)gameId, FileId = (FileId)fileId }; + + // Act + var result = uidForFile.AsUlong; + + // Assert + result.Should().Be(expectedUlong); + } + + [Theory] + [InlineData(7318624272789UL, 1704U, 405U)] + [InlineData(7318624272791UL, 1704U, 407U)] + [InlineData(7318624272790UL, 1704U, 406U)] + [InlineData(14315125997569UL, 3333U, 1U)] + [InlineData(14315125997570UL, 3333U, 2U)] + public void FromUlong_ReturnsCorrectUidForFile(ulong input, uint expectedGameId, uint expectedFileId) + { + // Act + var result = UidForFile.FromUlong(input); + + // Assert + result.GameId.Should().Be((GameId)expectedGameId); + result.FileId.Should().Be((FileId)expectedFileId); + } + + [Theory] + [InlineData(1704U, 405U)] + [InlineData(3333U, 1U)] + public void RoundTrip_UlongConversion_PreservesValues(uint gameId, uint fileId) + { + // Arrange + var original = new UidForFile { GameId = (GameId)gameId, FileId = (FileId)fileId }; + + // Act + var asUlong = original.AsUlong; + var roundTripped = UidForFile.FromUlong(asUlong); + + // Assert + roundTripped.Should().Be(original); + } +} + + +/* + Deriveration of test cases. + + Original Request(s): + + ``` + query ModFiles($modId: ID!, $gameId: ID!) { + modFiles(modId: $modId, gameId: $gameId) { + fileId + uid + } + } + ``` + + Input: + ``` + { + "modId": 1, + "gameId": "1704" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 405, + "uid": "7318624272789" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 2, + "gameId": "1704" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 407, + "uid": "7318624272791" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 3, + "gameId": "1704" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 406, + "uid": "7318624272790" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 1000, + "gameId": "1704" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 5564, + "uid": "7318624277948" + }, + { + "fileId": 5565, + "uid": "7318624277949" + }, + { + "fileId": 163337, + "uid": "7318624435721" + }, + { + "fileId": 163338, + "uid": "7318624435722" + }, + { + "fileId": 296267, + "uid": "7318624568651" + }, + { + "fileId": 296268, + "uid": "7318624568652" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 1, + "gameId": "3333" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 1, + "uid": "14315125997569" + }, + { + "fileId": 2, + "uid": "14315125997570" + }, + { + "fileId": 41002, + "uid": "14315126038570" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 1, + "gameId": "3333" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 1, + "uid": "14315125997569" + }, + { + "fileId": 2, + "uid": "14315125997570" + }, + { + "fileId": 41002, + "uid": "14315126038570" + } + ] + } + } + ``` + + Input: + ``` + { + "modId": 3, + "gameId": "3333" + } + ``` + + Response: + ```json + { + "data": { + "modFiles": [ + { + "fileId": 4, + "uid": "14315125997572" + } + ] + } + } +*/ diff --git a/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForModTests.cs b/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForModTests.cs new file mode 100644 index 0000000000..f93af91551 --- /dev/null +++ b/tests/Networking/NexusMods.Networking.NexusWebApi.Tests/Types/V2/UidForModTests.cs @@ -0,0 +1,400 @@ +using FluentAssertions; +using NexusMods.Abstractions.NexusWebApi.Types.V2; +using NexusMods.Abstractions.NexusWebApi.Types.V2.Uid; +namespace NexusMods.Networking.NexusWebApi.Tests.Types.V2; + +public class UidForModTests +{ + [Fact] + public void UidForMod_IsCorrectSize() + { + // Arrange + unsafe + { + // UidForMod does an unsafe cast in FromUlong and AsUlong + // This test ensures nobody tampers with the size of the struct + // or its components; ensuring those unsafe casts are safe. + sizeof(UidForMod).Should().Be(8); + sizeof(GameId).Should().Be(4); + sizeof(FileId).Should().Be(4); + } + } + + [Theory] + [InlineData(1704U, 130248U, "7318624402632")] + [InlineData(1704U, 130167U, "7318624402551")] + [InlineData(1704U, 130246U, "7318624402630")] + [InlineData(1704U, 130245U, "7318624402629")] + [InlineData(1704U, 130243U, "7318624402627")] + [InlineData(1704U, 130244U, "7318624402628")] + [InlineData(1704U, 130242U, "7318624402626")] + [InlineData(1704U, 130240U, "7318624402624")] + [InlineData(1704U, 130241U, "7318624402625")] + [InlineData(1704U, 130191U, "7318624402575")] + [InlineData(1704U, 130239U, "7318624402623")] + [InlineData(1704U, 129994U, "7318624402378")] + [InlineData(1704U, 130237U, "7318624402621")] + [InlineData(1704U, 130238U, "7318624402622")] + [InlineData(1704U, 130234U, "7318624402618")] + [InlineData(1704U, 130235U, "7318624402619")] + [InlineData(1704U, 130230U, "7318624402614")] + [InlineData(1704U, 130233U, "7318624402617")] + [InlineData(1704U, 130232U, "7318624402616")] + [InlineData(1704U, 130231U, "7318624402615")] + [InlineData(2500U, 76U, "10737418240076")] + [InlineData(2500U, 75U, "10737418240075")] + [InlineData(2500U, 74U, "10737418240074")] + [InlineData(2500U, 73U, "10737418240073")] + [InlineData(2500U, 72U, "10737418240072")] + [InlineData(2500U, 70U, "10737418240070")] + [InlineData(2500U, 69U, "10737418240069")] + [InlineData(2500U, 68U, "10737418240068")] + [InlineData(2500U, 67U, "10737418240067")] + [InlineData(2500U, 66U, "10737418240066")] + [InlineData(2500U, 65U, "10737418240065")] + [InlineData(2500U, 64U, "10737418240064")] + [InlineData(2500U, 63U, "10737418240063")] + [InlineData(2500U, 62U, "10737418240062")] + [InlineData(2500U, 60U, "10737418240060")] + [InlineData(2500U, 59U, "10737418240059")] + [InlineData(2500U, 58U, "10737418240058")] + [InlineData(2500U, 57U, "10737418240057")] + [InlineData(2500U, 56U, "10737418240056")] + [InlineData(2500U, 55U, "10737418240055")] + public void FromV2Api_ValidInput_ReturnsCorrectUidForMod(uint expectedGameId, uint expectedModId, string uidString) + { + // Act + var result = UidForMod.FromV2Api(uidString); + + // Assert + result.GameId.Should().Be((GameId)expectedGameId); + result.ModId.Should().Be((ModId)expectedModId); + } + + [Fact] + public void FromV2Api_InvalidInput_ThrowsFormatException() + { + // Arrange + var invalidUid = "not a number"; + + // Act & Assert + Action act = () => UidForMod.FromV2Api(invalidUid); + act.Should().Throw(); + } + + [Theory] + [InlineData(1704U, 130248U, 7318624402632UL)] + [InlineData(1704U, 130167U, 7318624402551UL)] + [InlineData(1704U, 130246U, 7318624402630UL)] + [InlineData(1704U, 130245U, 7318624402629UL)] + [InlineData(1704U, 130243U, 7318624402627UL)] + [InlineData(2500U, 76U, 10737418240076UL)] + [InlineData(2500U, 75U, 10737418240075UL)] + [InlineData(2500U, 74U, 10737418240074UL)] + [InlineData(2500U, 73U, 10737418240073UL)] + [InlineData(2500U, 72U, 10737418240072UL)] + public void AsUlong_ReturnsCorrectValue(uint gameId, uint modId, ulong expectedUlong) + { + // Arrange + var uidForMod = new UidForMod { GameId = (GameId)gameId, ModId = (ModId)modId }; + + // Act + var result = uidForMod.AsUlong; + + // Assert + result.Should().Be(expectedUlong); + } + + [Theory] + [InlineData(7318624402632UL, 1704U, 130248U)] + [InlineData(7318624402551UL, 1704U, 130167U)] + [InlineData(7318624402630UL, 1704U, 130246U)] + [InlineData(7318624402629UL, 1704U, 130245U)] + [InlineData(7318624402627UL, 1704U, 130243U)] + [InlineData(10737418240076UL, 2500U, 76U)] + [InlineData(10737418240075UL, 2500U, 75U)] + [InlineData(10737418240074UL, 2500U, 74U)] + [InlineData(10737418240073UL, 2500U, 73U)] + [InlineData(10737418240072UL, 2500U, 72U)] + public void FromUlong_ReturnsCorrectUidForMod(ulong input, uint expectedGameId, uint expectedModId) + { + // Act + var result = UidForMod.FromUlong(input); + + // Assert + result.GameId.Should().Be((GameId)expectedGameId); + result.ModId.Should().Be((ModId)expectedModId); + } + + [Theory] + [InlineData(1704U, 130248U)] + [InlineData(2500U, 76U)] + public void RoundTrip_UlongConversion_PreservesValues(uint gameId, uint modId) + { + // Arrange + var original = new UidForMod { GameId = (GameId)gameId, ModId = (ModId)modId }; + + // Act + var asUlong = original.AsUlong; + var roundTripped = UidForMod.FromUlong(asUlong); + + // Assert + roundTripped.Should().Be(original); + } +} + + +/* + Deriveration of test cases. + + Original Request(s): + + ``` + query Mods { + mods(filter: { gameId: { value: "1704", op: EQUALS } }) { + nodes { + gameId + modId + uid + } + } + } + ``` + + ``` + query Mods { + mods(filter: { gameId: { value: "2500", op: EQUALS } }) { + nodes { + gameId + modId + uid + } + } + } + ``` + + Original Response(s): + + ```json + { + "data": { + "mods": { + "nodes": [ + { + "gameId": 1704, + "modId": 130248, + "uid": "7318624402632" + }, + { + "gameId": 1704, + "modId": 130167, + "uid": "7318624402551" + }, + { + "gameId": 1704, + "modId": 130246, + "uid": "7318624402630" + }, + { + "gameId": 1704, + "modId": 130245, + "uid": "7318624402629" + }, + { + "gameId": 1704, + "modId": 130243, + "uid": "7318624402627" + }, + { + "gameId": 1704, + "modId": 130244, + "uid": "7318624402628" + }, + { + "gameId": 1704, + "modId": 130242, + "uid": "7318624402626" + }, + { + "gameId": 1704, + "modId": 130240, + "uid": "7318624402624" + }, + { + "gameId": 1704, + "modId": 130241, + "uid": "7318624402625" + }, + { + "gameId": 1704, + "modId": 130191, + "uid": "7318624402575" + }, + { + "gameId": 1704, + "modId": 130239, + "uid": "7318624402623" + }, + { + "gameId": 1704, + "modId": 129994, + "uid": "7318624402378" + }, + { + "gameId": 1704, + "modId": 130237, + "uid": "7318624402621" + }, + { + "gameId": 1704, + "modId": 130238, + "uid": "7318624402622" + }, + { + "gameId": 1704, + "modId": 130234, + "uid": "7318624402618" + }, + { + "gameId": 1704, + "modId": 130235, + "uid": "7318624402619" + }, + { + "gameId": 1704, + "modId": 130230, + "uid": "7318624402614" + }, + { + "gameId": 1704, + "modId": 130233, + "uid": "7318624402617" + }, + { + "gameId": 1704, + "modId": 130232, + "uid": "7318624402616" + }, + { + "gameId": 1704, + "modId": 130231, + "uid": "7318624402615" + } + ] + } + } + } + ``` + + ```json + { + "data": { + "mods": { + "nodes": [ + { + "gameId": 2500, + "modId": 76, + "uid": "10737418240076" + }, + { + "gameId": 2500, + "modId": 75, + "uid": "10737418240075" + }, + { + "gameId": 2500, + "modId": 74, + "uid": "10737418240074" + }, + { + "gameId": 2500, + "modId": 73, + "uid": "10737418240073" + }, + { + "gameId": 2500, + "modId": 72, + "uid": "10737418240072" + }, + { + "gameId": 2500, + "modId": 70, + "uid": "10737418240070" + }, + { + "gameId": 2500, + "modId": 69, + "uid": "10737418240069" + }, + { + "gameId": 2500, + "modId": 68, + "uid": "10737418240068" + }, + { + "gameId": 2500, + "modId": 67, + "uid": "10737418240067" + }, + { + "gameId": 2500, + "modId": 66, + "uid": "10737418240066" + }, + { + "gameId": 2500, + "modId": 65, + "uid": "10737418240065" + }, + { + "gameId": 2500, + "modId": 64, + "uid": "10737418240064" + }, + { + "gameId": 2500, + "modId": 63, + "uid": "10737418240063" + }, + { + "gameId": 2500, + "modId": 62, + "uid": "10737418240062" + }, + { + "gameId": 2500, + "modId": 60, + "uid": "10737418240060" + }, + { + "gameId": 2500, + "modId": 59, + "uid": "10737418240059" + }, + { + "gameId": 2500, + "modId": 58, + "uid": "10737418240058" + }, + { + "gameId": 2500, + "modId": 57, + "uid": "10737418240057" + }, + { + "gameId": 2500, + "modId": 56, + "uid": "10737418240056" + }, + { + "gameId": 2500, + "modId": 55, + "uid": "10737418240055" + } + ] + } + } + } + ``` + +*/ From e4349be69b8ef1fee3a2de7612df9ed01b4f8dd6 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Fri, 27 Sep 2024 02:04:10 +0100 Subject: [PATCH 15/15] Removed: Unused Tests.cs file --- .../NexusMods.Networking.ModUpdates.Tests/Tests.cs | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs diff --git a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs b/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs deleted file mode 100644 index d49bc2a468..0000000000 --- a/tests/Networking/NexusMods.Networking.ModUpdates.Tests/Tests.cs +++ /dev/null @@ -1,7 +0,0 @@ -namespace NexusMods.Networking.ModUpdates.Tests; - -public class Tests -{ - [Fact] - public void Test1() { } -}