ADR and Implementation of New Default App Locations (#783)

* ADR for Default DataStore Locations

w/ Minor Update (Sanity Test)

* Changed: Minor revision to make Last Section more Readable

* Added: Note about Home Directory Approach being Streamed over Network

* Added: ADR Update with Portable Mode & UX Discussion Notes

* Added: Documentation for DataStore Locations

* Part 1: Settings now default to chosen path from ADR & Feedback

* Added: Handling for AppImage(s)

[needs testing]

* Added: NexusMods.App folder to LocalAppData/XDG_DATA_HOME

* Move Logs to XDG_STATE_HOME as per XDG spec

* Make a note on TEMP Folder Location

* Added: Remaining Leftover to DataStore Locations

May change, after discussion

* Change Temp files to XDG_STATE_HOME

* Changed: On AppImage, OWD should take precedence

* Added: Link to AppImage Environment Variables

* Changed: NotSupportedException -> PlatformNotSupported Exception on macOS

Directory.Packages.props

@@ -2,7 +2,7 @@
   <ItemGroup>
     <!-- Custom Packages -->
     <PackageVersion Include="NexusMods.Hashing.xxHash64" Version="1.0.1" />
-    <PackageVersion Include="NexusMods.Paths" Version="0.2.0" />
+    <PackageVersion Include="NexusMods.Paths" Version="0.3.0" />
     <PackageVersion Include="NexusMods.Archives.Nx" Version="0.3.7" />
     <PackageVersion Include="NexusMods.Paths.TestingHelpers" Version="0.2.0" />
     <PackageVersion Include="NexusMods.ProxyConsole.Abstractions" Version="0.6.0" />

docs/decisions/backend/0013-default-datastore-locations.md

@@ -0,0 +1,241 @@
+```
+# These are optional elements. Feel free to remove any of them.
+status: {proposed}
+date: {2023-11-21 when the decision was last updated}
+deciders: {App Team}
+```
+
+# Default Storage Locations for NexusMods.App Files
+
+## Context and Problem Statement
+
+The App currently stores its files in the same folder as the application (`{EntryFolder}`).
+
+This approach has issues with certain packaging formats. For example, when the application is run from
+an AppImage, the `EntryFolder` becomes read-only, and our application fails to run.
+
+We need to identify alternative storage locations that are both accessible and writable, regardless of how the
+application is deployed or executed.
+
+## The Current Situation
+
+| Packaging System        | Can Write Entry Directory? | Notes                                                                       |
+|-------------------------|----------------------------|-----------------------------------------------------------------------------|
+| Windows (User Folder)   | ✅                          | No issues.                                                                  |
+| Windows (Program Files) | ⚠️                         | Requires elevated permissions.                                              |
+| macOS (User Folder)     | ✅                          | No issues.                                                                  |
+| macOS (/Applications)   | ⚠️                         | Requires elevated permissions. Reportedly affects code signing workflow.    |
+| Linux (AppImage)        | ⚠️                         | Read only. Can write to `.AppImage` folder via `$OWD` environment variable. |
+| Linux (Flatpak)         | ❌                          | Read only.                                                                  |
+
+## Per Distribution Method Notes
+
+| Packaging System | Notes                                                                                                             |
+|------------------|-------------------------------------------------------------------------------------------------------------------|
+| Windows          | Requires elevated permissions for non-user folders.                                                               |
+| macOS            | Requires elevated permissions for non-user folders.                                                               |
+| Linux (AppImage) | Single file app. Mounted on launch.                                                                               |
+| Linux (Flatpak)  | Can only access certain directories by default. Full FileSystem permissions may be requested via manifest change. |
+
+## Decision Drivers
+
+* On uninstall, the application should be able to remove all of its files from the system.
+* Default storage locations should be configurable.
+    * This is already the case today, but stating this here marks the functionality as a requirement.
+* The application should be 'portable'.
+    * In other words, it should be able to be run from a USB stick, etc.
+* Application data should be user accessible.
+    * In other words, easy to find/navigate to.
+    * Because application data, also contains logs and user accessible files.
+    * e.g. If the App crashes on boot, finding logs should be easy.
+* Cross-platform consistency.
+    * The paths should be consistent across different operating systems.
+    * For example, if on one platform it is in entry directory, it should do the same on other platforms too.
+* The user must have write access to the directory.
+
+## Considered Options
+
+* **Option 0: Keep using Entry Directory**
+* **Option 1: User's Home Directory**
+* **Option 2: Operating System's AppData Directory**
+
+## Decision Outcome
+
+We went with `Option 2: Operating System's AppData Directory`.
+
+With the following caveats/notes:
+
+- We will assume the machine is single user for now. (no `Roaming` directory)
+- We will use idiomatic paths for each operating system.
+
+| Directory       | Windows        | Linux             |
+|-----------------|----------------|-------------------|
+| DataModel       | %localappdata% | XDG_DATA_HOME     |
+| Temporary Files | %temp%         | XDG_STATE_HOME ⚠️ |
+| Logs            | %localappdata% | XDG_STATE_HOME    |
+
+⚠️ Non-standard location. This is because we risk RAM starvation on large downloads as `tmpfs` is often in RAM+swap.
+
+Inside `NexusMods.App` subfolder, of course.
+macOS is not yet implemented, so is currently omitted.
+
+## Pros and Cons of the Options
+
+### Option 0: Keep using Entry Directory
+
+* Good, because it easily makes the application portable.
+* Good, because it is consistent across deployment methods.
+* Good, because it makes user data very easy to find.
+* Bad, because it is incompatible with many packaging systems.
+
+### Option 1: User's Home Directory
+
+In a subfolder of:
+
+- `~` on Linux/macOS
+- `C:\Users\{User}` on Windows
+
+* Good, because it is easy to find.
+* Good, because it remains consistent across different deployment methods.
+* Bad, because it can clutter the user's personal space with application data. ('yet another folder in my home
+  directory')
+
+### Option 2: Operating System's AppData Directory
+
+In a subfolder of:
+
+- `AppData/Roaming` on Windows
+- `~/Library/Application Support` on macOS
+- `~/.config` on Linux a.k.a. `XDG_CONFIG_HOME`
+
+* Good, because it's the idiomatic (intended) location for application data.
+* Good, because it separates application data from user files, reducing clutter.
+* Bad, because user might struggle to find these locations.
+
+---------
+
+# Additional Considerations
+
+## Issue: Separate Per User and Per Machine Data
+
+---------
+
+Decision: Discussed
+in [Meeting Notes: Default Storage Locations](../meeting-notes/0000-datastore-locations.md#default-storage-location). We
+will pretend multi-user systems don't exist for now.
+
+---------
+
+!!! note "We will be using Windows as an example here, but this also applies to other operating systems"
+
+The current issue with idiomatically using [AppData](#option-2-operating-systems-appdata-directory) is
+the presence of multi user systems. For example, offices, LAN centres, etc.
+
+There are two specific issues:
+
+### 1. Network Synchronization
+
+!!! Addressed
+
+Files in `AppData/Roaming` are usually downloaded upon login in these configurations, and this download typically
+happens on every login.
+
+This is bad because it means having to potentially wait a very long time (even at Gigabit speed) to download a lot of
+data on login. In the case of something like Starfield, game backup + mods could exceed 100GB, meaning a 10+ minute
+download if no data is locally cached.
+
+In the case of alternative [Home Directory](#option-1-users-home-directory) approach however, the home directory can
+just be accessed over the network, avoiding the need for a long synchronization wait time. (At expense of slow
+deployment times as mod archives would be accessed over the network)
+
+### 2. Disk Space (Local)
+
+In the other case of multiple users on same local machine, all mod + game backup data is duplicated on storage, wasting
+potentially a huge amount of space.
+
+### Additional Context
+
+The idiomatic approach for this kind of problem is storing mods + backup game files
+in a machine wide location such as `C:\ProgramData`. Per user data (loadouts, mod configs etc.) in `AppData/Roaming`.
+
+When using in a multi-machine setup, like an office, the App would load the loadout from `AppData/Roaming` and start
+downloading any mod assets it may be missing. If they are modding a game which has been modded by another user on the
+same local system, the shared backup game files in `C:\ProgramData` would be used to first restore the game to its
+original state.
+
+Such as system may mean a slight rework of the DataStore however, as it means having to know where each file was
+originally sourced from and (possibly) having a separate data store for user and machine wide data.
+
+### The Alternative
+
+Pretend the problem doesn't exist. Not uncommon in software development (sadly).
+More convenient for developers, but the App however would work very poorly in multi user environments.
+
+## Action Plan: Moving Default Configuration File
+
+---------
+
+Decision: [Portable mode to be implemented](../meeting-notes/0000-datastore-locations.md#portable-mode) with
+`AppConfig.json` override method. Lack of `AppConfig.json` indicates 'use default paths'. Remaining settings
+to be moved to DataStore.
+
+For now, as a compromise, as not to not cause a regression in functionality, we will blank out the default
+paths in the `AppConfig.json` file. If they are blank, we will apply the default paths decided upon in this ADR.
+
+---------
+
+Currently the App stores its configuration file (`AppConfig.json`) for custom paths in the same folder as the
+application (`{EntryFolder}`). This is of course problematic, as many packaging systems make this folder read-only.
+
+### Moving the Configuration File
+
+Once changes tied to this ADR are applied, this config file should be moved to whatever is the new default decided
+as a result of this ADR. If no file exists, the App should create one in the default location with the default values.
+
+### Required: Support for 'Portable Mode'
+
+For users who wish to run the App in 'portable mode' (e.g. in a non-static location), an override will need to exist to
+read paths from the local folder (which is current behaviour).
+
+This can be done in one of following ways:
+
+- `AppConfig.json` in App folder takes precedence over default location `AppConfig.json`.
+- `portable.txt` file forces a default set of 'portable' paths to be used.
+
+The first option is more powerful, but the second option may be more familiar to users. In any case, implementing either
+and switching between them code wise should be rather trivial.
+
+!! danger
+
+'Portable Mode' can only be supported for distribution methods where the App folder is writeable, for a list of these,
+[see 'The Current Situation'](#the-current-situation). How we communicate this to the user is TBD.
+
+## Future Questions: User Path Configuration UX
+
+!! note "This is a pending discussion topic. To be moved to future ADRs."
+
+Discussion on the User Experience of configuring paths is still pending.
+
+Here are some (current) proposed discussion topics:
+
+- How do we advertise 'portable' mode.
+    - Either we copy the App files from a read-only directory to a new location, then tell user to uninstall their '
+      installed' version.
+    - Or users will need to download a separate build (e.g. AppImage instead of Flatpak, i.e. `ZIP` instead of `MSI` on
+      Windows) to use it.
+    - We can't have custom installer logic, as not all packaging formats support it. Do we make a custom installer?
+
+- Support installing archives to multiple locations.
+    - Do we do automated per game rules?
+    - Size based rules?
+    - Delegating to secondary locations if primary is full?
+
+- Support dynamically moving the Archives between folders.
+    - Do we use Steam inspired UI for this?
+    - Reboot required?
+
+---------
+
+Decision: To be further discussed later. Location per game to be supported in the future.
+
+---------

docs/decisions/meeting-notes/0000-datastore-locations.md

@@ -0,0 +1,54 @@
+# Meeting Summary on Default Storage Locations
+
+**Date of Meeting**: 26th of November, 2023
+
+**Attendees**: App Team Only (No Vortex/Design)
+
+**Agenda**:
+
+1. Final decision on default storage location for app files.  
+2. Discussion on Portable Mode implementation.  
+3. Decision on handling separate per user and per machine data.  
+4. Additional context and future considerations.  
+
+## Decisions Made
+
+### Default Storage Location
+
+**Agreed upon LocalAppData**. The team unanimously agreed to use the `LocalAppData` on Windows 
+and `XDG_DATA_HOME` on Linux. This is Option 2 from the ADR, however with only the use of LocalAppData. 
+
+We chose to pretend (for now) that Multi-User systems aren't within our scope as they are very 
+infrequently used.
+
+This keeps our DataModel simpler, without the need to track file source locations (where to 
+download them from) or splitting the DataModel itself into two.
+
+### Portable Mode
+
+**Support for Portable Mode Confirmed**. Team chose to vote in favour of supporting a portable 
+mode based on past experience with end users and and their mod managers.
+
+@erri120 made an note that one of the fact that part of the main appeal of a 'portable mode' is 
+being able to transfer the data between one machine to another. So technically, that can already be 
+fulfilled using built-in export/import functionality. 
+
+In any case, as not much work is currently required to deliver this (it already is portable, in fact!), 
+the team decided to support it.
+
+There are some concerns, however, such as the user potentially managing game with multiple 
+managers, leading to an inconsistent state. For now, we expect the user to be responsible for
+not making this mistake.
+
+### Future Plans
+
+**Auto Installs to Multiple Locations**: This feature will be addressed in the future. The goal 
+is to allow users to specify storage locations on a per-game basis.
+
+## Next Steps and Action Items
+
+- **Implementing LocalAppData as the Default Storage**: Transition the app's default storage location 
+  to `LocalAppData` / `XDG_DATA_HOME` as per the decision.
+
+- **Developing Portable Mode**: Basically make the current config file we have 'optional', if it 
+  exists, use it, otherwise default to a 'default' storage location.

src/ArchiveManagement/NexusMods.FileExtractor/IFileExtractorSettings.cs

@@ -21,26 +21,44 @@ public interface IFileExtractorSettings
 [PublicAPI]
 public class FileExtractorSettings : IFileExtractorSettings
 {
-    // Note: We can't serialize AbsolutePath because it contains more fields than expected. Just hope user sets correct paths and pray for the best.
-    private readonly IFileSystem _fileSystem;
-
-
+    /// <inheritdoc />
+    public ConfigurationPath TempFolderLocation { get; set; }
+
     /// <summary>
     /// Default constructor for serialization.
     /// </summary>
-    public FileExtractorSettings() : this(FileSystem.Shared) {}
+    public FileExtractorSettings() : this(FileSystem.Shared) { }
 
     /// <summary>
     /// Creates a default new instance of <see cref="FileExtractorSettings"/>.
     /// </summary>
     /// <param name="fileSystem"></param>
     public FileExtractorSettings(IFileSystem fileSystem)
     {
-        _fileSystem = fileSystem;
-        TempFolderLocation = new ConfigurationPath(_fileSystem
-            .GetKnownPath(KnownPath.EntryDirectory).Combine("Temp"));
+        TempFolderLocation = new ConfigurationPath(GetDefaultBaseDirectory(fileSystem));
+    }
+
+    private static AbsolutePath GetDefaultBaseDirectory(IFileSystem fs)
+    {
+        // Note: The idiomatic place for this is Temporary Directory (/tmp on Linux, %TEMP% on Windows)
+        //       however this can be dangerous to do on Linux, as /tmp is often a RAM disk, and can be
+        //       too small to handle large files.
+        return fs.OS.MatchPlatform(
+            () => fs.GetKnownPath(KnownPath.TempDirectory).Combine("NexusMods.App/Temp"),
+            () => fs.GetKnownPath(KnownPath.XDG_STATE_HOME).Combine("NexusMods.App/Temp"),
+            () => throw new PlatformNotSupportedException(
+                "(Note: Sewer) Paths needs PR for macOS. I don't have a non-painful way to access a Mac."));
+    }
+
+    /// <summary>
+    /// Ensures default settings in case of placeholders of undefined/invalid settings.
+    /// </summary>
+    public void Sanitize(IFileSystem fs)
+    {
+        // Set default locations if none are provided.
+        if (string.IsNullOrEmpty(TempFolderLocation.RawPath))
+            TempFolderLocation = new ConfigurationPath(GetDefaultBaseDirectory(fs));
+
+        TempFolderLocation.ToAbsolutePath().CreateDirectory();
     }
-
-    /// <inheritdoc />
-    public ConfigurationPath TempFolderLocation { get; set; }
 }