Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files recognized as archives inside mods are extracted and not properly added to the mod #2367

Open
Al12rs opened this issue Dec 11, 2024 · 6 comments
Labels
Bug Something isn't working Tech: data-store This is related to the Data Store.

Comments

@Al12rs
Copy link
Contributor

Al12rs commented Dec 11, 2024

Bug Report

Outline:

If a mod added to the library contains a file that the app recognizes as an archive, the app will attempt to extract the contents of that archive, and the file entry is recorded as 0 sized file. It's unclear where the nested files end up.

This breaks installation of any mod that contains a file in archive format that should not be extracted.

Details

This was noticed in #2193, where a Credits.xlsx file (archive format containing Excel sheet) was missing from the App installation of a SDV collection.
Investigation revealed that the file was being recognized as an archive, and a 0 sized ghost entry was present rather than the full archive as expected.

Steps to reproduce

What is the expected behaviour?

Not sure yet.
The reason this happens in the first place is to support the installation of certain mods that come with nested archives that need to be extracted during installation. We want to have all the files available upfront, before installation.

Tentatively, my assumption would be that we would want to potentially index all the child files, but also index the parent archive.

This could have the very bad downside of the app backing up data twice, both the archive contents and the archive itself.
If we encounter an entire game where such archives are common (e.g. if this where to happen for something like BSA archives), that would be pretty disastrous space efficiency wise and have big negative impact on Download finalization time (overhead of extracting the inner data).

@Al12rs Al12rs added Bug Something isn't working Tech: data-store This is related to the Data Store. labels Dec 11, 2024
@Al12rs Al12rs added this to MVP Dec 11, 2024
@erri120
Copy link
Member

erri120 commented Dec 11, 2024

I think we can look into repacking if we need to deploy an archive as an archive.

@Al12rs
Copy link
Contributor Author

Al12rs commented Dec 11, 2024

I don't know if I would want to have something like a BSA extracted as loose files in our data store. Repacking on Apply seems quite a potential slowdown if you are dealing with a higher number of archives.

I think I would rather lean towards the opposite, of extracting during mod installation, for only those installers that require this.

@Al12rs
Copy link
Contributor Author

Al12rs commented Dec 11, 2024

See relevant comment:

I think the best way to fix this for now, and perhaps forever, is to have a specific list of archives we descend into. We should do this analysis for .zip, .rar, .7z, etc but not for .xslx files.

Sounds good, there are some games where mods are supposed to contain zip files, so we should store the archive in the filestore still, hopefully restricting the extraction by file extension limits the size overhead problem.

@erri120
Copy link
Member

erri120 commented Dec 12, 2024

I don't know if I would want to have something like a BSA extracted as loose files in our data store. Repacking on Apply seems quite a potential slowdown if you are dealing with a higher number of archives.

I think I would rather lean towards the opposite, of extracting during mod installation, for only those installers that require this.

The issue is that we index and extract archives when they get added to the app. We don't know how the files are going to be installed, since we're just adding the files to the app. As such, when we actually get to the installer we already have indexed, extracted, and backed up all files. The only thing the installer can do at that point is repack the files from the NX archive into whatever archive format we had originally.

@erri120
Copy link
Member

erri120 commented Dec 12, 2024

I think the best way to fix this for now, and perhaps forever, is to have a specific list of archives we descend into. We should do this analysis for .zip, .rar, .7z, etc but not for .xslx files.

.xlsx is an oddball, so let's use BSA files as an example instead. If we don't index those files, we'll never know what is in them, we won't able to provide diagnostics or anything else if we just skip them.

@Al12rs
Copy link
Contributor Author

Al12rs commented Dec 12, 2024

I think the best way to fix this for now, and perhaps forever, is to have a specific list of archives we descend into. We should do this analysis for .zip, .rar, .7z, etc but not for .xslx files.

.xlsx is an oddball, so let's use BSA files as an example instead. If we don't index those files, we'll never know what is in them, we won't able to provide diagnostics or anything else if we just skip them.

There is a difference from getting the list of files contained in BSA and extracting and backing up all the files contained in a BSA.

Doing the second would be big waste of time and space.

For BG3, I get the list of files in the pak files from the pak header, I don't extract the pak files and backup all the contents.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Tech: data-store This is related to the Data Store.
Projects
Status: No status
Development

No branches or pull requests

2 participants