[Feature Request] Async options for UnloadDirectory

[Malkierian]

The file listing in the Directory commands is very process intensive, and "unloading" assets should never have to block the game loops, so an option to have that happen on a separate thread rather than on the main one would be nice to have.

[briaguya-ai]

i'm curious about the

file listing in the Directory commands is very process intensive

part of this

in LoadDirectoryAsync the file listing part isn't async

libultraship/src/resource/ResourceManager.cpp

Lines 226 to 239 in 99d78d6

	std::shared_ptr<std::vector<std::shared_future<std::shared_ptr<IResource>>>>
	ResourceManager::LoadDirectoryAsync(const std::string& searchMask, uintptr_t owner, BS::priority_t priority) {
	auto loadedList = std::make_shared<std::vector<std::shared_future<std::shared_ptr<IResource>>>>();
	auto fileList = GetArchiveManager()->ListFiles(searchMask);
	loadedList->reserve(fileList->size());
	for (size_t i = 0; i < fileList->size(); i++) {
	auto fileName = std::string(fileList->operator[](i));
	auto future = LoadResourceAsync(fileName, owner, false, priority);
	loadedList->push_back(future);
	}
	return loadedList;
	}

so the following that pattern to "make unloading async" (just adding unloads to a queue instead of unloading on the same thread) wouldn't address the file listing issue

it's also probably worth looking into ways to make the ListFiles more performant as well

getting some measurements on

libultraship/src/resource/archive/ArchiveManager.cpp

Lines 79 to 85 in 99d78d6

	std::shared_ptr<std::vector<std::string>> ArchiveManager::ListFiles() {
	auto list = std::make_shared<std::vector<std::string>>();
	for (const auto& [hash, path] : mHashes) {
	list->push_back(path);
	}
	return list;
	}

and

libultraship/src/resource/archive/ArchiveManager.cpp

Lines 69 to 77 in 99d78d6

	std::shared_ptr<std::vector<std::string>> ArchiveManager::ListFiles(const std::string& filter) {
	auto list = ListFiles();
	auto result = std::make_shared<std::vector<std::string>>();
	std::copy_if(list->begin(), list->end(), std::back_inserter(*result),
	[filter](const std::string& filePath) { return glob_match(filter.c_str(), filePath.c_str()); });
	return result;
	}

would be good

if the performance issue is only in the version with a filter, we could look into adding (less robust but more performant) alternatives to glob (after reading https://research.swtch.com/glob i don't think we can get a much faster glob implementation than https://github.com/Kenix3/libultraship/blob/99d78d61380048bbcec2f8364f5d876681fd982e/src/utils/glob.c)

[Malkierian]

I think the issue is much more prevalent in the filter version, yes.