Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Pyodide lockFileURL + packages when a cache exists #121

Merged
merged 1 commit into from
Oct 2, 2024

Conversation

WebReflection
Copy link
Contributor

This MR introduces 2 loadPyodide extra features:

  • lockFileURL, created at runtime when a valid JSON content for specific packages is found in the IndexedDB
  • packages, used instead of manual micropip installation whenever the cache is already known

With the current tests and just numpy as foreign package the bootstrap went down 1 second on average so I expect even more improvements with more convoluted scenarios.

About packages cache

Only packages present in the py-config are considered, as runtime packages might be both heavy or undesired.

If a package is always added at runtime it should be moved into the packages field.

The order, version, amount of packages decides the the cache key, as these are serialized as JSON and then used as unique identifier for the cache.

Cleanup the whole cache

As projexts might change and grow in time, or even remove dependencies, it is possible to force-erase the whole cache for any specific Pyodide version by using packages_cache = "never" in the TOML / JSON config file.

This could also be an easy way to just clean the cache once and then recreate it, otherwise users would need to manually reach their devtool Application panel, dig the IndexedDB related folder and erase previous cache manually.

Each DB is based on the current version of Pyodide so that multiple versions cna use different caches from different times.

Note about mocked tests

Unfortunately it's extremely hard, but so (imho) pointless, to mimic this Pyodide only behavior so while I am sad to temporarily drop current unit tests in favor of integration manual tests, I am sure once this whole project will be refactored into PyScript core itself we'll have all tests that do all the things we expect to validate things just work as expected in there too.

@WebReflection WebReflection force-pushed the pyodide-packages-cache branch from f2e5aa6 to f5eae45 Compare October 2, 2024 13:25
@WebReflection WebReflection force-pushed the pyodide-packages-cache branch from f5eae45 to 6df90d7 Compare October 2, 2024 13:58
@WebReflection WebReflection merged commit 4edd654 into main Oct 2, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants