Add export command #24
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add command 'jmm export <reference>' to enable exporting contents of a built model to disk. The export command takes an optional -d|--dir parameter to optionally export into a specific directory (by default, the current directory is used).
* Add additional validation to paths to ensure layer paths are subdirectories of the overall context * Fix process of tarring layer directories to include directory entries in the tarball * Minor cleanup of small inconsistencies
Add flags --models --config --code --datasets to control which elements of the package are exported. If none are specified, the whole model is exported by default; otherwise, only those components are exported. For example, jmm export <ref> --models --config will only export the trained models and configuration file, ignoring datasets and code.
Enable jmm export to export artifacts directly from remote registries, without needing to pull the full artifact into local storage. This allows e.g. exporting the config file for a model without downloading the actual parameters.
gorkem
requested changes
Feb 21, 2024
Add flag --overwrite to the export command; if set, overwrite will overwrite existing directories and files while exporting.
Verify paths when exporting models to ensure everything is exported into a subdirectory of the export directory.
gorkem
approved these changes
Feb 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add
exportcommand to the CLI, allowing users to extract resources packaged into a model. Theexportcommand supports exporting from remote registries as well as local ones, without needing to pull the model into local storage first.Flags:
-d|--dir- export to a different directory (default is current dir)--http- If communicating with a registry, disable TLS (useful for testing against locally-running registries)--config,--models,--code,--datasets- export only some parts of the model (config, trained models, code, or datasets). Can be combined (e.g.--config --modelsto export both config and models, but not code/datasets)In addition, this PR fixes some minor bugs in the
buildcommand: previously the tarball we were building did not include directory entries and the prefixes for paths were inconsistent. Also we add some validation to try ensure file paths stay within the specified context when building.Currently, exporting via direct reference to a hash does not work; we need to figure out how to handle tagging first.