Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add relative paths to metadata for post-install script execution #19

Open
ncoghlan opened this issue Oct 14, 2024 · 3 comments
Open

Add relative paths to metadata for post-install script execution #19

ncoghlan opened this issue Oct 14, 2024 · 3 comments
Labels
Affects: Metadata Affects the stack output metadata Category: Enhancement New feature or request

Comments

@ncoghlan
Copy link
Collaborator

ncoghlan commented Oct 14, 2024

Adding the local-export support showed that using the venv Python to run postinstall.py to regenerate pyvenv.cfg is unreliable on Windows (as whether or not it works depends on exactly how Python is set up in the Windows installation). It only works on other platforms because the Python "binary" in the venv is a symlink to the real binary in the base runtime environment.

The local-export feature works around that by using the already set up build environment to run the exported environment's postinstall.py.

test_minimal_project relies on the fact there's exactly one runtime environment defined in that test case, and falls back to using that if using the unpacked environment's own Python binary fails.

The archive and environment metadata should be updated to include three new fields, all relative to the installed environment's base folder rather than to the metadata file:

  • postinstall_script: path to post-installation script (usually just postinstall.py, but allows migration to a different name without breaking compatibility). Set for all layers.
  • base_python: path to the base runtime Python for that environment (for layered envs, ../<base_env_name>/python.exe on Windows, ../<base_env_name>/bin/python elsewhere; for base runtimes, python.exe on Windows, bin/python elsewhere). Set for all layers.
  • app_python: path to Python inside the env (Scripts/python.exe on Windows, bin/python elsewhere). Only set for application layers.

With those fields added, local environment exports and consuming applications can always use base_python and postinstall_script to finish the export process, while app_python can be used to run application launch modules.

@ncoghlan ncoghlan added Category: Enhancement New feature or request Affects: Metadata Affects the stack output metadata labels Oct 14, 2024
@ncoghlan ncoghlan transferred this issue from another repository Oct 18, 2024
@ncoghlan
Copy link
Collaborator Author

For ease of access, it may be worth saving a subset of the layer metadata to the share/venv/metadata folder proposed in #17.

Specifically, starting from unpacking the application layer archive, an installer should be able to work out:

  • which framework environments need to be available
  • which base runtime environment needs to be available
  • how to run the post install script
  • how to run the application launch module

Each framework environment would need to report:

  • which base runtime environment needs to be available
  • how to run the post install script

And each base runtime would need to report:

  • how to run the post install script

Suggested file name would be share/venv/metadata/venvstacks_layer.json

@ncoghlan
Copy link
Collaborator Author

ncoghlan commented Oct 24, 2024

Another enhancement this change would unblock is to make even layered environment archives fully compatible with the tar_filter extraction filter.

While #23 adds that for the base runtime archives, layered environments currently still rely on external symlinks to the underlying Python implementation.

If postinstall.py were to always be executed with the base runtime directly, then the external symlink could be omitted from the archve, and the app_python and base_python metadata fields could be used to recreate the missing symlink from the post-installation script.

Framework layers wouldn't need to recreate the symlink, since everything executed on a deployed system should be running in an application layer environment.

@ncoghlan
Copy link
Collaborator Author

ncoghlan commented Nov 4, 2024

The share/venv/metadata/venvstacks_layer.json part of this is being implemented to resolve #66

ncoghlan added a commit that referenced this issue Nov 5, 2024
* Allow postinstall scripts to be executed with
  any Python interpreter (not just the deployed
  base runtime interpreter)
* Generate layer config file as part of layers
* Use a common postinstall script in all layers
* Generate the deployed `sitecustomize.py` file
  from the layer config in the postinstall script
* Add unit tests for the common postinstall script

Closes #66. Implements initial steps towards #19.
ncoghlan added a commit that referenced this issue Nov 7, 2024
* Allow postinstall scripts to be executed with
  any Python interpreter (not just the deployed
  base runtime interpreter)
* Generate layer config file as part of layers
* Use a common postinstall script in all layers
* Generate the deployed `sitecustomize.py` file
  from the layer config in the postinstall script
* Add unit tests for the common postinstall script
* Add build env creation test cases (separate from
  the slow lock-and-publish/export test cases)

Closes #66. Implements initial steps towards #19.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Affects: Metadata Affects the stack output metadata Category: Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant