Auto-install dependencies for build and deploy and rework for #174 #204

s3bc40 · 2025-02-10T10:44:28Z

Related issue: #174

This is my solution for the issue/feature. It is open to any feedback, to improve in my journey since I have done it with my current experience level. It might not be suited to the original demand, so I am not against closing this PR or improve it if needed!

Thanks for your time!

just test-all -> OK
live test -> OK

⚠️ Need to update the doc

Main Features Implemented: Automatic Dependency Management

Integrated automatic dependency installation when running build and deploy commands
System automatically checks and installs missing dependencies
Handles both PyPI packages and GitHub repositories

Key Components

Installation System (mox_install)

Central function for handling all installations
Used by build, deploy as requested
Supports both explicit installation requests and auto-installation

Dependency Utilities (_dependency_utils.py)

Smart dependency classification
Version tracking and management -> versions.toml inside each lib folders with Pypi SpecifierSet format
Handles both PyPI and GitHub package formats

Package Managers

PyPI: Uses uv package manager for Python packages -> added preprocessing with dep utilities (inspired from gihub implementation)
GitHub: Custom implementation for GitHub repository downloads -> reworked but kept Brownie logic.

Improvements proposed

Code Organization

Separated dependency utilities into a dedicated module
Clear separation between PyPI and GitHub package handling
Improved code documentation with Sphinx-style docstrings

Test cases

unit with install
integrations
zksync and live

Configuration Management

Version tracking in versions.toml

Potential next features/refacto

Auto-delete libs that are not in the config.dependencies anymore (removed from moccasin.toml)?
Improve test cases -> I think there is room to add better test case from what I did and also optimize them
Better code optimization from my PR -> maybe with more experienced Py dev to judge and improve my work
Possible edge-case where a user can add [package==0.0.5, package==0.1.0] when package==0.1.2 is already installed and triggers the uv pip error (yes the user has to intentionally add this error). But the fact is, maybe add a better filtering at first when we add conf_dependencies + requirements?

- added get types packages requirements - added get dependencies to install in mox install - wip detect updates or new installations

…lling - refacto pip and github install with dependency utils - getDependencies util and also get only the one needed to be installed or updated - run install only if they are new/to update pip or github packages - new constants added for test purposes - added one test for now to check if a classic install pass Need to up the tests and apply to compile and build cmd

still issue on updating package pip

PatrickAlphaC · 2025-02-11T00:59:50Z

oooooo exciting!! Give me a few days to review

s3bc40 · 2025-02-11T04:08:27Z

Yeah there are a lot of changes to take in. Feel free to ask if anything!

PatrickAlphaC

Love the direction, I've left a number of comments on how we can improve this, but this is going in the right direction!

moccasin/_dependency_utils.py

moccasin/commands/run.py

moccasin/commands/install.py

tests/cli/test_cli_compile.py

PatrickAlphaC · 2025-02-11T17:20:48Z

Potential next features/refacto
Auto-delete libs that are not in the config.dependencies anymore (removed from moccasin.toml)?

Let's not do this

Improve test cases -> I think there is room to add better test case from what I did and also optimize them

Agreed, but it's ok if we don't optimize them for now. I think we probably could add a test case about multiple installs at once? Maybe we have that and I missed it.

Better code optimization from my PR -> maybe with more experienced Py dev to judge and improve my work

Yeah, we should aim for this to be pretty performant. For example, we don't want to be making API calls to github if we don't have to, that can waste a LOT of time as a project gets bigger.

Possible edge-case where a user can add [package==0.0.5, package==0.1.0] when package==0.1.2 is already installed and triggers the uv pip error (yes the user has to intentionally add this error). But the fact is, maybe add a better filtering at first when we add conf_dependencies + requirements?

yes. If they have strict versioning, we should not update their packages. Or, if they have major version changes we should not auto-update. I left a comment about updates in general.

s3bc40 · 2025-02-11T20:57:57Z

Thanks for the feebacks and I'll respond to each demand 👍

I'll make some task on my own

tests/cli/test_cli_inspect.py

- output from logging install in other commands are force to quiet while install runs - helper created to for install process in other commands - fixed `RuntimeError: context has already been set` when running compile with install by moving multiprocessing set context as first process just test-all OK just format done

s3bc40 · 2025-02-13T11:11:31Z

feat: quiet output for other commands like compile, run `fd1d900`

output from logging install in other commands are force to quiet while install runs
helper created to for install process in other commands
fixed RuntimeError: context has already been set when running compile with install by moving multiprocessing set context as first process

just test-all OK
just format done

…cess - install command logs are quiet by default in other commands except `mox install` - `mox compile --no-install` - `mox compile` only run install package but no update - `mox compile --update-packages` will intall and check if updates are available - install code has been reworked to classify each package in upper level - conditions has been made to optimize installation process depending on option activated - used SpecifierSet from python packaging to check for status update pip package

s3bc40 · 2025-02-13T20:06:05Z

feat: options to command compile and run to optimize installation process `3235441`

install command logs are quiet by default in other commands except mox install
mox compile --no-install
mox compile only run install package but no update
mox compile --update-packages will intall and check if updates are available
install code has been reworked to classify each package in upper level
conditions has been made to optimize installation process depending on option activated
used SpecifierSet from python packaging to check for status update pip package

just test-all OK
format too

Note: did not do specific test for run with --update-packages but will come later

TODO

add test for run command
add install process for test and deploy
update doc
push tests with multiple successive install

- ``mox install`` added to ``deploy`` and ``test`` commands - helper available for rewriting moccasin dependencies while keeping config for tests - doc cli updated with ``--update-packages`` available - removed unecessary constant toml tests - fixed check mox install util just test-all OK format done

s3bc40 · 2025-02-14T11:15:44Z

feat: add install to deploy and test and improve tests `f875d24`

mox install added to deploy and test commands
helper available for rewriting moccasin dependencies while keeping config for tests
doc cli updated with --update-packages available
removed unecessary constant toml tests
fixed check mox install util

just test-all OK
format done

- reworked parsing - reworked write dependencies to mocassin toml - added new tests with ``@pytest.mark.parametrize`` just test-all OK and format

s3bc40 · 2025-02-14T17:40:47Z

refactor: parsing dep and writing new dependencies `1d6c3ba`

reworked parsing
reworked write dependencies to mocassin toml
added new tests with @pytest.mark.parametrize

just test-all OK and format

Review is possible @PatrickAlphaC 👍

s3bc40 added 8 commits February 6, 2025 16:17

wip: move main logic cli install in its own function for future export

6fa8b9e

wip: refactor mox install with new dependencu utils

8de7c0d

- added get types packages requirements - added get dependencies to install in mox install - wip detect updates or new installations

wip: currently testing struggling on conftest module

985fa87

wip: updated tests with few fixes in main install

217ab7d

still issue on updating package pip

wip: huge refacto nearly there still need to check test-i

e68a7a4

feat: auto install dep compile amd run with refacto done

a817ee5

docs: adding docs to the main files with new features

091c7ed

s3bc40 mentioned this pull request Feb 10, 2025

auto-install dependencies if not installed #174

Open

PatrickAlphaC requested changes Feb 11, 2025

View reviewed changes

moccasin/_dependency_utils.py Show resolved Hide resolved

moccasin/commands/run.py Outdated Show resolved Hide resolved

moccasin/commands/install.py Outdated Show resolved Hide resolved

tests/cli/test_cli_compile.py Show resolved Hide resolved

feat: changed func names and updated imports

aaccc63

s3bc40 commented Feb 12, 2025

View reviewed changes

tests/cli/test_cli_inspect.py Outdated Show resolved Hide resolved

s3bc40 and others added 2 commits February 13, 2025 09:29

fix: capture output revert to true avoid

1720780

refactor: parsing dep and writing new dependcies

1d6c3ba

- reworked parsing - reworked write dependencies to mocassin toml - added new tests with ``@pytest.mark.parametrize`` just test-all OK and format

s3bc40 requested a review from PatrickAlphaC February 14, 2025 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-install dependencies for build and deploy and rework for #174 #204

Auto-install dependencies for build and deploy and rework for #174 #204

s3bc40 commented Feb 10, 2025 •

edited

Loading

PatrickAlphaC commented Feb 11, 2025

s3bc40 commented Feb 11, 2025

PatrickAlphaC left a comment

PatrickAlphaC commented Feb 11, 2025

s3bc40 commented Feb 11, 2025

s3bc40 commented Feb 13, 2025

s3bc40 commented Feb 13, 2025 •

edited

Loading

s3bc40 commented Feb 14, 2025

s3bc40 commented Feb 14, 2025

Auto-install dependencies for build and deploy and rework for #174 #204

Are you sure you want to change the base?

Auto-install dependencies for build and deploy and rework for #174 #204

Conversation

s3bc40 commented Feb 10, 2025 • edited Loading

Main Features Implemented: Automatic Dependency Management

Key Components

Installation System (mox_install)

Dependency Utilities (_dependency_utils.py)

Package Managers

Improvements proposed

Code Organization

Test cases

Configuration Management

Potential next features/refacto

PatrickAlphaC commented Feb 11, 2025

s3bc40 commented Feb 11, 2025

PatrickAlphaC left a comment

Choose a reason for hiding this comment

PatrickAlphaC commented Feb 11, 2025

s3bc40 commented Feb 11, 2025

s3bc40 commented Feb 13, 2025

feat: quiet output for other commands like compile, run fd1d900

s3bc40 commented Feb 13, 2025 • edited Loading

feat: options to command compile and run to optimize installation process 3235441

TODO

s3bc40 commented Feb 14, 2025

feat: add install to deploy and test and improve tests f875d24

s3bc40 commented Feb 14, 2025

refactor: parsing dep and writing new dependencies 1d6c3ba

s3bc40 commented Feb 10, 2025 •

edited

Loading

feat: quiet output for other commands like compile, run `fd1d900`

s3bc40 commented Feb 13, 2025 •

edited

Loading

feat: options to command compile and run to optimize installation process `3235441`

feat: add install to deploy and test and improve tests `f875d24`

refactor: parsing dep and writing new dependencies `1d6c3ba`