v0.8.0 #723
Merged
v0.8.0 #723
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR brings develop up to date with master for release.
This PR udpates tutorials to the new SmartSim and SmartRedis APIs. In addition, the tutorial containers' Docker files were also updated, to prevent an error caused by IPython. [ committed by @al-rigazzi ] [ reviewed by @amandarichardsonn ]
Fix an error that would prevent ``smart build`` from moving a successfully compiled RedisAI shared object to the install location expected by SmartSim if no ML backend installations were found. The reason is that after RedisAI is built using the `smart` tool, the resulting library is only installed to the `lib` folder if and only if the folder `backends/` exists, which it does not if no backends are installed. Since after this step the original build folder is deleted and with it the compiled library. This problem does not occur if any of the backends (TF, PT, ONNX) is installed. However, since they are not needed for many applications it would additional complications and effort to compile them if not necessary. Also compiling RedisAI by itself on pointing `RAI_PATH` to the installation also works, but poses additional effort. To circumvent this problem this change will install the RedisAI library by itself if it was built. [ committed by @m-kurz ] [ reviewed by @MattToast ]
Fix unit tests that omit an experiment path and write to the root directory of the project. _Secondary Changes_ - Added typehints to enable successfully passing `mypy test_manifest.py` - Refactored test using global variables in pytest parameterization to use a fixture [ committed by @ankona ] [ reviewed by @MattToast ]
SGE shares some similarities to PBS/Torque-like launchers, but the differences are significant enough to warrant their own separate implementations. Notably, SGE has a qacct utility (similar to SLURM's sacct) to query for the historical record of a job. Additionally, unique amongst the launchers, SGE does not allow a way for a user to specify the number of nodes needed and requires the user to select a admin-configured parallel environment profile (e.g. mpi or smp). The changes add a new SGE launcher and SGEQsubBatchSettings to enable support for SmartSim applications on SGE machines. [ committed by @ashao ] [ reviewed by @al-rigazzi ]
The new major version release of Numpy is incompatible with modules compiled against Numpy 1.x. For both SmartSim and SmartRedis we request a 1.x version of numpy. This is needed in SmartSim because some of the downstream dependencies request NumPy. [ committed by @ashao ] [ reviewed by @al-rigazzi ]
This PR will remove a broken link to oss.redis.com that is breaking the documentation build [ committed by @ankona ] [ approved by @MattToast ]
Adds the ability to specify hardware affinities for cpu/gpu devices. Creates a dragon policy that uses provided policy to modify the resulting dragon ProcessGroup. [ committed by @ankona ] [ approved by @AlyssaCote @mellis13 @al-rigazzi ]
setup.py used to define dependencies in a way that was not amenable to code scanning tools. Direct dependencies now appear directly in the setup call and the definition of the SmartRedis version has been removed. Additionally, the code scanning tool was failing to detect some of the dependencies due to the existence of the requirements-doc.txt file. These requirements are now listed in the `docs` extra. [ committed by @ashao ] [ reviewed by @ankona ]
This PR addresses an inconstent internal host name representation in the Dragon backend. [ committed by @al-rigazzi ] [ reviewed by @ankona ]
This PR mitigates two issues encountered during installation on build agents ## mypy/typing_extensions Installation of mypy or dragon in separate build actions caused some dependencies (typing_extensions, numpy) to be upgraded. Those upgrades result in runtime failures. The build actions were tweaked to allow pip to consider all optional dependencies during resolution. ## dragon/numpy Additionally, the numpy version was capped on dragon installations. [ committed by @ankona] [ approved by @ashao @MattToast ]
The builder module was included in `setup.py` to allow us to ship the main Redis binaries (not RedisAI) with installs from PyPI. The changes in this PR remove our ability to do this and requires users to build Redis as part of the `smart build`. This change in behaviour was deemed reasonable to allow for easier maintenance and extension of the Builder class as well as simplify the deployment of wheels. [ committed by @ashao ] [ reviewed by @MattToast ]
The version of codecov has been updated to v4.5.0 for the github actions. [ committed by @mellis13 ] [ reviewed by @amandarichardsonn ]
The release of watchdog v5 introduced new types which caused further errors with mypy. To mitigate these errors for now, we pin the watchdog version to 4.x and will resolve these errors in the future. [ committed by @ashao ] [ reviewed by @al-rigazzi ]
Allow specifying Model and Ensemble parameters with number-like types. The constructors for parameters on Model and Ensemble now validate that the input is number-like and convert them to strings. [ committed by @juliaputko ] [ reviewed by @ashao]
- The RedisAIBuilder class was completely overhauled to allow users to express a wider range of support for hardware/software stacks. This will be extended to support ROCm, CUDA-11, and CUDA-12. - Versions for each of these packages are no longer specified in an internal class. Instead a default set of JSON files specifies the sources and versions. Users can specify their own custom specifications at smart build time --------- [ committed by @ashao ] [ reviewed by @MattToast @juliaputko ] Co-authored-by: Matt Drozt <[email protected]> Co-authored-by: Julia Putko <[email protected]>
After discussing with admins at OLCF, miniforge is the preferred solution for creating virtual environments on Frontier. The instructions for installing SmartSim have been updated accordingly. Additionally, perlmutter did not have a step for compiling the SmartRedis libraries. This has been rectified to bring the two systems to parity. [ committed by @ashao ] [ reviewed by @MattToast @AlyssaCote ]
On Frontier, the recommended way to activate conda environments is to go through source activate. This also means that ``conda init`` is not needed. The instructions for Frontier have been updated to reflect this. [ committed by @ashao ] [ reviewed by @MattToast ]
Bump the version number for the release, last minute actions and docs fixes [ committed by @MattToast ] [ reviewed by @ashao ]
MattToast
added
area: release
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge develop to master for release