[INSTALL]: METcalcpy, METplotpy, METdataio #1112
Comments
|
Just checking in on this if there's any more action or information needed from our end for this installation request. We would like to have a test install ready on Hera (or any other UFS tier-1 platform aside from WCOSS) in the next few weeks so we can demonstrate at least a test capability by the end of our period of performance (end of June), is this a realistic timeline? |
|
FWIW it looks like there are not currently recipes for these three packages in Spack. Thankfully, they all have setup.py scripts, so creating recipes should be pretty straightforward. Since this is for SRW, can someone from EPIC take the lead on this? @ulmononian @natalie-perlin @RatkoVasic-NOAA ? |
Sure, since I never did that, I'm asking Rick Grubin to show me how to do it. |
|
I believe metcalpy is there - I just stumbled over it when working on an NRL plotting package. |
Which |
|
I mixed it up with metpy - my bad |
|
@AlexanderRichert-NOAA @RatkoVasic-NOAA Thanks for your work so far. Let me know if there's any help or info our team can provide to help the process along. |
|
@mkavulich must the components called out in respective
It seems 'yes' in that, for the simple example stated here, Thanks. |
|
@rickgrubin-noaa Our group doesn't maintain the MET code so I'm not sure, but I am pretty these are minimum versions, not hard requirements. Is the problem that this numpy version is in conflict with other python packages in spack-stack? |
Thanks; after consultation with |
|
@RatkoVasic-NOAA, @rickgrubin-noaa, @mkavulich, @climbfuji, just checking to see if there is an update on inclusion of METplotpy, METcalcpy, and METdataio into spack-stack. Thank you! |
|
The blocker is gone (update of versions in package.yaml), my understanding is that @rickgrubin-noaa is still working on this. Once we've got the updates in spack-stack develop, they will be slated for roll-out in spack-stakc-1.8.0 (around early September). |
Hi @JeffBeck-NOAA -- was away for ~3wks. Will be merging the latest changes to |
|
Thanks, @rickgrubin-noaa! |
|
We should target numpy 1.25.x if at all possible. That's the latest version that works for all our Python packages - 1.26 breaks many of them. |
|
On Requirements for metcalcpy: As configured, concretization settles on When attempting an install, Need assistance / discussion as to whether or not this work requires using the role account, and the accepted method(s) for explicitly requiring certain package versions / range of versions for requirements. |
|
Marking as blocked; versions for various concretize to a version that won't support minimum package version requirements for: metcalcpy: With the above concretization, |
|
Steps to demonstrate package dependencies are concretized to versions that are insufficient to meet requirements for metcalcpy / metdataio / metplotpy:
With the above concretization, I suspect that part of the problem is linked to: which seems to necessarily downgrade some package versions in order to be satisfied. File |
|
I see that this was removed from the list of spack-stack 1.8 features. Can we get an update on the status of this issue? If I am following correctly, the holdup is that there are other packages already in spack-stack that would need to be upgraded in order to accommodate the requirements of |
As described in the comment from August 6, there is at least this issue for attempting to integrate into
Further, the pinning of appears to necessarily downgrade some other package versions -- which then do not satisfy other package requirements -- in order for said pinned version's requirements to be met. Per today's biweekly |
|
@mkavulich This doesn't necessarily mean that you have to wait until the next release. We can provide a test stack on one platform after we figured out the dependencies, and if that works we can make addon environments to the existing 1.8.0 stack available on selected platforms (likely not all), |
|
With Created a
This yielded a concretized environment that successfully built However, it also creates a duplicate concretization, choosing The entire env successfully installs. It's not clear to me how to rectify this duplicate concretization -- suggestions / help appreciated, thanks. |
|
Spack allows duplicates for dependencies with type=build, even for unify:true. This happens a lot with py-setuptools and other packages. One solution might be to just ignore it, then blacklist the py-setuptools module (though this is still not an ideal solution as it can still lead to problems when reconcretizing an environment part way through installation). Are you able to pin a single version of py-setuptools? I'm a bit confused as to why it's allowing [email protected] to be concretized based on the version requirement... |
|
I misspoke above (fixed the comment, I apolgize for confusion). With For a simple environment for only Pinning Pinning Pinning Pinning Pinning Further, with this simplified This is the same as the However, as above, it also creates a duplicate concretization, choosing This confuses me; allowing the concretizer to choose package versions is OK, but then pinning packages to those versions is not OK. I'll compare concretization logs of the two scenarios and try to ascertain what's different. |
|
Following on to the previous comment: Pinning Can't seem to come up with anything that doesn't have duplicate concretizations for |
|
The culprit is probably [email protected]. If you look at the package definition, it limits setuptools to @:63. Try to use [email protected] (and remove the pinning of py-shapely entirely - 1.8.0 is ancient). |
|
|
|
I was able to build a decent stack on my dev system (Oracle Linux 9) with gcc@13. It's just a suggestion to see if it helps your problem. |
|
That said, with: https://github.com/rickgrubin-noaa/spack-stack/tree/SI-1052 which sets up the versions noted above, and https://github.com/rickgrubin-noaa/spack/tree/SI-1052 which contains
envs for
whereas for
fails to install Not sure where to start! I'm new to creating |
|
does metdataio have py-setuptools listed as a build-time dependency? |
|
@mkavulich -- a test env can be built for you; questions, please: is hera an acceptable host? Thanks. |
@mkavulich -- how would you like to proceed? |
|
@rickgrubin-noaa Sorry for the delay in getting back to you. Hera is acceptable for testing. I am not sure about the differences between environments; is |
@mkavulich yes -- |
|
In that case then unified-dev sounds correct. Thanks! |
Package name
METcalcpy, METplotpy, METdataio
Package version/tag
v2.1
Build options
none
Installation timeframe
Please install on Hera for testing, and then include in the next release.
Other information
The DTC Agile Framework group is seeking to add plotting of verification statistics to the Short-Range Weather App workflow. This work relies on the three packages of the METplus Analysis Suite: METcalcpy, METplotpy, and METdataio. The latest versions of all these tools are v2.1, as they are released on the same cadence as all MET products.
We do not need these tools installed on WCOSS2, though as I understand it they have already been installed independently for use by EMC, so they should pass all necessary checks if that was desired eventually.
WCOSS2
WCOSS2: General questions
No response
WCOSS2: Installation and testing
No response
WCOSS2: Technical & security review list
WCOSS2: Additional comments
No response
The text was updated successfully, but these errors were encountered: