Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to spack-stack 1.8.0 for running standalone #118

Open
malloryprow opened this issue Jan 11, 2024 · 23 comments
Open

Upgrade to spack-stack 1.8.0 for running standalone #118

malloryprow opened this issue Jan 11, 2024 · 23 comments
Assignees

Comments

@malloryprow
Copy link
Collaborator

EMC_verif-global standalone will update to using the spack-stack modules in the next spack-stack release (June-July timeframe).

@malloryprow malloryprow self-assigned this Jan 11, 2024
@DavidHuber-NOAA
Copy link
Contributor

FYI I tried giving this a shot for release 1.7.0, but cartopy was causing issues with the gsi-addon Python version. When the unified-env spack-stack environment upgrades to Python 3.11.x, we should be able to get the needed Python modules into the gsi-addon spack-stack environment. The target is the following release (1.8.0) in June-July. This work is being tracked in JCSDA/spack-stack#942.

@malloryprow
Copy link
Collaborator Author

Great, thank you! It will be great to use the spack-stack for running EMC_verif-global standalone! I'll update the title to reflect 1.8.0.

@malloryprow malloryprow changed the title Upgrade to spack-stack 1.?.? for running standalone Upgrade to spack-stack 1.8.0 for running standalone Mar 8, 2024
@InnocentSouopgui-NOAA
Copy link
Contributor

I successfully ran using spack-stack 1.6.0 on S4.
This requires two things:

  1. loading python packages (matplolib, panda, numpy, cartopy, etc. ) as modules;
  2. update the python code to change some features that were remove from matplolib in version 3.4
    • matplotlib.axes.SubplotBase.colNum -> ax.get_subplotspec().colspan.start
    • matplotlib.axes.SubplotBase.rowNum -> ax.get_subplotspec().rowspan.start
    • matplotlib.axes.SubplotBase.is_last_row() -> ax.get_subplotspec().is_last_row()
    • matplotlib.axes.SubplotBase.is_first_col() -> ax.get_subplotspec().is_first_col()

See the changes on my fork, branch migration/spack/stack
https://github.com/InnocentSouopgui-NOAA/EMC_verif-global/tree/migration/spack-stack

The change I made in the code is not backward compatible, however, it is easy to adjust it for backward compatibility with matplotlib 3.3 and below.

There is still a couple of things to fix

  • I ran a test where I get 751 plot files instead of 770 with the change to spack stack
  • some numbers (tick labels, etc.) are printed on way more digits, see the images below (one with spack sstack and the other with hpc-stack). Possibly something changed from using single precision to double precision. Those "ill"-formatted numbers appear on a handfull of plots here and there. I think it should be a fairly easy fix.

Plotted with spack stack (see the number on top of the top left panel)
model2model_capecin_CAPE_sfc_anl

Plotted with hpc-stack
model2model_capecin_CAPE_sfc_anl

We can also notice a big change in the data as presented: Look inside the South America for the bottom left panel of each of the plot; the difference is big. For the two bottom panel, there seems to be no negative value with HPC stack, while there are negative values with spack-stack.
Those are plotted from the same data, using the same version of MET and METplus, the major difference is in the external libraries. However, this one is an outlier, I did not look at all the images, but this is the only one that I saw with such a difference.

Another plot with ill-formatted numbers
model2model_sfc_GUST_sfc_d1

@malloryprow
Copy link
Collaborator Author

This is interesting, thanks for putting this out @InnocentSouopgui-NOAA. What if you do ncview on the two files? I guess I first want to make sure all is well with the MET and METplus installations. If it is the external libraries, I would suspect using ncview the files would look the same. If you need help investigating and there is a way to get this all on Hera, I can help investigate.

@InnocentSouopgui-NOAA
Copy link
Contributor

@malloryprow, I put the tes data on hera at /scratch2/NESDIS/nesdis-rdo1/Innocent.Souopgui/verif_global_input_data
I also did a quick check with neview and did not spot any obvious difference. I will take the time to rerun both tests and see.

@malloryprow
Copy link
Collaborator Author

Screenshot from 2024-06-10 10-39-54

Here is what I got running on Hera. It is using hpc-stack.

@InnocentSouopgui-NOAA
Copy link
Contributor

@malloryprow, that looks consistent with what I got using spack stack on S4.

@InnocentSouopgui-NOAA
Copy link
Contributor

I did a little more investigations,
I think that the version of the plotting library (matplotlib) has something to do with the resulting plot.
The plot with everything from spack stack is the correct one. See the screenshot below.
I wrote a simple python script to compute the difference T_ten - ctrl and save it back to netcdf. This is after verif_global run.
The left is from the spack stack run and the right is from hpc-stack run.
They look identical, with the negatives that we are seeing in the plot with external libraries from spack stack.

image

@InnocentSouopgui-NOAA
Copy link
Contributor

We might need to move quickly to spack stack.

@malloryprow
Copy link
Collaborator Author

Tank you for the investigation @InnocentSouopgui-NOAA!

@DavidHuber-NOAA Is there a status update for spack-stack v1.8.0?

@DavidHuber-NOAA
Copy link
Contributor

@malloryprow The tentative release date is the end of July/beginning of August. This is the issue that is tracking the Python upgrade: JCSDA/spack-stack#1038.

@malloryprow
Copy link
Collaborator Author

@InnocentSouopgui-NOAA Which version of spack stack were you using the produced the good plots? This was on S4 correct?

@InnocentSouopgui-NOAA
Copy link
Contributor

@InnocentSouopgui-NOAA Which version of spack stack were you using the produced the good plots? This was on S4 correct?

I used spack-stack 1.6.0 on S4.
You can see the module file https://github.com/InnocentSouopgui-NOAA/EMC_verif-global/blob/migration/spack-stack/modulefiles/emc_verif_global_s4.lua

@malloryprow
Copy link
Collaborator Author

Thanks! That is interesting. I was never able to get the standalone working with 1.6.0 on Hera. I had to us the gsi-addon because that is where MET and METplus were installed but then that version's python didn't have the needed python modules like cartopy to use.

@InnocentSouopgui-NOAA
Copy link
Contributor

InnocentSouopgui-NOAA commented Jun 13, 2024

I also built MET using spack-stack to be able to do that on S4. So I have MET built with spack-stack on s4 as well.

@InnocentSouopgui-NOAA
Copy link
Contributor

Also important to say that with spack-stack, python packages come as modules. So you have to load cartopy, matplotlib. numpy, etc. like other modules

module load py-netcdf4
module load py-numpy
module load py-pandas
module load py-matplotlib
module load py-cartopy

before it is available for use in python.

@DavidHuber-NOAA
Copy link
Contributor

I have a test installation of PR JCSDA/spack-stack#1252 on Hera. I will be trying it out today to run step2 plotting (g2g, g2o, and precip) as well as maps2d.

If you would like to play around with it, I believe this is the load sequence that will be needed:

module use /scratch1/NCEPDEV/global/David.Huber/SPACK/ss_python_3117/envs/met913_down/install/modulefiles/Core
module load stack-intel
module load stack-python
module load py-netcdf4
module load py-numpy
module load py-pandas
module load py-matplotlib
module load py-cartopy
module load wgrib2
module load met/9.1.3
module load metplus/3.1.1
module load prod-util
module load grib-util

@malloryprow
Copy link
Collaborator Author

Ah, that is great news, thanks!

@DavidHuber-NOAA
Copy link
Contributor

@malloryprow I have been able to run verif-global through the step1 jobs successfully, but the step2 jobs and maps2d are failing. Would you mind taking a look at my branch and seeing if I have something misconfigured? For instance, the log file /scratch1/NCEPDEV/global/David.Huber/para/stmp/verif_standalone/tmp/verif_global.2444030/grid2grid_step2/metplus_output/logs/master_metplus_grid2grid_step2_anom_plotbyVALID_SAL1L2_TM P_G002_runon20240820180947.log shows an error when attempting to calculate the axis. Perhaps I have bad data to work with?

My working directory is available here on Hera: /scratch1/NCEPDEV/global/David.Huber/VERIF/verif-global_ss180 and I created the following branch on GitHub that includes the configs: https://github.com/DavidHuber-NOAA/EMC_verif-global/tree/feature/ss_180.

@malloryprow
Copy link
Collaborator Author

I'll try to give it a look here today. I have some EVS PRs I need to work through that are my priority right now. I'll definitely look at it this week.

@DavidHuber-NOAA
Copy link
Contributor

Sounds good, thanks @malloryprow!

@malloryprow
Copy link
Collaborator Author

I ran a test for grid2grid step 2 and I'm seeing the same errors your seeing. I'm using different data though (operational GFS stat files).

@malloryprow
Copy link
Collaborator Author

Are we sure everything is okay with this python installation? I can even get a simple matplotlib plot to be created with it.

https://matplotlib.org/stable/gallery/lines_bars_and_markers/simple_plot.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants