Use of the NWP container

[jwraby]

I have output produced by WRF V4.1 which are in sub-hourly netcdf files (wrfout) on the HPC. Would I be able to post-process them using the container? I have previously used the MET and METplus containers on the HPC using Singularity. If I produced the post-processed output would I be able to use the container to run the MET tools and produce the expected output? Is there User's Guide to describe the details of these tasks? Thank you.

Today (26 JUL 2023) I tried testing the NWP container for UPP to see how it works. I pulled the latest version of the UPP container (V4.1) and installed it as a Singularity container on the HPC. When I try to configure it for running the UPP I am denied permission due to a read-only file system. An example of this is the inability to create the postprd and wrfprd directories. I am also prevented from editing the run_unipost script to configure and run the UPP. I can't copy the WRF output from outside the container into it due to the same problem. Is there a solution for this?

[JohnHalleyGotway]

Hi @jwraby, sorry for letting this discussion languish for so long!

While the containers website and online tutorial are still available and this GitHub discussions board remains open, unfortunately the funding for this project has ended. So I won't be able to dive into the details very much.

I will say that yes, we have definitely run into permissions issues in the past when using containers. And I can't comment on what impact translating from Docker to Singularity would have on the permissions issue.

If I were you, I would try running though the exact commands for the Sandy case. Start by confirming that you're able to run WPS, WRF, and UPP with Singularity using the commands exactly as written in the tutorial. And if that's successful, try swapping in your WRF output in place of the WRF output for sandy. There's just so many things that can go wrong here... directory structure, permissions, volume mounts... I would try making as incremental of changes as possible.

If you have questions about using the MET or METplus containers, please feel free to post to METplus Discussions.

[jwraby]

No problem, John. Always glad to get your input. It's kind of an ongoing project for me to dabble in the NWP containers so any feedback is helpful and the delay isn't impacting me. I recently got the UPP running from source code on the HPC so I am back in business. I'm curious as to how to approach getting the containers to work given that the MET and METplus containers work so well. When I have time I will revisit using the Sandy case as a way to gain understanding about how the containers work. Thanks for the suggestion and feedback. I know that the HPC system admin folk are working on an issue which recently happened on our new HPC which is "INFO: fuse2fs not found, will not be able to mount EXT3 filesystems" They are in the process of getting the Singularity and its replacement software "Apptainer" running on this new HPC and they are working on this issue. Is this issue something like what you were referring to? I saw the email yesterday announcing the new METplus release. There was a statement "METviewer: New Dockerfile that creates a single image with database and METviewer in it. Can be used to create a Singularity image" which implied that one could pull/convert the container to Singularity and then run it on the HPC. But I read further down and it said "NOTE: METviewer and METexpress are intended to be installed on your local server." So can the METviewer container be run on an HPC? Thanks. R/ John From: John Halley Gotway ***@***.***> Sent: Thursday, August 3, 2023 1:08 PM To: NCAR/container-dtc-nwp ***@***.***> Cc: Raby, John W CIV USARMY DEVCOM ARL (USA) ***@***.***>; Mention ***@***.***> Subject: [URL Verdict: Neutral][Non-DoD Source] Re: [NCAR/container-dtc-nwp] Use of the NWP container (Discussion #103) Hi @jwraby <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.c om%2Fjwraby&data=05%7C01%7Cjohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637c a08db94552663%7Cfae6d70f954b481192b60530d6f84c43%7C0%7C0%7C63826686568430388 9%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW wiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=e3GaDK6ceFGu%2FHe2aVOdJEY54HKfTrwn7Yh Qt7rqiGo%3D&reserved=0> , sorry for letting this discussion languish for so long! While the containers website <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fdtcenter .org%2Fcommunity-code%2Fnumerical-weather-prediction-nwp-containers&data=05% 7C01%7Cjohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637ca08db94552663%7Cfae6 d70f954b481192b60530d6f84c43%7C0%7C0%7C638266865684303889%7CUnknown%7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 000%7C%7C%7C&sdata=Tt3T7tPQaycQWCh2tghRolWTGk3F%2FySsX1nry%2B6U3pE%3D&reserv ed=0> and online tutorial <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fdtcenter .org%2Fnwp-containers-online-tutorial&data=05%7C01%7Cjohn.w.raby2.civ%40army .mil%7C2ce2f4e26f7b486637ca08db94552663%7Cfae6d70f954b481192b60530d6f84c43%7 C0%7C0%7C638266865684303889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=032iWrbOokZ 9Dw%2FiNYZGCYC4MwXwr1sUkuFh62yenHA%3D&reserved=0> are still available and this GitHub discussions board remains open, unfortunately the funding for this project has ended. So I won't be able to dive into the details very much. I will say that yes, we have definitely run into permissions issues in the past when using containers. And I can't comment on what impact translating from Docker to Singularity would have on the permissions issue. If I were you, I would try running though the exact commands for the Sandy case <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fdtcenter .org%2Fnwp-containers-online-tutorial%2Fhurricane-sandy-case-27-oct-2012&dat a=05%7C01%7Cjohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637ca08db94552663%7 Cfae6d70f954b481192b60530d6f84c43%7C0%7C0%7C638266865684303889%7CUnknown%7CT WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3 D%7C3000%7C%7C%7C&sdata=xbHdg9oZhb6lBVDTp7XAtXPKTt9cB%2F73o8xFAiH6zKU%3D&res erved=0> . Start by confirming that you're able to run WPS, WRF, and UPP with Singularity using the commands exactly as written in the tutorial. And if that's successful, try swapping in your WRF output in place of the WRF output for sandy. There's just so many things that can go wrong here... directory structure, permissions, volume mounts... I try making as incremental of changes as possible. If you have questions about using the MET <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fhub.dock er.com%2Frepository%2Fdocker%2Fdtcenter%2Fmet%2Fgeneral&data=05%7C01%7Cjohn. w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637ca08db94552663%7Cfae6d70f954b4811 92b60530d6f84c43%7C0%7C0%7C638266865684303889%7CUnknown%7CTWFpbGZsb3d8eyJWIj oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C &sdata=h35bowSdBkFWBcSPbxBQ27D0hSQ8NQgHhDDLeQCaRqU%3D&reserved=0> or METplus <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fhub.dock er.com%2Frepository%2Fdocker%2Fdtcenter%2Fmetplus%2Fgeneral&data=05%7C01%7Cj ohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637ca08db94552663%7Cfae6d70f954b 481192b60530d6f84c43%7C0%7C0%7C638266865684303889%7CUnknown%7CTWFpbGZsb3d8ey JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 C%7C&sdata=IinjGP%2FDamAtOc6Ab3djVqRDMJ%2BNnGU12KlVr0dqmTw%3D&reserved=0> containers, please feel free to post to METplus Discussions <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.c om%2Fdtcenter%2FMETplus%2Fdiscussions&data=05%7C01%7Cjohn.w.raby2.civ%40army .mil%7C2ce2f4e26f7b486637ca08db94552663%7Cfae6d70f954b481192b60530d6f84c43%7 C0%7C0%7C638266865684303889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=60GKffY8zpk Lweu%2FUyhBu2zXlJ0zKgkOo%2FkeRP96koQ%3D&reserved=0> . - Reply to this email directly, view it on GitHub <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.c om%2FNCAR%2Fcontainer-dtc-nwp%2Fdiscussions%2F103%23discussioncomment-663109 8&data=05%7C01%7Cjohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b486637ca08db94552 663%7Cfae6d70f954b481192b60530d6f84c43%7C0%7C0%7C638266865684303889%7CUnknow n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%3D%7C3000%7C%7C%7C&sdata=dMXwdBytSp3TzX%2B7qtiEMyAZoTW%2FLGq2Ow9VeDSl1D0 %3D&reserved=0> , or unsubscribe <https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.c om%2Fnotifications%2Funsubscribe-auth%2FAUX5FH6MTKTKPWPK4DKMTLTXTPZJRANCNFSM 6AAAAAA2R57KL4&data=05%7C01%7Cjohn.w.raby2.civ%40army.mil%7C2ce2f4e26f7b4866 37ca08db94552663%7Cfae6d70f954b481192b60530d6f84c43%7C0%7C0%7C63826686568430 3889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1 haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LZvJrzH0WHbpcSpt04RBuZxbYhuVoSSO0Q 8JD4h3%2BdM%3D&reserved=0> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AUX5FH3MJFNU3TMIKT4T423XTPZJRA5CNFS M6AAAAAA2R57KL6WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3T UL5UWJTQAMUXLU.gif> Message ID: ***@***.*** ***@***.*** com> >