Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a dockerfile to build CAM-SIMA #222

Merged
merged 17 commits into from
Aug 3, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# ignore all
*

# include things to copy
!Externals*
!src/
!cime_config/
!manage_externals/
!test/
!.config_files.xml
!docker
85 changes: 85 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# parts of CAM require x86 architecture (gptl, which relies on the rdtsc x86 assembly instruction)
# esmf is am image you are expected to have built. Read the README file for instructions
FROM esmf:latest

###################################################
## Install necessary packages
###################################################
RUN dnf -y update \
&& dnf -y install \
ftp \
git \
hostname \
m4 \
python \
sudo \
svn \
tree \
vim \
&& dnf clean all

###################################################
## Make sure the mpi compilers can be found
###################################################
ENV PATH="${PATH}:/usr/lib64/mpich/bin/"
ENV OMP_NUM_THREADS=5

###################################################
## Build and install Parallel-netcdf
###################################################
RUN wget -q https://parallel-netcdf.github.io/Release/pnetcdf-1.12.3.tar.gz
RUN tar zxf pnetcdf-1.12.3.tar.gz
RUN cd pnetcdf-1.12.3 && \
./configure --prefix=/usr/local && \
make -j 8 install && \
ldconfig

###################################################
## Build CAM-SIMA
###################################################

# create a user to run the case
RUN adduser cam_sima_user \
&& echo "cam_sima_user ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/cam_sima_user \
&& chmod 0440 /etc/sudoers.d/cam_sima_user

# copy in the CAM-SIMA code and give the proper user permissions
COPY --chown=cam_sima_user . /home/cam_sima_user/CAM-SIMA

USER cam_sima_user
WORKDIR /home/cam_sima_user/CAM-SIMA

# pull the dependencies
RUN ./manage_externals/checkout_externals

# Copy in the machine information for the container
RUN cp /home/cam_sima_user/CAM-SIMA/docker/config_machines.xml /home/cam_sima_user/CAM-SIMA/ccs_config/machines/

# Set environment variables needed to create and build the case
ENV USER=$(whoami)
ENV CASE_NAME=/home/cam_sima_user/case_name
ENV CESMDATAROOT=/home/cam_sima_user/cesm_data
ENV CIME_MACHINE=container
ENV CIME_MODEL=cesm
ENV ESMFMKFILE=/usr/local/lib/esmf.mk

# Create a case
RUN ./cime/scripts/create_newcase --case $CASE_NAME --compset FPHYStest --res ne5_ne5_mg37 --run-unsupported

WORKDIR $CASE_NAME

RUN ./case.setup

RUN ./xmlchange CAM_CONFIG_OPTS="--dyn none --physics-suites held_suarez_1994"
RUN ./xmlchange ROF_NCPL=48
RUN ./xmlchange STOP_OPTION=nsteps
RUN ./xmlchange STOP_N=5

# Copy in the grid files and a snapshot file
RUN chmod +x /home/cam_sima_user/CAM-SIMA/docker/ftp_download.sh
RUN /home/cam_sima_user/CAM-SIMA/docker/ftp_download.sh

# # add the snapshot file
RUN echo "ncdata='/home/cam_sima_user/run_heldsuarez_cam6_nt2_bigg_try005.cam.h5.0001-01-01-00000.nc'" >> user_nl_cam

RUN ./case.build
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,22 @@ git checkout development

Good luck, and have a great day!

## Build the docker file
1. First, build the esmf docker image. You must tag the build with `esmf`
```
cd docker
docker build -f Dockerfile.esmf -t esmf .
cacraigucar marked this conversation as resolved.
Show resolved Hide resolved
cacraigucar marked this conversation as resolved.
Show resolved Hide resolved
```
2. Build the CAM-SIMA iamge
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo - image

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

```
cd ../
docker build -t cam-sima .
```
3. Run the image
```
docker run --rm -it cam-sima
```
4. Submit the case
```
./case.submit
```
54 changes: 54 additions & 0 deletions docker/Dockerfile.esmf
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# parts of CAM require x86 architecture (gptl, which relies on the rdtsc x86 assembly instruction)
FROM --platform=linux/amd64 fedora:latest

###################################################
## Install necessary packages
###################################################
RUN dnf -y update \
&& dnf -y install \
blas-devel \
cmake \
gcc-c++ \
gcc-gfortran \
lapack-devel \
netcdf-devel \
netcdf-fortran-devel \
make \
mpich \
mpich-devel \
wget \
&& dnf clean all

###################################################
## Make sure the mpi compilers can be found
###################################################
ENV PATH="${PATH}:/usr/lib64/mpich/bin/"
ENV OMP_NUM_THREADS=5

###################################################
## Build and install ESMF
###################################################

ENV ESMF_TAG="8.4.2"

# set necessary environment variables
ENV ESMF_DIR=/esmf-${ESMF_TAG}
ENV ESMF_COMM=mpich3
ENV ESMF_BOPT="g"
ENV ESMF_NETCDF="nc-config"
ENV ESMF_NETCDFF_INCLUDE=/usr/lib64/gfortran/modules
ENV ESMF_INSTALL_PREFIX=/usr/local
ENV ESMF_INSTALL_BINDIR=${ESMF_INSTALL_PREFIX}/bin
ENV ESMF_INSTALL_DOCDIR=${ESMF_INSTALL_PREFIX}/doc
ENV ESMF_INSTALL_HEADERDIR=${ESMF_INSTALL_PREFIX}/include
ENV ESMF_INSTALL_LIBDIR=${ESMF_INSTALL_PREFIX}/lib
ENV ESMF_INSTALL_MODDIR=${ESMF_INSTALL_PREFIX}/mod
ENV ESMF_TESTEXHAUSTIVE="OFF"

RUN wget -q https://github.com/esmf-org/esmf/archive/refs/tags/v${ESMF_TAG}.tar.gz && \
tar zxf v${ESMF_TAG}.tar.gz && \
cd esmf-${ESMF_TAG} && \
# This command lets you see what esmf thinks its build options are but may not necessary to build, not sure
make info && \
make -j 8 && \
make install
95 changes: 95 additions & 0 deletions docker/config_machines.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
<?xml version="1.0"?>

<!--

===============================================================
COMPILER and COMPILERS
===============================================================
If a machine supports multiple compilers - then
- the settings for COMPILERS should reflect the supported compilers
as a comma separated string
- the setting for COMPILER should be the default compiler
(which is one of the values in COMPILERS)

===============================================================
MPILIB and MPILIBS
===============================================================
If a machine supports only one MPILIB is supported - then
the setting for MPILIB and MPILIBS should be blank ("")
If a machine supports multiple mpi libraries (e.g. mpich and openmpi)
- the settings for MPILIBS should reflect the supported mpi libraries
as a comma separated string

The default settings for COMPILERS and MPILIBS is blank (in config_machines.xml)

Normally variable substitutions are not made until the case scripts are run, however variables
of the form $ENV{VARIABLE_NAME} are substituted in create_newcase from the environment
variable of the same name if it exists.

===============================================================
PROJECT_REQUIRED
===============================================================
A machine may need the PROJECT xml variable to be defined either because it is
used in some paths, or because it is used to give an account number in the job
submission script. If either of these are the case, then PROJECT_REQUIRED
should be set to TRUE for the given machine.


mpirun: the mpirun command that will be used to actually launch the model.
The attributes used to choose the mpirun command are:

mpilib: can either be 'default' the name of an mpi library, or a compiler name so one can choose the mpirun
based on the mpi library in use.

the 'executable' tag must have arguments required for the chosen mpirun, as well as the executable name.

unit_testing: can be 'true' or 'false'.
This allows using a different mpirun command to launch unit tests

-->

<config_machines version="2.0">

<machine MACH="container">
<DESC>
Containerized development environment (Docker/Singularity) for CESM w/ GNU compilers
</DESC>
<OS>LINUX</OS>
<COMPILERS>gnu</COMPILERS>
<MPILIBS>mpich</MPILIBS>
<CIME_OUTPUT_ROOT>$ENV{HOME}/scratch</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>$DIN_LOC_ROOT/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$ENV{HOME}/archive/$CASE</DOUT_S_ROOT>
<GMAKE>make</GMAKE>
<GMAKE_J>4</GMAKE_J>
<BATCH_SYSTEM>none</BATCH_SYSTEM>
<SUPPORTED_BY>cgd</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>4</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>4</MAX_MPITASKS_PER_NODE>
<PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
<mpirun mpilib="mpich">
<executable>mpiexec</executable>
<arguments>
<arg name="anum_tasks"> -n {{ total_tasks }}</arg>
</arguments>
</mpirun>
<module_system type="none">
</module_system>
<environment_variables>
<env name="NETCDF_PATH">/usr/local</env>
<env name="PNETCDF_PATH">/usr/local</env>
<env name="FPATH">/usr/lib64</env>
<env name="CPATH">/usr/lib64</env>
</environment_variables>
<resource_limits>
<resource name="RLIMIT_STACK">-1</resource>
</resource_limits>
</machine>

<default_run_suffix>
<default_run_exe>${EXEROOT}/cesm.exe </default_run_exe>
<default_run_misc_suffix> >> cesm.log.$LID 2>&amp;1 </default_run_misc_suffix>
</default_run_suffix>

</config_machines>
19 changes: 19 additions & 0 deletions docker/ftp_download.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/sh

FTP_HOST="cesm-inputdata-lowres1.cgd.ucar.edu"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It this a good site to have the ftp data reside? Two concerns:

  1. Typically the CGD ftp site has an expiration data available on it. I'm not sure if that policy is for this site as well.
  2. With the threat of the removal of izumi in the future, I'm not sure it that will expand to wherever this site is located as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a temporary solution until we start using snapshot files with the ne3 grid that have been processed to be as small as possible, which we will then bring in using the standard inputdata server methods.

The reason we haven't done that already is simply because there isn't an agreed-upon plan (yet) for where to put test data on the inputdata server, and because these files are larger than they need to be and so we didn't want to add them to a place where they couldn't be deleted but would stop being used in a few months.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nusbaume or @briandobbins will better know how to answer this questions.

FTP_USER="anonymous"
FTP_PASS="[email protected]"


FILES_TO_DOWNLOAD=(
"/testdata/run_heldsuarez_cam6_nt2_bigg_try005.cam.h5.0001-01-01-00000.nc"
)

# Download the file using FTP
ftp -inv $FTP_HOST << EOF
user $FTP_USER $FTP_PASS
binary
cd $(dirname ${FILES_TO_DOWNLOAD[0]})
get $(basename ${FILES_TO_DOWNLOAD[0]}) /home/cam_sima_user/$(basename ${FILES_TO_DOWNLOAD[0]})
bye
EOF