SoftwareTools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CMake\n",
    "\n",
    "* cmake can take \"defines\" on the command line that can be used to set paths in the makefiles\n",
    "    * Example:\n",
    "~~~~~~~~\n",
    "cmake -DCMAKE_EXE_LINKER_FLAGS_INIT=\"-L/Users/stephenh/projects/py-ncepbufr/src\" ..\n",
    "~~~~~~~~\n",
    "\n",
    "* Other controls  \n",
    "    https://cmake.org/cmake/help/v3.0/manual/cmake-variables.7.html\n",
    "    \n",
    "    CMAKE_ARGC  \n",
    "    CMAKE_ARGV0  \n",
    "    CMAKE_BINARY_DIR  \n",
    "    CMAKE_COMMAND  \n",
    "    ...  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git Flow\n",
    "\n",
    "## When the developer doesn't have push permission to the github repository\n",
    "\n",
    "### Developer in their local repo\n",
    "~~~~~~~~\n",
    "git flow init\n",
    "\n",
    "git flow feature start <name>\n",
    "\n",
    "make edits\n",
    "\n",
    "git flow feature publish <name>\n",
    "~~~~~~~~\n",
    "\n",
    "### Developer on GitHub\n",
    "\n",
    "~~~~~~~~\n",
    "initiate a pull request from branch featrue/<name> to branch develop\n",
    "~~~~~~~~\n",
    "\n",
    "### Owner on GitHub\n",
    "\n",
    "~~~~~~~~\n",
    "once feature/<name> is approved, process the pull request\n",
    "    merge feature/<name> into develop\n",
    "    delete feature/<name>\n",
    "~~~~~~~~\n",
    "\n",
    "### Developer in their local repo\n",
    "\n",
    "~~~~~~~~\n",
    "git branch -D feature/<name>\n",
    "\n",
    "git remote update -p  # -p prunes the remote feature/<name> branch from the local repo\n",
    "\n",
    "git checkout master\n",
    "git pull origin master\n",
    "\n",
    "git checkout develop\n",
    "git pull origin develop\n",
    "~~~~~~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ecbuild\n",
    "\n",
    "* To enable features, such as OpenMP, you use the ecbuild_add_option macro\n",
    "    * The user sets an ENABLE\\_ environment var (via ecbuild_add_option)\n",
    "    * And the macro sets HAVE\\_ accordingly\n",
    "    \n",
    "    * Example: OpenMP\n",
    "        * User places the following in a CMakeLists.txt file\n",
    "\n",
    "~~~~~~~~\n",
    "ecbuild_add_option( FEATURE OMP\n",
    "                    DEFAULT ON\n",
    "                    DESCRIPTION \"Use OpenMP\" )\n",
    "~~~~~~~~\n",
    "\n",
    "* This macro will set the HAVE_OMP cmake variable which is then used downstream by other cmake processes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# How to resolve mege conflict with a git lfs file\n",
    "\n",
    "```\n",
    "cd ioda-bundle/iodaconv\n",
    "\n",
    "git checkout develop\n",
    "git pull\n",
    "\n",
    "git checkout feature/mycode\n",
    "git pull\n",
    "\n",
    "git merge develop\n",
    "\n",
    "# for each of the lfs files with conflicts\n",
    "git checkout --theirs path_to_file    # choose develop version\n",
    "git checkout --ours path_to_file      # choose feature/mycode version\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build and test fv3-bundle on S4, 1/18/22\n",
    "\n",
    "* When building my own jedi-stack using intel/impi, the formula for loading modules is a bit tricky. Here is what I did to get this accomplished.\n",
    "\n",
    "```\n",
    "git lfs install # I may have not needed this, but it did seem to help\n",
    "\n",
    "module use /data/users/sherbener/projects/jedi-stack/modulefiles/apps\n",
    "\n",
    "module load license_intal\n",
    "module load git/2.30.0\n",
    "module load miniconda/3.8-s4\n",
    "module load jedi/intel-impi\n",
    "module load odc/ecmwf-1.4.4\n",
    "module load jedi-gcc/8.3\n",
    "\n",
    "cd /data/users/sherbener/projects\n",
    "git clone https://github.com/jcsda-internal/fv3-bundle\n",
    "cd fv3-bundle\n",
    "git clone https://github.com/jcsda-internal/jedi-cmake jedicmake\n",
    "mkdir build\n",
    "cd build\n",
    "ecbuild --toolchain=../jedicmake/cmake/Toolchains/jcsda-S4-Intel.cmake ..\n",
    "make -j4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build jedi-stack with debug settings on libraries, 1/31/22\n",
    "\n",
    "* MPICH\n",
    "    * Autoconfig\n",
    "        * `--enable-g=debug`\n",
    "        * `--disable-fast`\n",
    "        * `--enable-debug-info`\n",
    "\n",
    "* hdf5\n",
    "    * Autoconfig\n",
    "        * `--enable-build-mode=debug`\n",
    "        * `--enable-optimization=debug`\n",
    "\n",
    "* netcdf\n",
    "    * Autoconfig\n",
    "    * export CFLAGS, CXXFLAGS, FFLAGS, FCFLAGS\n",
    "        * \"-g -O0\"\n",
    "\n",
    "* pnetcdf\n",
    "    * Autocoonfig\n",
    "        * `--enable-debug`\n",
    "\n",
    "* eccodes\n",
    "    * cmake\n",
    "    * export CFLAGS, etc.\n",
    "\n",
    "* pio\n",
    "    * cmake\n",
    "    * export CFLAGS, etc.\n",
    "\n",
    "* There are a number of packages that have `--build=production` or `--build=release` settings which can be changed to `--build=Debug`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# fv3-bundle issue with building eckit, fckit, atlas, 2/23/22\n",
    "\n",
    "* Tried different combinations of running ecbuild using the `BUNDLE_SKIP_<REPO>` controls which controls whether you build the repo in your bundle (BUNDLE_SKIP_ = OFF) or use the installation in modules (BUNDLE_SKIP_ = ON, which is the default).\n",
    "\n",
    "| eckit | fckit | atlas | ecbuild status |\n",
    "|-------|-------|-------|----------------|\n",
    "| bundle | bundle | bundle | Fail |\n",
    "| bundle | bundle | module | Pass |\n",
    "| bundle | module | bundle | Fail |\n",
    "| bundle | module | moduel | Pass |\n",
    "| module | bundle | bundle | Pass |\n",
    "| module | bundle | module | Pass |\n",
    "| module | module | bundle | Pass |\n",
    "| module | module | moduel | Pass |\n",
    "\n",
    "* It appears that building eckit and atlas together in the bundle does not work. All other combinations do work.\n",
    "\n",
    "* Used `--trace-expand` to see what's going on with a failing combination\n",
    "    * Found that `ENABLE_OMP=OFF` when the atlas `atlas-import.cmake` file is being constructed\n",
    "    * This causes the `atlas_OMP, atlas_OMP_Fortran, atlas_OMP_CXX` features to be recorded in the `atlas-import.cmake` file as unavailable.\n",
    "    * Downstream oops is looking for these features, via a check script in eckit, that doesn't find them and throws the error\n",
    "\n",
    "* Found that adding `-DENABLE_OMP=ON` to the ecbuild command line fixes the problem since `ENABLE_OMP=ON` when the `atlas-import.cmake` file is being constructed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to debug python scripts with underlying C++ libraries, 5/12/22\n",
    "\n",
    "* The idea is to run python in one window, then run the debugger in another window (ie another process) and attach the debugger to the python process.\n",
    "    * You need to run the pyhton process where you can stop and start the script\n",
    "        * Either run python interactively or in pdb (python debugger) mode\n",
    "\n",
    "* The sequence goes like:\n",
    "\n",
    "```\n",
    "# in one window run:\n",
    "python3 -m pdb my_script.py\n",
    "\n",
    "# in another window, find the process id of the python job\n",
    "# that was started above\n",
    "ps aux | less\n",
    "\n",
    "# Then attach the debugger to that process and \"release\" that\n",
    "# process by telling the debugger to continue\n",
    "lldb\n",
    "(lldb) attach -pid py_proc_id\n",
    "(lldb) continue\n",
    "\n",
    "# in the debugger set a breakpoint in the C++ code\n",
    "(lldb) br s -f mysource.cpp -l myline\n",
    "\n",
    "# back in the python window tell the process to continue\n",
    "(pdb) c\n",
    "\n",
    "# After continuing the python process, you should see the lldb window\n",
    "# stop on the breakpoint and you can start debugging.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EWOK flow, 6/1/22\n",
    "\n",
    "## How to test my feature branch\n",
    "\n",
    "* load ewok environment\n",
    "\n",
    "* clone fv3-bundle and build\n",
    "    * Include my feature branch\n",
    "\n",
    "* Change JEDI_BIN environment variable to point to my bundle build/bin directory\n",
    "    * Don't change JEDI_SRC variable\n",
    "\n",
    "* After bundle is built, in a new terminal\n",
    "\n",
    "```\n",
    "# set up modules, environment for ewok\n",
    "source /work/noaa/da/jedipara/ewok/bin/ewok\n",
    "\n",
    "# point to the applications using my feature branch\n",
    "JEDI_BIN=\"path-to-my-bundle/build/bin\"\n",
    "\n",
    "# create an R2D2 config file (if not done yet)\n",
    "# check the instruction in the ewok/README.md documentation\n",
    "#\n",
    "# location is: $HOME/.r2d2/config.yaml\n",
    "\n",
    "# create experiment\n",
    "# take note of experiment id at end of messaging from\n",
    "# the create_experiment.py script\n",
    "create_experiment.py $JEDI_SRC/ewok/experiments/<exp-id>.yaml\n",
    "\n",
    "# Error messages\n",
    "#\n",
    "# Look in:\n",
    "#   For most tasks: /work/noaa/da/herbener/ewok/tmp/ecflow/exp-id\n",
    "#   For hofx task: /work/noaa/da/herbener/ewok/tmp/ewok/datetime-id\n",
    "\n",
    "# note if env vars change you need to restart the server\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack on EC2 instance, 6/10/22\n",
    "\n",
    "* Place a copy of the EWOK PEM file on your system and restrict access\n",
    "    * `~/.aws/ewok-tutorial-20220330.pem`\n",
    "    * `chmod 400 ~/.aws/ewok-tutorial-20220330.pem`\n",
    "\n",
    "* ssh to the instance\n",
    "    * `ssh -i /Users/steveherbener/.aws/ewok-tutorial-20220330.pem ec2-user@ec2-44-201-179-82.compute-1.amazonaws.com`\n",
    "\n",
    "* On the instance:\n",
    "\n",
    "```\n",
    "# set up modules and environment\n",
    "scl enable gcc-toolset-11 bash\n",
    "module use /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/modulefiles/Core\n",
    "module load stack-gcc/11.2.1 stack-openmpi/4.1.3 stack-python/3.9.7\n",
    "module load jedi-fv3-env/main jedi-ewok-env/main\n",
    "\n",
    "# clone fv3-bundle and fix CMakeLists.txt files (until the\n",
    "# CMakeLists.txt files get updated)\n",
    "#\n",
    "# In top-level CMakeLists.txt replace the\n",
    "# include git_functions.cmake with:\n",
    "\n",
    "# Use external jedi-cmake or build in bundle\n",
    "if(DEFINED ENV{jedi_cmake_ROOT})\n",
    "  include( $ENV{jedi_cmake_ROOT}/share/jedicmake/Functions/git_functions.cmake )\n",
    "else()\n",
    "  ecbuild_bundle( PROJECT jedicmake GIT \"https://github.com/jcsda-internal/jedi-cmake.git\" BRANCH develop UPDATE )\n",
    "  include( jedicmake/cmake/Functions/git_functions.cmake )\n",
    "endif()\n",
    "\n",
    "# The first run of ecbuild will fail, then in\n",
    "# fv3-jedi/test/CMakeLists.txt replace the\n",
    "# include git-functions.cmake with:\n",
    "\n",
    "if(DEFINED ENV{jedi_cmake_ROOT})\n",
    "  include( $ENV{jedi_cmake_ROOT}/share/jedicmake/Functions/git_functions.cmake )\n",
    "else()\n",
    "  include( ${CMAKE_SOURCE_DIR}/jedicmake/cmake/Functions/git_functions.cmake )\n",
    "endif()\n",
    "\n",
    "# move the the sandpit area, clone fv3-bundle and proceed\n",
    "\n",
    "cd sandpit/steve\n",
    "git clone https://github.com/jcsda-intenal/fv3-bundle\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git Submodules, 6/24/22\n",
    "\n",
    "* .gitmodule file\n",
    "    * branch specs are for navigation only\n",
    "\n",
    "* The hash on the commit that is used for the submodule is stored in the git metadata\n",
    "    * Can view this using github OR by running `git log`\n",
    "* The point of the feature branches in the parent repo is to have a branch with the correct submodule commit hash\n",
    "    * The submodule commit hash can be changed to test different versions of the submodule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DDT/MAP Arm Forge, 12/13/22\n",
    "\n",
    "* Dom's DDT tutorial\n",
    "    * Orion\n",
    "    * DDT memory debugger\n",
    "    * `ddt --connect` # prefix for connecting to remote GUI client on Mac\n",
    "    * DDT version on Orion is 22.0.2\n",
    "        * MacBook has 22.0.4 which is good (the major and minor version numbers need to match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intel ifx, ifort update, 2/1/23\n",
    "\n",
    "* ifx is done and ready for beta testing\n",
    "    * backward compatibility with ifort\n",
    "    * Optimizations, machine code generation however are different\n",
    "        * Due to differnt backends: LLVM vs Intel\n",
    "\n",
    "* ifort is not going away soon (will probably last a couple years still)\n",
    "\n",
    "* ifort and ifx both produce ABI compatible binaries\n",
    "    * This should allow us to use ifort as a debug tool\n",
    "        * Say we have switched to icx, ifx and ifx throws a mysterious compiler error. We can switch to icx, ifort and see if we get better diagnostics. Or we can switch to icx, ifort for this particular file as a workaround (if ifort completes)\n",
    "\n",
    "* LLVM (icx, icpx and ifx) will not be available for Mac/Intel hardware\n",
    "    * We should be able to use icx, ifort in this case (see above)\n",
    "\n",
    "* New development will only go into ifx\n",
    "    * This include support for offloading code execution (ie, GPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# jedipara access, 6/14/24\n",
    "\n",
    "* Need on S4 and Discover\n",
    "\n",
    "* S4 Done\n",
    "  * `sudo -iu jedipara`\n",
    " \n",
    "* Discover\n",
    "  * `sudo -iu jedipara`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WorkDay entering time, 7/19/24\n",
    "\n",
    "* Navigation for entering time\n",
    "  * Actions -> Enter Time by Type\n",
    "    * Create new row\n",
    "    * Click on first column\n",
    "    * Select Projects -> select account\n",
    "    * If there is no account nubmer to select\n",
    "      * select \"Account key not assigned\"\n",
    "      * Click second column\n",
    "      * Click down to \"All UCAR Division Keys\"\n",
    "      * Scroll to \"58\" and select\n",
    "      * Scroll to specific account and select"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 10/21/24\n",
    "\n",
    "* LD_DEBUG is useful for tracking down shared library issues at runtime\n",
    "  * LD_DEBUG=libs -> show the libraries that are being selected for loading\n",
    "\n",
    "* Orion, intel, spack-stack-1.8.0 env\n",
    "  * tar command is messed up\n",
    "  * libz is in the spack-stack installation and in /usr/lib64\n",
    "  * libcrypt is in the intel compiler tools installation and in /usr/lib64\n",
    "  * tar needs to load the /usr/lib64 versions in order to work properly\n",
    "  * LD_DEBUG_libs tracing shows\n",
    "    * In the good environment (ie, right after login and before sourcing the spack-stack-1.8.0 env) the loading goes as intended and tar works\n",
    "    * In the spack-stack-1.8.0 environment, the loading goes astray when libcrypt is loaded (ie, the intel compiler version gets loaded instead of the /usr/lib64 version)\n",
    "  * In the spack-stack-1.8.0 environment, the following works for tar\n",
    "  * `LD_LIBRARY_PATH=\"\" tar <args>`\n",
    "  * This issue is being tracked in: https://github.com/JCSDA/spack-stack/issues/1355"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 10/22/24\n",
    "\n",
    "* 1.9.0 release is targeted for end of this year\n",
    "  * LLVM based Intel compilers: icx, icpx, ifx\n",
    "\n",
    "* Dom will start the spack develop sync effort soon\n",
    "  * Need to get the openmpi fix into the auth spack repo soon\n",
    "\n",
    "* Carmeron, Mac Support, NOAA-EPIC\n",
    "  * They are also questioning the effort it takes to maintain the native Mac builds\n",
    "  * Looking at alternatives too"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# David H's instructions for gsi-addon env on S4, spack-stack-1.6.0, 10/28/24\n",
    "\n",
    "```\n",
    ". setup.sh\n",
    "spack stack create env --site <site name> --template unified-dev --name unified-env\n",
    "cd envs/unified-env\n",
    "spack env activate .\n",
    "spack concretize 2>&1 | tee log.concretize && spack install --verbose --fail-fast 2>&1 | tee log.install\n",
    "spack module lmod refresh\n",
    "spack stack setup-meta-modules\n",
    "spack env deactivate\n",
    "spack stack create env --site <site name> --template gsi-addon-dev --name gsi-addon --upstream $(pwd)/install\n",
    "cd ../gsi-addon\n",
    "vim spack.yaml # manually add \"compilers: [  '%gnu', '%intel' ]\"\n",
    "spack env activate .\n",
    "spack concretize 2>&1 | tee log.concretize && spack install --verbose --fail-fast 2>&1 | tee log.install\n",
    "spack module lmod refresh --upstream-modules\n",
    "spack stack setup-meta-modules\n",
    "```\n",
    "* Need to make sure the compiler specs are updated with spack-stack-1.6.0 ue-intel-2021.10.0 specs\n",
    "  * site/compilers.yaml\n",
    "  * site/packages.yaml\n",
    "  * Find all other places where \"intel\" exists and check those"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}