Mac.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes, 9/6/18\n",
    "\n",
    "## Mac homebrew\n",
    "\n",
    "### Build from source\n",
    "\n",
    "* This can be used to make sure your libraries have been compiled using the same compilers that you are going to use for JEDI\n",
    "\n",
    "* Need to do this on the mac since the default compilers are clang\n",
    "    * These have access to gcc, but it's an ancient version (4.x)\n",
    "    \n",
    "* Add the --build-from-source option to the brew install command\n",
    "    * --HEAD will check out a version from a repository\n",
    "    * you can define \"head\" in the brew formula which gives you the ability to define a particular branch or tag to use\n",
    "\n",
    "### Edit a formula\n",
    "\n",
    "* Formulas are ruby scripts\n",
    "    * To set an environment variable in a formula\n",
    "        * ENV\\['FC'\\] = '/usr/local/bin/gfortran-7'\n",
    "\n",
    "* brew edit <formula\\>\n",
    "\n",
    "\n",
    "* After finishing you can check your formula by \"auditing\"\n",
    "    * brew audit <formula\\>\n",
    "    * This checks for acceptability to merge your formula, or edits to and existing formula into the brew repository\n",
    "\n",
    "### Create your own formula\n",
    "\n",
    "* Need source code\n",
    "    * Brew wants source code so that it can be demonstrated that you can build on your platform\n",
    "\n",
    "* Two ways to provide source code:\n",
    "    * Tar file\n",
    "        * Give brew path to a tarball containing the source code and build configuration\n",
    "        * Can use autoconf, cmake, etc.\n",
    "        * Give brew the sha256 sum for the tar file\n",
    "            * shasum -a256 <tar_file\\>\n",
    "    * Repository\n",
    "        * Give brew url for cloning\n",
    "        * Can specify particular branches or tags\n",
    "        * Give brew the sha1 value for the commit you are checking out\n",
    "\n",
    "* Create a new formula\n",
    "    * brew edit <formula\\>\n",
    "    \n",
    "### Environment for homebrew\n",
    "\n",
    "* export HOMEBREW_CC=\"gcc-7\"\n",
    "* export HOMEBREW_CXX=\"g++-7\"\n",
    "\n",
    "* brew --env\n",
    "* brew config\n",
    "\n",
    "### Building boost\n",
    "\n",
    "* brew install --verbose --build-from-source --c++11 boost\n",
    "   * says it is ignoring c++11, but it uses c++1 anyway\n",
    "   * c++11 is necessary to get boost name mangling to sync up with JEDI name mangling\n",
    "\n",
    "### Formulas for open-mpi, hdf5, netcdf\n",
    "\n",
    "* brew edit open-mpi, hdf5, netcdf\n",
    "    * Change depends_on \"gcc\" to \"gcc@7\"\n",
    "    * For Fortran compiling, add: ENV\\['FC'\\] = '/usr/local/bin/gfortran:q'\n",
    "    \n",
    "* For open-mpi only\n",
    "    * brew edit open-mpi\n",
    "        * add to the head section\n",
    "            * :using =\\> :git,\n",
    "            * :branch =\\> \"v3.2.1\",\n",
    "            * :revision =\\> \"<full sha value from git log\\>\"\n",
    "        * ENV\\['CC'\\] = '/usr/local/bin/gcc-7'\n",
    "        * ENV\\['CXX'\\] = '/usr/local/bin/g++-7'\n",
    "\n",
    "* For netcdf only\n",
    "    * do a brew edit to see the url for downloading netcdf-fortran source tarball\n",
    "    * do a wget to download this tarball, unpack it, and edit the top level CMakeLists.txt\n",
    "        * get rid of the macro-backtrace options to gfortran (these are for clang, not gcc)\n",
    "    * repack the tar ball, and store in a safe place\n",
    "    * run shasum -a256 on the tarball\n",
    "    * brew edit netcdf\n",
    "        * in resource \"fortran\" do\n",
    "            * comment out the specs and replace with url and sha256 values in accordance with the new tarball\n",
    "                * use the file://<full path\\> form of the url\n",
    "\n",
    "* brew install --verbose --build-from-source open-mpi, hdf5, netcdf\n",
    "   * Add the --c++11 option for netcdf\n",
    "   * Don't use the --HEAD option for open-mpi\n",
    "        \n",
    "### Formulas for ecbuild, eckit, fckit\n",
    "\n",
    "* Need to create new formulas for these\n",
    "\n",
    "* brew create ecbuild, eckit, fckit\n",
    "\n",
    "* For ecbuild:\n",
    "~~~~~~~~\n",
    "class Ecbuild < Formula\n",
    "  desc \"CMake macros for building JEDI\"\n",
    "  homepage \"https://github.com/ECMWF/\"\n",
    "  url \"https://github.com/ECMWF/ecbuild.git\",\n",
    "    :using => :git,\n",
    "    :tag => \"2.9.0\",\n",
    "    :revision => \"10d46b3d22df192405aa4da148e5c80bd7a814e0\"\n",
    "\n",
    "  depends_on \"cmake\" => :build\n",
    "\n",
    "  def install\n",
    "    mkdir \"build\" do\n",
    "      system \"cmake\", \"..\", *std_cmake_args\n",
    "      system \"make\", \"install\"\n",
    "    end\n",
    "  end\n",
    "\n",
    "  test do\n",
    "  end\n",
    "end\n",
    "~~~~~~~~\n",
    "\n",
    "* For eckit:\n",
    "~~~~~~~~\n",
    "class Eckit < Formula\n",
    "  desc \"C++ utilities for JEDI\"\n",
    "  homepage \"https://github.com/ECMWF\"\n",
    "  url \"https://github.com/ECMWF/eckit.git\",\n",
    "    :using => :git,\n",
    "    :tag => \"0.22.0\",\n",
    "    :revision => \"a1d8af4a48cdfc9088761a2ddf69eceb69e4ba13\"\n",
    "\n",
    "  depends_on \"cmake\" => :build\n",
    "  depends_on \"ecbuild\"\n",
    "  depends_on \"eigen\"\n",
    "  depends_on \"open-mpi\"\n",
    "\n",
    "  def install\n",
    "    # Allow open-mpi to oversubscribe processes since\n",
    "    # only have 2 cores on this system.\n",
    "    ENV['OMPI_MCA_rmaps_base_oversubscribe'] = '1'\n",
    "    mkdir \"build\" do\n",
    "      system \"ecbuild\", \"..\", *std_cmake_args\n",
    "      system \"make\"\n",
    "      system \"ctest\"\n",
    "      system \"make\", \"install\"\n",
    "    end\n",
    "  end\n",
    "\n",
    "  test do\n",
    "  end\n",
    "end\n",
    "~~~~~~~~\n",
    "\n",
    "* For fckit:\n",
    "~~~~~~~~\n",
    "class Fckit < Formula\n",
    "  desc \"Fortran utilities for JEDI\"\n",
    "  homepage \"https://github.com/ECMWF/\"\n",
    "  url \"https://github.com/ECMWF/fckit.git\",\n",
    "    :using => :git,\n",
    "    :tag => \"0.5.2\",\n",
    "    :revision => \"dad53e01cb6353476ca9c890071713dca353da10\"\n",
    "\n",
    "  depends_on \"cmake\" => :build\n",
    "  depends_on \"ecbuild\"\n",
    "  depends_on \"eckit\"\n",
    "\n",
    "  def install\n",
    "    # Allow open-mpi to oversubscribe processes since\n",
    "    # only have 2 cores on this system.\n",
    "    ENV['OMPI_MCA_rmaps_base_oversubscribe'] = '1'\n",
    "    mkdir \"build\" do\n",
    "      system \"ecbuild\", \"..\", *std_cmake_args\n",
    "      system \"make\"\n",
    "      system \"ctest\"\n",
    "      system \"make\", \"install\"\n",
    "    end\n",
    "  end\n",
    "\n",
    "  test do\n",
    "  end\n",
    "end\n",
    "~~~~~~~~\n",
    "\n",
    "* To install: brew install --verbose ecbuild, eckit, fckit\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MPICH2 on Mac, 7/30/19\n",
    "\n",
    "* JEDI compiles okay using\n",
    "    * Mojave\n",
    "    * Clang 10.0.1\n",
    "    * GNU 9.1.0\n",
    "    * Mpich 3.3.1\n",
    "* But get a lot of test failures\n",
    "    * \"channel initialization failed\"\n",
    "    * \"sethostbyname failed, imac.fin.ucar.edu\"\n",
    "* Fix for this is to add an entry to the /etc/hosts file:\n",
    "\n",
    "~~~~~~~~\n",
    "hostname      # get the local host name\n",
    "              # on my imac this returns \"sysadmins-imac.fin.ucar.edu\"\n",
    "\n",
    "sudo vi /etc/hosts\n",
    "  # add entry that uses the result of above as the second item on the line\n",
    "  # For my imac this is:\n",
    "  \n",
    "  127.0.0.1 sysadmins-imac-fin.ucar.edu\n",
    "  127.0.0.1 imac-fin.ucar.edu            # this works too\n",
    "~~~~~~~~\n",
    "\n",
    "* When running on my MacBook, using Mpich, I get the following error in addition to the ones I saw on the iMac\n",
    "    * MacBook config:\n",
    "        * High Sierra\n",
    "        * Clang 10.0.0\n",
    "        * GNU 7.4.0\n",
    "        * Mpich2 3.3.1\n",
    "* From OOPS, the test called test_util_intset_parser fails\n",
    "    * for the test_get_channels_invalid section, the EXCEPT_THROWS() doesn't catch the exception and the program crashes\n",
    "    * Don't have this dubugged yet\n",
    "* Went back to openMPI on my MacBook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Catalina OS, 11/8/19\n",
    "\n",
    "* Fix file permissions for iTerm\n",
    "    * Without this, ls won't work on ~/Documents and others\n",
    "    * Set \"Full Disk Access\" for iTerm in the Security and Privacy system preferences\n",
    "    * Instructions: http://osxdaily.com/2018/10/09/fix-operation-not-permitted-terminal-error-macos/\n",
    "* Use brew to uninstall, then reinstall everything with Catalina compiled versions\n",
    "    * Use Nan's list from his notes on mac build\n",
    "* Go to \"System Preferences\" and install 11.0 and 11.2 versions of command line librarys\n",
    "* Use jedi-stack build process to re-build with Clang 11.0\n",
    "    * Got a link error during the build of eckit\n",
    "        * Cannot link dylib directly, crypto library from openssl is out of date\n",
    "        * Workaround:\n",
    "\n",
    "```\n",
    "brew upgrade openssl # openssl should already be there, but if not: brew install openssl\n",
    "\n",
    "cd /usr/local/lib\n",
    "ln -s ../Cellar/openssl/lib/libssl.1.0.0.dylib libssl.dylib\n",
    "ln -s ../Cellar/openssl/lib/libcrypto.1.0.0.dylib libcrypto.dylib\n",
    "\n",
    "# repeat the jedi-stack build process, eckit should complete this time\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vagrant notes, 3/6/20\n",
    "\n",
    "* Error: dpkg-reconfigure: unable to re-open stdin: No file or directory\n",
    "    * From ubuntu VM\n",
    "    * apt-get is attmpting to run in interactive mode, and is waiting for response from console\n",
    "    * Fix:\n",
    "        * export DEBIAN_FRONTEND=noninteractive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mac OS, System Integrity Protection (SIP), 5/4/20\n",
    "\n",
    "* This is enabled by default\n",
    "\n",
    "* Among other things, this strips DYLD_LIBRARY_PATH and LD_LIBRARY_PATH from environment\n",
    "    * Big impact on python scripts\n",
    "    * #!/usr/bin/env ptyhon\n",
    "        * This uses python as the interpreter, but the library path variables are missing\n",
    "\n",
    "* To shut off\n",
    "    * Reboot and hold dowd Cmd-R\n",
    "    * This goes into Recovery Mode\n",
    "    * From Recovery Mode\n",
    "        * Select Utilities -> Terminal\n",
    "        * In the terminal, enter\n",
    "\n",
    "```\n",
    "csrutil status  # check if enabled, disabled\n",
    "csrutil disable # shut of SIP\n",
    "reboot\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ctest abort, 5/4/20\n",
    "\n",
    "* Tests that try to catch expecptions sometimes abort on Mac OS\n",
    "\n",
    "* One example is test_ufo_parameters\n",
    "    * from build directory, run the follwing which results in \"Abort trap: 6\"\n",
    "    \n",
    "```\n",
    "ufo/test/test_ufo_parameters ufo/test/testinput/parameters.yaml\n",
    "```\n",
    "\n",
    "* Running a backtrace in the debugger on the above command shows that the C++ code in Parameters calls the GNU C++ library to do the stack unwind function\n",
    "    * Abort is on routine \\_Unwind\\_Resume\n",
    "\n",
    "* In the debugger (lldb) you can run the following to find where the text for any routine lives\n",
    "\n",
    "```\n",
    "image lookup -s _Unwind_Resume # if add -r option, then the arg to -s is a regexp\n",
    "```\n",
    "\n",
    "* Running this shows that \\_Unwind\\_Resume exists in 2 places:\n",
    "    * ``libunwind.dylib`` - Clang version\n",
    "    * ``libgcc_s.1.dylib`` - GNU version\n",
    "\n",
    "* Turns out these are incompatible and the abort is coming from C++ code (compiled by Clang) attempting to call the GNU unwind code.\n",
    "    * There are several articles on the web that discuss this issue\n",
    "    * gcc.gnu.org/buzzilla/show-bug for bug number 42159 is an example with good information\n",
    "\n",
    "* UFO and other repos have a mix of Fortran and C++ which causes problems when different compilers are used for C++ and Fortran\n",
    "    * On most systems we use all Intel or all GNU for C/C++/Fortran\n",
    "        * Just one path to unwind code, all is well\n",
    "    * On the Mac, using Clang and GNU Fortran\n",
    "        * This brings in two paths to the unwind code, causing ambiguous references\n",
    "        * Running ``otool -L lib/libufo.dylib`` shows both paths to unwind code exist\n",
    "\n",
    "```\n",
    "# GNU\n",
    "/usr/local/lib/gcc/9/libgfortran.5.dylib\n",
    "/usr/local/lib/gcc/9/libgcc_s.1.dylib\n",
    "\n",
    "# Clang\n",
    "/usr/lib/libc++.1.dylib\n",
    "/usr/lib/libSystem.B.dylib\n",
    "/usr/lib/system/libunwind.dylib\n",
    "```\n",
    "\n",
    "* Whichever the dynamic loader finds first is what gets called\n",
    "    * In the abort case, libgcc_s.1.dylib gets found before libunwind.dylib, but libunwind.dylib is what is needed\n",
    "    * If rename libgcc_s.1.dylib (so loader can't find it), then the tests pass\n",
    "        * This is because in this case, libunwind.dylib gets loaded which has the proper unwind code\n",
    "        * However, two atlas tests that used to pass now fail because of unresolved references\n",
    "            * The missing references must have been in libgcc_s.1.dylib\n",
    "\n",
    "* Seems the proper fix is to split the C++ and Fortran code into two library files\n",
    "    * The C++ code references only the Clang unwind code\n",
    "    * The Fortran code refrences only the GNU unwind code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic loader search order, 5/21/20\n",
    "\n",
    "* Order is different depending on whether or not the file is specified with a path\n",
    "    * Ie, in the call to dlopen()\n",
    "\n",
    "* When just a file name (no path)\n",
    "    1. LD_LIBRARY_PATH\n",
    "    2. DYLD_LIBRARY_PATH\n",
    "    3. process' working direcotry\n",
    "    4. DYLD_FALLBACK_LIBRARY_PATH\n",
    "\n",
    "* WHen path is included\n",
    "    1. DYLD_LIBRARY_PATH\n",
    "    2. Given path/filename\n",
    "    3. DYLD_FALLBACK_LIBRARY_PATH\n",
    "\n",
    "* Surprising that when the path is included, the first thing tried is DYLD_LIBRARY_PATH instead of the given path/filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic loader namespace, 5/21/20\n",
    "\n",
    "* The Mac OS dynamic loader (dyld) by default uses a feature called two-level namespace\n",
    "   * two-level namespace causes the link step to record where every reference in one library is resoloved in another\n",
    "   * during run time, the dynamic loader follow the recording that the link step created\n",
    "   * this allows both Clang and GNU std libraries to be linked to libraries containing C/C++/Fortran code and have the dynamic loader load the proper std library dylib files\n",
    "\n",
    "* Using -flat_namespace flag during the link step disables two-level namespace (using a flat namespace instead)\n",
    "   * This disables the recording of the link structure during the link step\n",
    "   * The dynamic loader at runtime uses the typical LD_LIBARAY_PATH, ... search\n",
    "   * This allows the Mac OS to emulate what most other Linux systems do (which can be useful)\n",
    "   * However, this can cause issues with libraries that have a C/C++/Fortran code mix\n",
    "       * Because the proper std library under each call is lost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MAS (mac app store CLI), 9/30/20\n",
    "\n",
    "* Can do updates from the App Store on the command line\n",
    "\n",
    "* `brew install mas`\n",
    "\n",
    "* `mas list`\n",
    "    * shows available upgrades\n",
    "* `mas search Xcode`\n",
    "    * shows id numbers for each upgrade\n",
    "* `mas install Xcode`\n",
    "    * install Xcode\n",
    "* `mas upgrade`\n",
    "    * upgrade all packages with pending updates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Alternative to app store for Xcode, 9/30/20\n",
    "\n",
    "* `developer.apple.com/download/all`\n",
    "    * sign in with Apple ID\n",
    "    * Xcode 11.5 -> clang 11.0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vagrant, 10/29/20\n",
    "\n",
    "* Updated to vagrant version 2.2.10\n",
    "\n",
    "* Upgrade Vagrantfile\n",
    "    * download from AWS\n",
    "        * `aws s3 cp s3://data.jcsda.org/containers/Vagrantfile .`\n",
    "* Update vagrant plugins\n",
    "    * `vagrant plugin update`\n",
    "* Rebuild vagrant\n",
    "\n",
    "```\n",
    "# remove old machine\n",
    "vagrant destroy\n",
    "rm -rf .vagrant\n",
    "\n",
    "# rebuild new machine\n",
    "vagrant up\n",
    "vagrant ssh\n",
    "\n",
    "# restore rc files in vagrant home directory\n",
    "cp vagrant_data/.bash_profile .\n",
    "cp vagrant_data/.bashrc .\n",
    "cp vagrant_data/.gitconfig .\n",
    "cp vagrant_data/.vimrc .\n",
    "exit\n",
    "\n",
    "# restart just to be sure\n",
    "exit\n",
    "vagrant halt\n",
    "vagrant up\n",
    "vagrant ssh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# jedi-stack building 7/12/21\n",
    "\n",
    "* Using the latest Xcode, CommandLineTools breaks when building the netcdf C++ library\n",
    "    * Get strange error when reading the \"version\" include file\n",
    "    * Looks like a version string gets returned instead of the inclusion of the file\n",
    "    \n",
    "    * Xcode 12.5.1\n",
    "    * CommandLineTools -> MacOSX11.3.sdk\n",
    "    * Clang 12.0.5\n",
    "    * Big Sur -> MacOSX 11.4\n",
    "\n",
    "* Tried Xcode 13 beta 2, but this is for newer OS beyond Big Sur\n",
    "\n",
    "* Worked through earlier versions of Xcode/CommandLineTools and found that the following works:\n",
    "    * Xcode 12.4\n",
    "    * CommandLineTools -> MacOSX11.1\n",
    "    * Clang 12.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# jedi-stack building 7/23/21\n",
    "\n",
    "* The build system is failing on the latest Xcode/CommandLineTools version (12.5.1)\n",
    "    * Get the wierd version issue noted in the prior entry when building netcdf C++ API\n",
    "\n",
    "* I tried different versions of Xcode/CommandLineTools to see which work and which don't work\n",
    "    * Big Sur, 11.5, iMac\n",
    "\n",
    "| Xcode/CommandLineTools version | SDK version | Clang version | Does build work? |\n",
    "|--------------------------------|-------------|---------------|------------------|\n",
    "| 12.5.1 | 11.4 | 12.0.5 | No |\n",
    "| 12.5   | 11.3 | 12.0.5 | No |\n",
    "| 12.4   | 11.1 | 12.0.0 | Yes |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# jedi-stack building 8/10/21\n",
    "\n",
    "* issue is file name conflict between new version include file in C++ stdlib and VERSION file that autoconf creates\n",
    "\n",
    "* Temporary fix is to rename the VERSION file from autoconf after configure is run (and before make is run)\n",
    "\n",
    "* Unidata is already aware of this and has a fix in their development track\n",
    "    * https://github.com/Unidata/netcdf-cxx4/commit/41c0233cb964a3ee1d4e5db5448cd28d617925fb\n",
    "    * Renames VERSION to VERSION.txt in the configure.ac file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mac debug shared libraries, 10/11/21\n",
    "\n",
    "* DYLD_PRINT_LIBRARIES=YES\n",
    "    * This causes the paths of dynamic libraries to be printed as they are loaded\n",
    "    * Very useful for tracking down problems with multiple versions of libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# M1 vs Intel, 11/13/21\n",
    "\n",
    "* On the MacBook Pro, two architectures are supported\n",
    "    * Native M1\n",
    "        * ARM processor\n",
    "    * Emulated Intel\n",
    "        * Rosetta emulator for x86_64\n",
    "    * /usr/bin/arch\n",
    "        * Returns which architecture you are using\n",
    "        * \"arm64\" --> native M1\n",
    "        * \"i386\" --> x86_64 emulation\n",
    "\n",
    "* Need to keep things consistent across architectures\n",
    "    * VS Code build exercise created arm64, but lldb crashed when starting up\n",
    "        * This is because VS Code was arm64, while lldb was i386\n",
    "        * Fixed this by installing the x86_64 only VS Code binaries\n",
    "        * The original installation was a universal binary which can run either as arm64 or i386\n",
    "            * Not sure why it didn't adjust since it was started in iTerm2 which is i386\n",
    "\n",
    "* iTerm2 is i386\n",
    "    * Installed this before switching to the new laptop\n",
    "\n",
    "* jedi-stack built as x86_64\n",
    "    * Perhaps because iTerm2 is x86_64?\n",
    "    * Try re-installing iTerm2, making sure to get the i386 version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spack-stack, 5/2/22\n",
    "\n",
    "## Intructions from Dom\n",
    "\n",
    "* Repo: NOAA-EMC/spack-stack\n",
    "\n",
    "* Make a recursive clone of the spack-stack repo\n",
    "    * `git clone -b develop --recursive https://github.com/NOAA-EMC/spack-stack`\n",
    "\n",
    "* Explore\n",
    "    * `config/sties/`\n",
    "        * Check out default, macos, llvm and gnu\n",
    "    * compilers\n",
    "        * gnu\n",
    "        * apple-clang\n",
    "        * llvm clang\n",
    "    * MPI\n",
    "        * mpich\n",
    "        * openmpi\n",
    "\n",
    "* make sure you have curl, and python poetry install\n",
    "    * `brew install curl`\n",
    "        * check list in the README.md file inside the `macos*` directory\n",
    "    * `python3 -m pip install poetry`\n",
    "\n",
    "* Spack-isms\n",
    "    * In spack what comes first takes precedence over subsequent calls\n",
    "    * '+' --> enable\n",
    "    * '~' --> disable\n",
    "    * '::' --> override (instead of append)\n",
    "        * look in common/packages.yaml for examples\n",
    "\n",
    "### Steps to do the build\n",
    "\n",
    "```\n",
    "# build apple clang, gnu fortran, openmpi\n",
    "\n",
    "# create spack environment and enter the environment\n",
    "cd spack-stack\n",
    "source setup.h\n",
    "./create-env.py --site=default --app-jedi-ufs-all --name=jedi-ufs-all-apple-clang-openmpi\n",
    "\n",
    "spack env activate envs/jedi-ufs-all-apple-clang-openmpi\n",
    "\n",
    "spack env status      # confirm you are inside the environment\n",
    "\n",
    "# Use spack to auto edit the config and packages yaml.\n",
    "cd envs/jedi-ufo-all-apple-clang-openmpi/site\n",
    "SPACK_SYSTEM_CONFIG_PATH=`pwd` spack external find --all --scope system\n",
    "SPACK_SYSTEM_CONFIG_PATH=`pwd` spack compiler find --scope system\n",
    "\n",
    "# Check that site/compilers.yaml has the correct configurations for apple clang\n",
    "# and gnu. Apple clang should be using gnu fortran.\n",
    "#\n",
    "# Edit the envs/jedi-ufs-all-apple-clang-openmpi/site/packages.yaml file\n",
    "#\n",
    "#  Remove old python spce (3.8)\n",
    "#  Remove curl from /usr\n",
    "#  Remove sqlite from /usr\n",
    "#\n",
    "#  At the top, add the following entry\n",
    "#\n",
    "#     all:\n",
    "#       compiler:: [apple-clang]\n",
    "#       providers:\n",
    "#         mpi:: [openmpi]\n",
    "#\n",
    "\n",
    "# Edit the envs/jedi-ufs-all-apple-clang-openmpi/spack.yaml file\n",
    "#\n",
    "# Under the \"specs:\" section:\n",
    "#     Comment out: \"- ufs-weather-model/debug\" line and replace with\n",
    "#                  \"- openmpi\"\n",
    "\n",
    "# set up and run the build\n",
    "#\n",
    "# The concretize command builds the dependency graph and places the result\n",
    "# in enva/jedi-ufs-all-apple-clang-openmpi/spack.lock file.\n",
    "#\n",
    "# spack install command:\n",
    "#     takes several hours when building the first time\n",
    "#     can use more options\n",
    "#         --fail-fast    # stop on first error\n",
    "#         --reuse        # for adding a new package in a subsequent call\n",
    "#                        # can use this option on the concretize command too\n",
    "#\n",
    "cd spack-stack   # top level directory in the clone\n",
    "spack concretize 2>&1 | tee log.concretize       # builds dependency graph\n",
    "spack install -v 2>&1 | tee log. install         # build and install stack\n",
    "```\n",
    "\n",
    "### Steps to create the lua module scripts\n",
    "\n",
    "```\n",
    "# Edit the envs/jedi-ufs-all-apple-clang-openmpi/common/modules.yaml file\n",
    "#\n",
    "#  Remove openmpi from the two blacklist files\n",
    "\n",
    "spack module lmod refresh   # create the lua scripts\n",
    "\n",
    "# If the above command reports duplicate errors, then need to remove the\n",
    "# duplication from the build. This can be done using:\n",
    "#\n",
    "#   spack find -l hdf5\n",
    "#   spack uninstall --dependents /HASH   # Get HASH from the find results associated\n",
    "#                                        # with ESMF, etc.\n",
    "\n",
    "./meta_modules/setup_meta_modules.py   # create lua scripts for compiler, mpi,\n",
    "                                       # python, etc.\n",
    "```\n",
    "\n",
    "* Sometimes the setup_mate_modules.py doesn't work. \n",
    "    * Could be the python installation.\n",
    "    * May need to use a bare-bones python installation as described below\n",
    "\n",
    "* Spack builds separate modules for each python package that is required for the stack\n",
    "    * This means that you can use a bare-bones python installation and reference everthing needed through PYTHONPATH\n",
    "        * Ie, don't install all of the python packages in your python installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack debug, 5/2/22\n",
    "\n",
    "* Two issues:\n",
    "    * flat namespace creates collision with `_Unwind_Resume` in clang and GNU libs\n",
    "    * rpaths not getting filled in for executables in JEDI build\n",
    "        * These are getting filled in for the stack build\n",
    "\n",
    "## Enabling two level namespace in the spack build\n",
    "* openmpi doesn't have two level namespace option, but mpich does\n",
    "* switch to mpich\n",
    "* enable two level namespace\n",
    "    * one way to do this is to enable on the spack install command line\n",
    "    * `spack install -v mpich two_level_namespace=true`\n",
    "    * but it would be better to find a way to put this in the yaml configuration\n",
    "\n",
    "```\n",
    "# go through build instructions for openmpi and substitute \"mpich\" for \"openmpi\"\n",
    "#\n",
    "# create envs/jedi-ufs-all-apple-clang-mpich environment\n",
    "# \n",
    "# edit envs/jedi-ufs-all-apple-clang-mpich/common/packages.yaml\n",
    "#   add \"+two_level_namespace\" to mpich.variants spec\n",
    "#\n",
    "#   entry should look like this:\n",
    "#\n",
    "#        mpich:\n",
    "#            variants: ~hwloc +two_level_namespace\n",
    "#\n",
    "\n",
    "spack concretize --force 2>&1 | tee log.concretize  # can be used to rebuild\n",
    "                                                    # spack.lock file\n",
    "spack install -v | tee log.install      # picks up two_level_namespace from config\n",
    "```\n",
    "\n",
    "* Check that two_level_namespace is applied\n",
    "    * `otool -hV file` should show \"TWOLEVEL\" in the flags column\n",
    "    * There should be no `-Wl,-flat_namespace` arguments in the install log file\n",
    "\n",
    "## RPATHs\n",
    "\n",
    "* Watch out for lldb and DYLD_LIBRARY_PATH\n",
    "    * Since lldb is in the protected area, DYLD_LIBRARY_PATH gets stripped out\n",
    "    * Some executables/libraries are getting setup with the full paths to the library files defined in their file, while others don't\n",
    "    * The files without the rpath specs rely on DYLD_LIBRARY_PATH and won't work proplerly in lldb\n",
    "        * Have to add in the rpath space using install_name_tool\n",
    "            * `install_name_tool -add_rpath path file`\n",
    "        * Once this is done lldb will work\n",
    "\n",
    "* Need to get build process to fill in the rpath directories\n",
    "    * The spack install process does this with the libraries in the stack\n",
    "    * JEDI CMake config does not do this\n",
    "    * Check out https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling for help with cmake configuration\n",
    "\n",
    "* Use `otool -l file` to see rpath specs and configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack debug, 5/2/22\n",
    "\n",
    "* once two level namespace was enabled, looks like everything gets built with two level namespace\n",
    "\n",
    "* fv3-bundle\n",
    "    * build succeeds\n",
    "    * ctest has 73 failures out of 1560 tests\n",
    "        * Down from 170 failures for the apple clang, openmpi build\n",
    "        * Lots of `test_qg_*, saber_test_*`\n",
    "        * `test_ioda_obsspace_put_db_channels` (Bus Error)\n",
    "        * Seven `test_ufo_*`\n",
    "        * `fv3jedi_test_tier1_lgetkf`\n",
    "    * Fair amount of tests finish with a report of status = 0, but then the process crashes\n",
    "\n",
    "* Debug test_ioda_obsspace_put_db_channels\n",
    "    * Repaired RPATHs using install_name_tool\n",
    "    * hdf5 library is crashing after the test finishes\n",
    "        * Looks like hdf5 is getting loaded from the correct location\n",
    "        * Is the linking step using the wrong location?\n",
    "    * `~/projects/SPACK_STACK/fv3-bundle`\n",
    "        * `build_mpich/ioda/test` check out log.test file\n",
    "\n",
    "* Do we need to create the bare-bones Python installation to work around this\n",
    "    * Ie, get rid of the hdf5 libs installed in the python area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specs for new laptop, 6/17/22\n",
    "\n",
    "* MacBookPro, 16 inch\n",
    "    * Space Gray\n",
    "    * M1 Max (10-core CPU, 32-core GPU, 16-core Neural Engine)\n",
    "    * 64GB Memory\n",
    "    * 1TB SSD Storage\n",
    "    * `$3899`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to quiet the deprecated warnings from Clang, 6/9/23\n",
    "\n",
    "* `ecbuild -D CMAKE_CXX_FLAGS=\"-Wno-deprecated-declarations\" ...`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to run debugger on MPI jobs, 8/14/23\n",
    "\n",
    "* `mpiexec -np 3 xterm -e lldb bin/time_IodaIO.x iodatest_time_io.yaml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hidden /usr/lib files, 8/24/23\n",
    "\n",
    "* Info: https://developer.apple.com/forums/thread/692383\n",
    "\n",
    "* How to extract lib files: https://github.com/keith/dyld-shared-cache-extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic loader does not load a library file, 10/16/23\n",
    "\n",
    "* Error: \"dyld cache (null) not loaded: syscall to map cache into shared region failed\"\n",
    "\n",
    "* If a program has a large static array allocation (> 2GB) it will encroach on the program space being used for the load of a library that is part of the dynamic linker shared cache.\n",
    "\n",
    "* See notes 10/11/23\n",
    "\n",
    "* See \"An Apple Library Primer\"\n",
    "\n",
    "* See https://developer.apple.com/forums/thread/676684 for a concise explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack-1.8.0 build - Ashley, 9/26/24\n",
    "\n",
    "* Had to remove linux homebrew\n",
    "  * Was using Rosetta\n",
    "  * homebrew removal didn't clean up properly\n",
    "  * Some pieces of /usr/local got confused with /opt/homebrew\n",
    "  * We have a note in spack-stack already saying we don't support Rosetta\n",
    "\n",
    "* Clingo issue\n",
    "  * Leftovers from prior uninstall corrupted clingo\n",
    "\n",
    "* spack update that is not in our fork\n",
    "  * FMS variant \"+shared\" didn't work\n",
    "  * Need changes from aut spack repo\n",
    "  * Turns out that Ashley was building from develop branchs and the release/1.8.0 branches had not been merged into develop yet\n",
    "  * Ashley will try again using the release/1.8.0 branches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ObsDataFrame, 10/7/24\n",
    "\n",
    "* Phil's last day is 11/29/24\n",
    "\n",
    "* New PR\n",
    "  * No sort for column-priority view\n",
    "    * Don't have a way to modify the view without modifying the underlying data framefor the column priority mode\n",
    "  * Sort does work on row-priority view\n",
    "    * You need to use row-priority frame if you want to sort the view\n",
    "  * Sort works on both row- and column-priority data frames\n",
    "  * Observer pattern\n",
    "    * Updating the underlying data frame will, for now, reset the view\n",
    "    * Eg, if the view had a slice run on it, that slice will be wiped out and the view will be reset to the total underlying data frame\n",
    "    * For now this is best\n",
    "      * It's a lot of effort to track all of the history of the view functions called, and to bring the view back into a consistent state with that history.\n",
    "      * Not worth the effort given that we are not clear on what the use cases are for the view\n",
    "  * A new app has been added which tests the view operations\n",
    "    * Also tests that the view and parent frame are indpependent of each other\n",
    "    * Also that the view doesn't make its own copy\n",
    "\n",
    "* Plan moving forward\n",
    "  * Phil work on the translation layer\n",
    "  * Approach will be to add a new test that gets developed along with the translation layer\n",
    "    * This will allow for small, incremental PRs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Infra, 10/7/24\n",
    "\n",
    "* Evan is creating relaunch slide presentations - later this week\n",
    "  * Need to review\n",
    "\n",
    "* Interview\n",
    "  * gist.github.com\n",
    "    * Nice for very small code snippets, tutorials, demos etc.\n",
    "  * Send Evan my ssh public key\n",
    "  * Will use a small AWS EC2 instance\n",
    "  * If I run out of time at end, ask candidate if they have any questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using spack-stack-1.8.0 on the Mac with skylab, 10/28/24\n",
    "\n",
    "* sourcing environment\n",
    "  * Copy jedi-tools/buildscripts/setup.sh\n",
    "  * Use `COMPILER=clang` in your copy of setup.sh (not \"apple-clang\")\n",
    "* runing ecbuild\n",
    "  * `ecbuild -DPython3_FIND_STRATEGY=LOCATION $JEDI_SRC`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}