Some_Conda_Env_Notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Python, R & CUDA: How to handle them on conda environments**\n",
    "## **Python Environment (For verify a specific Python version)**\n",
    "In this case, we want to use `Python 3.8.10` to run FADCIL. Note that we have created the directory `environment/PyVersionTest` for better organization and to isolate the following `.yml` file from other environments. Let's take a look:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:~]$ cd ~/environments/PyVersionTest && ls\n",
    "PyVersion.yml\n",
    "(base) [u6059911@kingspeak2:PyVersionTest] cat PyVersion.yml\n",
    "name: Env_Py_3_8\n",
    "channels:\n",
    "  - defaults\n",
    "  - conda-forge\n",
    "dependencies:\n",
    "  - python=3.8.10\n",
    "  - pip\n",
    "  - numpy\n",
    "  - pandas\n",
    "  - matplotlib\n",
    "  - scikit-learn\n",
    "  - jupyterlab\n",
    "  ```\n",
    "Where `PyVersion.yml` defines the dependencies we want in the `Env_py_3_8` environment. Furthermore, once we activate that environment, we have:\n",
    "  ```bash\n",
    "(base) [u6059911@kingspeak2:env_test]$ conda activate Env_Py_3_8\n",
    "(Env_Py_3_8) [u6059911@kingspeak2:env_test]$ python3 --version\n",
    "Python 3.8.10\n",
    "```\n",
    "Where we have check that, in this environment, we are working with `Python3.8.10`. If we deactivate the environment (i.e, return  to the `base` environment), we have:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:~]$ python3 --version\n",
    "Python 3.12.4\n",
    "```\n",
    "Therefore, note that conda environments work in isolation.\n",
    "## **R environment (For verify that R also works under the same package download scheme on conda)**\n",
    "Let's verify that we can download R dependencies using `.yml` files and create R conda environments. Note the path `~/environments/Rtest`:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:~]$ cd ~/environments/Rtest && ls\n",
    "Renv.yml CheckDependencies.R\n",
    "```\n",
    "Now let's see what these files contain on:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:Rtest] cat Renv.yml\n",
    "name: R_env\n",
    "channels:\n",
    "  - defaults\n",
    "  - conda-forge\n",
    "  - r\n",
    "dependencies:\n",
    "  - r-base=4.2.0 \n",
    "  - r-dplyr\n",
    "  - r-tidyverse\n",
    "  - openjdk=8\n",
    "(base) [u6059911@kingspeak2:Rtest] cat CheckDependencies.R\n",
    "packages <- c(\"dplyr\", \"tidyverse\", \"h2o\")\n",
    "\n",
    "for (pkg in packages) {\n",
    "  if (require(pkg, character.only = TRUE)) {\n",
    "    cat(sprintf(\"Paquete %s está instalado.\\n\", pkg))\n",
    "  } else {\n",
    "    cat(sprintf(\"Paquete %s no está instalado.\\n\", pkg))\n",
    "  }\n",
    "}\n",
    "```\n",
    "Where `Renv.yml` defines the `R_env` environment (with R dependencies), and `CheckDependencies.R` is a script that checks all dependencies were installed correctly and are ready to be used. Now, we need to activate the environment:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:Rtest]$ conda activate R_env\n",
    "```\n",
    "And then run the checking script:\n",
    "```bash\n",
    "(R_env) [u6059911@kingspeak2:Rtest]$ Rscript CheckDependencies.R\n",
    "Loading required package: dplyr\n",
    "\n",
    "Attaching package: ‘dplyr’\n",
    "\n",
    "The following objects are masked from ‘package:stats’:\n",
    "\n",
    "    filter, lag\n",
    "\n",
    "The following objects are masked from ‘package:base’:\n",
    "\n",
    "    intersect, setdiff, setequal, union\n",
    "\n",
    "Paquete dplyr está instalado.\n",
    "Loading required package: tidyverse\n",
    "── Attaching packages ────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.1 ──\n",
    "✔ ggplot2 3.3.6     ✔ purrr   0.3.4\n",
    "✔ tibble  3.1.7     ✔ stringr 1.4.0\n",
    "✔ tidyr   1.2.0     ✔ forcats 0.5.1\n",
    "✔ readr   2.1.2     \n",
    "── Conflicts ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──\n",
    "✖ dplyr::filter() masks stats::filter()\n",
    "✖ dplyr::lag()    masks stats::lag()\n",
    "Paquete tidyverse está instalado.\n",
    "Loading required package: h2o\n",
    "\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "Your next step is to start H2O:\n",
    "    > h2o.init()\n",
    "\n",
    "For H2O package documentation, ask for help:\n",
    "    > ??h2o\n",
    "\n",
    "After starting H2O, you can use the Web UI at http://localhost:54321\n",
    "For more information visit https://docs.h2o.ai\n",
    "\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "\n",
    "Attaching package: ‘h2o’\n",
    "\n",
    "The following objects are masked from ‘package:stats’:\n",
    "\n",
    "    cor, sd, var\n",
    "\n",
    "The following objects are masked from ‘package:base’:\n",
    "\n",
    "    &&, %*%, %in%, ||, apply, as.factor, as.numeric, colnames,\n",
    "    colnames<-, ifelse, is.character, is.factor, is.numeric, log,\n",
    "    log10, log1p, log2, round, signif, trunc\n",
    "\n",
    "Paquete h2o está instalado.\n",
    "Warning message:\n",
    "package ‘h2o’ was built under R version 4.2.3 \n",
    "\n",
    "```\n",
    "Which indicates that the R dependencies can also be used using the conda environment scheme.\n",
    "## **CUDA environment (For set up a fixed CUDA version on an isolate environment)**\n",
    "Now we want to know how to install a specific CUDA version in a coda environment. We hope that the CUDA version we install will be available only in that specific environment for better isolation. Let's take a look:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:~]$ cd ~/environments/CUDAtest && ls\n",
    "CuVersion.yml\n",
    "(base) [u6059911@kingspeak2:Rtest] cat CuVersion.yml\n",
    "name: CUDA_env\n",
    "channels:\n",
    "  - defaults\n",
    "  - nvidia\n",
    "dependencies:\n",
    "  - cudatoolkit=10.1\n",
    "```\n",
    "Now once we activate the environment:\n",
    "```bash\n",
    "(base) [u6059911@kingspeak2:Rtest]$ conda activate CUDA_env\n",
    "```\n",
    "We need to take the following step to ensure that the CUDA version module is automatically loaded each time we activate the environment:\n",
    "```bash\n",
    "(CUDA_env) [u6059911@kingspeak2:Rtest]$ mkdir -p $CONDA_PREFIX/etc/conda/activate.d\n",
    "(CUDA_env) [u6059911@kingspeak2:Rtest]$ echo \"module load cuda/10.1\" > $CONDA_PREFIX/etc/conda/activate.d/load_cuda.sh\n",
    "```\n",
    "Now, we can test the result leaving the environment and joining again:\n",
    "```bash\n",
    "(CUDA_env) [u6059911@kingspeak2:Rtest]$ conda deactivate CUDA_env\n",
    "(base) [u6059911@kingspeak1:cuda_test]$ conda activate CUDA_env\n",
    "```\n",
    "Let's see the CUDA version:\n",
    "```bash\n",
    "(CUDA_env) [u6059911@kingspeak1:cuda_test]$ nvcc --version\n",
    "nvcc: NVIDIA (R) Cuda compiler driver\n",
    "Copyright (c) 2005-2019 NVIDIA Corporation\n",
    "Built on Wed_Apr_24_19:10:27_PDT_2019\n",
    "Cuda compilation tools, release 10.1, V10.1.168\n",
    "```\n",
    "So the CUDAV10.1 was installed successfully. We can also verify that `nvcc` will only work inside `CUDA_env`. Note that:\n",
    "```bash\n",
    "(CUDA_env) [u6059911@kingspeak2:Rtest]$ echo $CONDA_PREFIX\n",
    "/uufs/chpc.utah.edu/common/home/u6059911/software/pkg/miniconda3/envs/CUDA_env\n",
    "```\n",
    "Returns the `CUDA_env` path, so we can find `load_cuda.sh` under this way: \n",
    "```bash\n",
    "(CUDA_env) [u6059911@kingspeak2:Rtest]$ ls $CONDA_PREFIX/etc/conda/activate.d/\n",
    "load_cuda.sh\n",
    "```\n",
    "But, for example, if we would have been on another environment, like `Env_Py_3_8`:\n",
    "```bash\n",
    "(Env_Py_3_8) [u6059911@kingspeak1:cuda_test]$ ls $CONDA_PREFIX/etc/conda/activate.d/\n",
    "libblas_mkl_activate.sh\n",
    "```\n",
    "That means, we would'nt have found the `load_cuda.sh`, which means thant `nvcc` would'nt be available in `Env_Py_3_8` (and, therefore, CUDA would not be available)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}