From 1078a44f58fcbd7b6480b00d047bf680132036bf Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Thu, 22 Aug 2024 14:29:56 -0400 Subject: [PATCH 01/19] Remove old Centos 7 Directions --- Languages/Python/Python_FASRC.md | 114 ------------------------------- Languages/Python/README.md | 1 - 2 files changed, 115 deletions(-) delete mode 100644 Languages/Python/Python_FASRC.md diff --git a/Languages/Python/Python_FASRC.md b/Languages/Python/Python_FASRC.md deleted file mode 100644 index bf99c272..00000000 --- a/Languages/Python/Python_FASRC.md +++ /dev/null @@ -1,114 +0,0 @@ -# Python (Anaconda) on the FASRC cluster - -anaconda-logo -

- -We like the [Anaconda](https://www.anaconda.com/products/distribution) python distribution from Continuum Analytics. It includes hundreds of the most popular packages for large-scale data processing, predictive analytics, and scientific computing (numpy, scipy,ipython, pandas, scikit-learn, mpi4py, etc.). It generally does a great job of including the latest versions of all the packages while maintaining compatibility between them all. - -You can use Python and Anaconda on the cluster by running: - -```bash -# Load the Python module -$ module load python/3.9.12-fasrc01 - -# Check Python version -$ python --version -Python 3.9.12 -``` - -### Customizing the environment - -Anaconda has a concept of *environments*; these are used to manage alternative package sets and versions. For example, if you want newer versions of some packages in the default environment, you can make a new environment with your customizations. - -Load the base environment based on the version of python. To see the available versions, visit our [Portal Module Lookup](https://portal.rc.fas.harvard.edu/p3/build-reports/python). - -**Create a new environment:** -```bash -$ conda create -n ENV_NAME PACKAGES -``` - -After loading the Anaconda module, you can use the following command to list the existing environments. - -```bash -$ conda env list -``` - -[The Conda documentation](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) has more details on different options. Use the following command to run the environment. - -```bash -$ source activate ENV_NAME -``` - -If you want to use this environment all the time, add the above line to your ~/.bashrc (or other appropriate shell config file) after the line that loads the module. - -> **WARNING:** It is advised not to use conda activate but instead to use source activate. conda activate will ask you to run conda init to modify your .bashrc. However doing this will permanently activate the conda environment, and will create problems later on. Instead use source activate. If you have run conda init and you want to undo it see this [document](https://docs.ycrc.yale.edu/clusters-at-yale/guides/conda/#conda-version-doesnt-match-the-module-loaded) for how. If you want to maintain conda init see this [example](https://kb.iu.edu/d/axgp#conda-init). - -At this point you can upgrade or install a package named PACKAGE with the following command (it’s the same whether installing or upgrading): - -```bash -$ conda install PACKAGE -``` - -The commands conda list and conda show list installed and available packages, respectively. See the [the conda documentation](https://docs.conda.io/en/latest/index.html) for all the details. If the package is not available from conda, you can install it into your environment using pip: - -```bash -$ pip install PACKAGE -``` - -> **Note:** Anaconda generally has the latest versions of all packages that are compatible. When you install an extra package, it will often update core packages like numpy and scipy; other packages may then downgrade them back. This is why we recommend sticking to the default environment if possible. - -If you have problems updating a package that is already installed in the Anaconda environment, you might need to remove the package first and then reinstall it: - -```bash -$ conda remove PACKAGE -$ conda install PACKAGE -``` - -This will often bypass update errors, especially with certain versions of matplotlib. - -To stop using the custom environment, run the below command: -```bash -$ source deactivate -``` - -To remove old environments, you can run. - -```bash -$ conda env remove -n ENV_NAME -``` - -### Submitting Jobs from within Conda Environments - -Since Slurm clones the current environment into a submitted job, that means that jobs submitted with an active Conda Environment will have an odd environment which can lead to problems. It is recommended that users add the - -```bash ---export=NONE -``` - -option to their submissions, this will prevent this cloning. The submitted job will then run in a pristine environment. - -### Choosing the Right Cluster Node to Install From - -**If you are working on the Cannon cluster** - -If you are installing packages in your home directory, it would be best to install from a Boston login node, not a Holyoke login node, because the cluster home directories are located in the Boston datacenter. To ensure you are routed to a Boston login node when you ssh to the cluster, do the following: - -```bash -ssh @boslogin.rc.fas.harvard.edu -``` - -**If you are working on FASSE** - -All FASSE login nodes are in Boston, so simply login to FASSE as normal: - -```bash -ssh @fasselogin.rc.fas.harvard.edu -``` - -## Useful Commands - -The Conda Cheatsheet has other helpful commands. - -anaconda-cheatsheet - -anaconda-cheatsheet diff --git a/Languages/Python/README.md b/Languages/Python/README.md index 426d233a..65db9a7c 100755 --- a/Languages/Python/README.md +++ b/Languages/Python/README.md @@ -9,7 +9,6 @@ Python is dynamically typed and garbage-collected. It supports multiple programm ### Content: -* CentOS 7: [Python on the FASRC Cluster](./Python_FASRC.md) * Rocky 8: [Speeding up conda environment management with Mamba](./Mamba.md) ### Examples: From 325d9c3c65f3d71fe7628cb344a7f4f05169cbfd Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Thu, 22 Aug 2024 15:10:19 -0400 Subject: [PATCH 02/19] Mamba.md is out of date version of User_Docs Mamba. --- Languages/Python/Mamba.md | 197 -------------------------------------- 1 file changed, 197 deletions(-) delete mode 100644 Languages/Python/Mamba.md diff --git a/Languages/Python/Mamba.md b/Languages/Python/Mamba.md deleted file mode 100644 index 38095651..00000000 --- a/Languages/Python/Mamba.md +++ /dev/null @@ -1,197 +0,0 @@ -mamba-logo - -[Mamba](https://mamba.readthedocs.io/en/latest/index.html#) is a fast, robust, and cross-platform package manager. - -mamba is a drop-in replacement and uses the same commands and configuration options as conda. You can swap almost all commands between conda & mamba. - -Mamba is available on the FASRC cluster as a software module either as Mambaforge or as python/3* which is aliased to Mamba: - -```bash -$ module load python/3.10.13-fasrc01 -$ python -V -Python 3.10.13 -``` - -You can create conda environments with mamba in the same way as with conda: - -```bash -$ mamba create -n ENV_NAME PACKAGES -``` - -> **NOTE:** The major advantage of using mamba instead of conda is that the environment creation, and installing / uninstalling of packages is *much* faster with mamba, e.g., - -```bash -$ mamba create -n python_env1 python=3.10 pip wheel - - __ __ __ __ - / \ / \ / \ / \ - / \/ \/ \/ \ -███████████████/ /██/ /██/ /██/ /████████████████████████ - / / \ / \ / \ / \ \____ - / / \_/ \_/ \_/ \ o \__, - / _/ \_____/ ` - |/ - ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ - ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ - ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ - ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ - ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ - ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝ - - mamba (0.25.0) supported by @QuantStack - - GitHub: https://github.com/mamba-org/mamba - Twitter: https://twitter.com/QuantStack - -█████████████████████████████████████████████████████████████ - - -Looking for: ['python=3.10', 'pip', 'wheel'] - -pkgs/main/noarch No change -pkgs/r/linux-64 No change -pkgs/r/noarch No change -pkgs/main/linux-64 5.3MB @ 4.5MB/s 1.2s -conda-forge/noarch 11.5MB @ 3.8MB/s 3.1s -conda-forge/linux-64 30.1MB @ 4.4MB/s 7.1s -Transaction - - Prefix: /scratch/pkrastev/conda/python_env1 - - Updating specs: - - - python=3.10 - - pip - - wheel - - - Package Version Build Channel Size -────────────────────────────────────────────────────────────────────────────────── - Install: -────────────────────────────────────────────────────────────────────────────────── - - + _libgcc_mutex 0.1 main pkgs/main/linux-64 Cached - + _openmp_mutex 5.1 1_gnu pkgs/main/linux-64 Cached - + bzip2 1.0.8 h7b6447c_0 pkgs/main/linux-64 Cached - + ca-certificates 2023.01.10 h06a4308_0 pkgs/main/linux-64 Cached - + certifi 2022.12.7 py310h06a4308_0 pkgs/main/linux-64 Cached - + ld_impl_linux-64 2.38 h1181459_1 pkgs/main/linux-64 Cached - + libffi 3.4.2 h6a678d5_6 pkgs/main/linux-64 Cached - + libgcc-ng 11.2.0 h1234567_1 pkgs/main/linux-64 Cached - + libgomp 11.2.0 h1234567_1 pkgs/main/linux-64 Cached - + libstdcxx-ng 11.2.0 h1234567_1 pkgs/main/linux-64 Cached - + libuuid 1.41.5 h5eee18b_0 pkgs/main/linux-64 Cached - + ncurses 6.4 h6a678d5_0 pkgs/main/linux-64 Cached - + openssl 1.1.1t h7f8727e_0 pkgs/main/linux-64 Cached - + pip 23.0.1 py310h06a4308_0 pkgs/main/linux-64 3MB - + python 3.10.9 h7a1cb2a_2 pkgs/main/linux-64 28MB - + readline 8.2 h5eee18b_0 pkgs/main/linux-64 Cached - + setuptools 65.6.3 py310h06a4308_0 pkgs/main/linux-64 1MB - + sqlite 3.40.1 h5082296_0 pkgs/main/linux-64 Cached - + tk 8.6.12 h1ccaba5_0 pkgs/main/linux-64 Cached - + tzdata 2022g h04d1e81_0 pkgs/main/noarch Cached - + wheel 0.38.4 py310h06a4308_0 pkgs/main/linux-64 66kB - + xz 5.2.10 h5eee18b_1 pkgs/main/linux-64 Cached - + zlib 1.2.13 h5eee18b_0 pkgs/main/linux-64 Cached - - Summary: - - Install: 23 packages - - Total download: 32MB - -────────────────────────────────────────────────────────────────────────────────── - -Confirm changes: [Y/n] Y -wheel 65.6kB @ 337.5kB/s 0.2s -setuptools 1.2MB @ 5.4MB/s 0.2s -pip 2.7MB @ 10.7MB/s 0.3s -python 28.1MB @ 73.0MB/s 0.4s -Preparing transaction: done -Verifying transaction: done -Executing transaction: done - -To activate this environment, use - - $ mamba activate python_env1 - -To deactivate an active environment, use - - $ mamba deactivate -``` - -To use the environment, do: - -```bash -$ source activate python_env1 -``` - -You can list the packages currently installed in the conda environment with: - -```bash -$ mamba list -``` - -You can install new packages with: - -```bash -$ mamba install -y numpy -``` - -To uninstall packages, use: - -```bash -$ mamba uninstall PACKAGE -``` - -When you finish using the conda environment, you can deactivate it with: - -```bash -$ conda deactivate -``` - -For additional features, please refer to the [Mamba documentation](https://mamba.readthedocs.io/en/latest/index.html). - -## Use conda environment in Jupyter Notebooks - -If you would like to use a conda environment as a kernel in a Jupyter Notebook on Open OnDemand ([Cannon OOD](https://rcood.rc.fas.harvard.edu) or [FASSE OOD](fasseood.rc.fas.harvard.edu)), you have to install packages, `ipykernel` and `nb_conda_kernels`. These packages will allow Jupyter to detect conda environments that you created from the command line. - -For example, if your environment name is `python_env1`: - -```bash -module load python -source activate python_env1 -mamba install ipykernel -mamba install nb_conda_kernels -``` - -**After** these packages are installed, launch a new Jupyter Notebook job (existing Jupyter Notebook jobs will fail to "see" this environment). Then - -1. Open a Jupyter Notebook (a `.ipynb` file) -2. On the top menu, click Kernel -> Change kernel -> select the conda environment - -## Conda environments in Lab space -By default conda/mamba install your packages in your home directory under ~/.conda. While a safe place to keep this data it is also very low performance due to filesystem latency. It also makes it hard to share your conda environments with other people on the cluster. - -You can tell conda to install to a different location by [setting](https://conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#specify-env-directories) up your ~/.condarc. We recommend using your holylabs directory space as it should have low latency. Before setting up your .condarc you will need to make the directory you want to put the environments into (ex. /n/holylabs/LABS/jharvard_lab/Lab/envs). Then in ~/.condarc set: - -```bash -envs_dirs: - - /n/holylabs/LABS/jharvard_lab/Lab/envs -``` - -For more on .condarc see the [official documentation](https://conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#). - -## Troubleshooting - -### Interactive vs. batch jobs - -If your code works in an interactive job, but fails in a slurm batch job, - -1. You are submitting your jobs from within a mamba/conda environment. - **Solution 1:** Deactivate your environment with the command `mamba deactivate` and submit the job or - **Solution 2:** Open another terminal and submit the job from outside the environment. - -2. Check if your `~/.bashrc` or `~/.bash_profile` files have a section of `conda initialize` or a `source activate` command. The `conda initialize` section is known to create issues on the FASRC clusters. - **Solution:** Delete the section between the two `conda initialize` statements. If you have `source activate` in those files, delete it or comment it out. - For more information on `~/.bashrc` files, see https://docs.rc.fas.harvard.edu/kb/editing-your-bashrc/ From 012b49e58b5a68f578816dae6afa08f2b60098e9 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:10:55 -0400 Subject: [PATCH 03/19] Updates to the README. --- Languages/Python/README.md | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-) diff --git a/Languages/Python/README.md b/Languages/Python/README.md index 65db9a7c..9eb3da9f 100755 --- a/Languages/Python/README.md +++ b/Languages/Python/README.md @@ -1,11 +1,9 @@ ## Python Python-logo -### What is Python? +### Purpose: -[Python](https://www.python.org/) is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation. - -Python is dynamically typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a "batteries included" language due to its comprehensive standard library. +Examples of using Python on the Harvard University FAS cluster. Feel free to contribute new examples. ### Content: @@ -14,15 +12,3 @@ Python is dynamically typed and garbage-collected. It supports multiple programm ### Examples: * [Example 1](Example1/): Monte-Carlo calculation of PI * [Example 2](Example2/): Text rendering With LaTeX - -### References: - -* [Official Python Documentation](https://docs.python.org/3/) -* [The Python Tutorial](https://docs.python.org/3/tutorial/index.html) -* [Software Carpentry Python Lesson](https://swcarpentry.github.io/python-novice-inflammation/index.html) -* [NumPy User Guide](https://numpy.org/doc/stable/user/index.html#user) -* [SciPy Official Documentation](https://docs.scipy.org/doc/scipy/index.html) -* [Matplotlib](https://matplotlib.org/) -* [Seaborn](https://seaborn.pydata.org/) -* [Pandas](https://pandas.pydata.org/) - From 7f334467c53b301b38a5db09a743d8641ad28de4 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:13:54 -0400 Subject: [PATCH 04/19] Python Example Workflow --- .../Example1/.github/workflows/test.yml | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 Languages/Python/Example1/.github/workflows/test.yml diff --git a/Languages/Python/Example1/.github/workflows/test.yml b/Languages/Python/Example1/.github/workflows/test.yml new file mode 100644 index 00000000..93610704 --- /dev/null +++ b/Languages/Python/Example1/.github/workflows/test.yml @@ -0,0 +1,46 @@ +name: CI Test for mc_pi + +on: + push: + branches: + - main + pull_request: + branches: + - main + +jobs: + test: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.x' # Specify the required Python version + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt # If you have dependencies listed in a requirements.txt + + - name: Run mc_pi.py + run: | + python mc_pi.py + + slurm: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Install SLURM + run: | + sudo apt-get update + sudo apt-get install -y slurm-wlm + + - name: Run SLURM batch file + run: | + sbatch run.sbatch From f74638f41db1c68838a90bd03f275a281b6726aa Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:17:44 -0400 Subject: [PATCH 05/19] reorg of py test --- .../test.yml => .github/workflows/Languages-Python-Example1.yml | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Languages/Python/Example1/.github/workflows/test.yml => .github/workflows/Languages-Python-Example1.yml (100%) diff --git a/Languages/Python/Example1/.github/workflows/test.yml b/.github/workflows/Languages-Python-Example1.yml similarity index 100% rename from Languages/Python/Example1/.github/workflows/test.yml rename to .github/workflows/Languages-Python-Example1.yml From 480ec77feef18e93f4ec10617e247e27f8e85a87 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:48:23 -0400 Subject: [PATCH 06/19] specify branch for testing --- .github/workflows/Languages-Python-Example1.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 93610704..0d517cbc 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -3,10 +3,10 @@ name: CI Test for mc_pi on: push: branches: - - main + - User_Doc-Python pull_request: branches: - - main + - User_Doc-Python jobs: test: From 7d4026763b9e5129d6e54f4b625572a95f60029a Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:50:50 -0400 Subject: [PATCH 07/19] add workflow dispatch --- .github/workflows/Languages-Python-Example1.yml | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 0d517cbc..8377f378 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -3,10 +3,11 @@ name: CI Test for mc_pi on: push: branches: - - User_Doc-Python + - main pull_request: branches: - - User_Doc-Python + - main + workflow_dispatch: # This enables manual triggering of the workflow jobs: test: @@ -24,11 +25,11 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip - pip install -r requirements.txt # If you have dependencies listed in a requirements.txt + pip install -r requirements.txt # If you have a requirements.txt file in the repo - name: Run mc_pi.py run: | - python mc_pi.py + python Languages/Python/Example1/mc_pi.py slurm: runs-on: ubuntu-latest @@ -43,4 +44,4 @@ jobs: - name: Run SLURM batch file run: | - sbatch run.sbatch + sbatch Languages/Python/Example1/run.slurm From 2d38a45e112393be36e1f39b4782b158a49d279d Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:55:47 -0400 Subject: [PATCH 08/19] Adding requirements.txt file for Example1, even though it is blank. --- Languages/Python/Example1/requirements.txt | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Languages/Python/Example1/requirements.txt diff --git a/Languages/Python/Example1/requirements.txt b/Languages/Python/Example1/requirements.txt new file mode 100644 index 00000000..cf7485ff --- /dev/null +++ b/Languages/Python/Example1/requirements.txt @@ -0,0 +1,2 @@ +# requirements.txt +# No external dependencies are required for mc_pi.py From dadc281223e8b2ae904e0799c1b98f5d908a0a86 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 12:58:49 -0400 Subject: [PATCH 09/19] Move to root directory. --- Languages/Python/Example1/requirements.txt => requirements.txt | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Languages/Python/Example1/requirements.txt => requirements.txt (100%) diff --git a/Languages/Python/Example1/requirements.txt b/requirements.txt similarity index 100% rename from Languages/Python/Example1/requirements.txt rename to requirements.txt From c08d69515799e6b8c294c715350cee1b8b1dab0c Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:04:12 -0400 Subject: [PATCH 10/19] rename job --- .github/workflows/Languages-Python-Example1.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 8377f378..95685ace 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -10,7 +10,7 @@ on: workflow_dispatch: # This enables manual triggering of the workflow jobs: - test: + Languages-Python-Example1: runs-on: ubuntu-latest steps: From 3b9b824ba38abd701cdaac27917775f00142fe19 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:05:30 -0400 Subject: [PATCH 11/19] rename job --- .github/workflows/Languages-Python-Example1.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 95685ace..71bbf1d6 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -1,4 +1,4 @@ -name: CI Test for mc_pi +name: CI Test for Languages-Python-Example1/mc_pi.py on: push: From 36bebf546f38eb123241850323e585b63528b8d1 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:10:27 -0400 Subject: [PATCH 12/19] Change to slurm emulator using docker. --- .github/workflows/Languages-Python-Example1.yml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 71bbf1d6..705233db 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -37,11 +37,14 @@ jobs: - name: Checkout repository uses: actions/checkout@v2 - - name: Install SLURM + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Pull SLURM Simulator Docker Image run: | - sudo apt-get update - sudo apt-get install -y slurm-wlm + docker pull hpcnow/slurm_simulator:20.11.9 - - name: Run SLURM batch file + - name: Run SLURM job in Docker run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ sbatch Languages/Python/Example1/run.slurm From 78efb13aed84db5f434a1ec5f1b6a701f34bc9f9 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:15:47 -0400 Subject: [PATCH 13/19] Use correct paths. --- .../workflows/Languages-Python-Example1.yml | 27 +++++++++++-------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 705233db..c5c569c6 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -33,18 +33,23 @@ jobs: slurm: runs-on: ubuntu-latest + steps: - - name: Checkout repository - uses: actions/checkout@v2 + - name: Checkout repository + uses: actions/checkout@v2 - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v2 + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 - - name: Pull SLURM Simulator Docker Image - run: | - docker pull hpcnow/slurm_simulator:20.11.9 + - name: Pull SLURM Simulator Docker Image + run: | + docker pull hpcnow/slurm_simulator:20.11.9 - - name: Run SLURM job in Docker - run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ - sbatch Languages/Python/Example1/run.slurm + - name: List Files in Mounted Directory + run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 ls -R /workspace + + - name: Run SLURM job in Docker + run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ + sbatch /workspace/Languages/Python/Example1/run.slurm \ No newline at end of file From d66ee11a3a98545f23014827105dc119b67188ac Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:22:42 -0400 Subject: [PATCH 14/19] Troubleshooting --- .../workflows/Languages-Python-Example1.yml | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index c5c569c6..c1689aca 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -33,23 +33,26 @@ jobs: slurm: runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v2 + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v2 + - name: Pull SLURM Simulator Docker Image + run: | + docker pull hpcnow/slurm_simulator:20.11.9 - - name: Pull SLURM Simulator Docker Image - run: | - docker pull hpcnow/slurm_simulator:20.11.9 + - name: List Files in Mounted Directory + run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 ls -la /workspace/Languages/Python/Example1 - - name: List Files in Mounted Directory - run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 ls -R /workspace + - name: Check File Permissions + run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 stat /workspace/Languages/Python/Example1/run.slurm - - name: Run SLURM job in Docker - run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ - sbatch /workspace/Languages/Python/Example1/run.slurm \ No newline at end of file + - name: Run SLURM job in Docker + run: | + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ + sbatch /workspace/Languages/Python/Example1/run.slurm \ No newline at end of file From e240fded271ef8f605e2de689062f513df2f6262 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:24:43 -0400 Subject: [PATCH 15/19] Typo --- .github/workflows/Languages-Python-Example1.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index c1689aca..58fa259f 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -55,4 +55,4 @@ jobs: - name: Run SLURM job in Docker run: | docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ - sbatch /workspace/Languages/Python/Example1/run.slurm \ No newline at end of file + sbatch /workspace/Languages/Python/Example1/run.sbatch \ No newline at end of file From de1dc375a840dbe4398010f5df7647ba1d7146e1 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 13:26:16 -0400 Subject: [PATCH 16/19] Typo --- .github/workflows/Languages-Python-Example1.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 58fa259f..c9a00aa7 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -50,7 +50,7 @@ jobs: - name: Check File Permissions run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 stat /workspace/Languages/Python/Example1/run.slurm + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 stat /workspace/Languages/Python/Example1/run.sbatch - name: Run SLURM job in Docker run: | From 1af76e9ab5bacd2ce5c375509c1a297de61794db Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 16 Sep 2024 15:02:41 -0400 Subject: [PATCH 17/19] Update docker commands --- .github/workflows/Languages-Python-Example1.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index c9a00aa7..89694971 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -54,5 +54,15 @@ jobs: - name: Run SLURM job in Docker run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ - sbatch /workspace/Languages/Python/Example1/run.sbatch \ No newline at end of file + # docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ + docker run --rm --detach \ + --name "${USER}_simulator" \ + -h "slurm-simulator" \ + --security-opt seccomp:unconfined \ + --privileged -e container=docker \ + -v /run -v /sys/fs/cgroup:/sys/fs/cgroup \ + -v ${{ github.workspace }}:/workspace \ + -w /workspace \ + --cgroupns=host \ + hpcnow/slurm_simulator:20.11.9 /usr/sbin/init + sbatch /workspace/Languages/Python/Example1/run.sbatch From 146769be47e414dec97f885d18fe0607a69fa1a6 Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Mon, 23 Sep 2024 18:11:57 -0400 Subject: [PATCH 18/19] Test Case for mc_pi.py --- Languages/Python/Example1/test_mc_pi.py | 35 +++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 Languages/Python/Example1/test_mc_pi.py diff --git a/Languages/Python/Example1/test_mc_pi.py b/Languages/Python/Example1/test_mc_pi.py new file mode 100644 index 00000000..6f137316 --- /dev/null +++ b/Languages/Python/Example1/test_mc_pi.py @@ -0,0 +1,35 @@ +# test_mc_pi.py +import unittest +import mc_pi +import random + +class TestMCPi(unittest.TestCase): + def test_pi_estimate(self): + # Set a seed for reproducibility + random.seed(0) + N = 100000 + count = 0 + for i in range(N): + x = random.random() + y = random.random() + z = x*x + y*y + if z <= 1.0: + count += 1 + PI = 4.0 * count / N + self.assertAlmostEqual(PI, 3.1415926535897932, places=2) + + def test_count_inside_circle(self): + # Set a seed for reproducibility + random.seed(0) + N = 100000 + count = 0 + for i in range(N): + x = random.random() + y = random.random() + z = x*x + y*y + if z <= 1.0: + count += 1 + self.assertEqual(count, 78539) # This value is based on the seed + +if __name__ == '__main__': + unittest.main() \ No newline at end of file From 71c1e20595c54913a7d85aee1f46889b6a7192bf Mon Sep 17 00:00:00 2001 From: Austin Swinney Date: Tue, 22 Oct 2024 20:00:52 -0400 Subject: [PATCH 19/19] Update Languages-Python-Example1.yml slurm_simulator version --- .github/workflows/Languages-Python-Example1.yml | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/.github/workflows/Languages-Python-Example1.yml b/.github/workflows/Languages-Python-Example1.yml index 89694971..5513bb7f 100644 --- a/.github/workflows/Languages-Python-Example1.yml +++ b/.github/workflows/Languages-Python-Example1.yml @@ -42,19 +42,18 @@ jobs: - name: Pull SLURM Simulator Docker Image run: | - docker pull hpcnow/slurm_simulator:20.11.9 + docker pull hpcnow/slurm_simulator:24.05.1 - name: List Files in Mounted Directory run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 ls -la /workspace/Languages/Python/Example1 + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:24.05.1 ls -la /workspace/Languages/Python/Example1 - name: Check File Permissions run: | - docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 stat /workspace/Languages/Python/Example1/run.sbatch + docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:24.05.1 stat /workspace/Languages/Python/Example1/run.sbatch - name: Run SLURM job in Docker run: | - # docker run --rm -v ${{ github.workspace }}:/workspace -w /workspace hpcnow/slurm_simulator:20.11.9 \ docker run --rm --detach \ --name "${USER}_simulator" \ -h "slurm-simulator" \ @@ -64,5 +63,5 @@ jobs: -v ${{ github.workspace }}:/workspace \ -w /workspace \ --cgroupns=host \ - hpcnow/slurm_simulator:20.11.9 /usr/sbin/init + hpcnow/slurm_simulator:24.05.1 /usr/sbin/init sbatch /workspace/Languages/Python/Example1/run.sbatch