Add level2 ingestion into repo

.github/workflows/ci.yml

@@ -17,6 +17,7 @@ jobs:
         source venv/bin/activate
         uv pip install -r requirements.txt
         uv pip install --upgrade --force-reinstall "numpy<2.0"
+        uv pip install -e ".[dev]"
 
     - name: Run tests
       run: |

.gitignore

@@ -171,4 +171,7 @@ dask-scratch-space
 
 fairmast-*.e*
 fairmast-*.o*
-.s5cfg*
+fair-mast-*.e*
+fair-mast-*.o*
+.s5cfg*
+*.db

README.md

@@ -1,14 +1,28 @@
 ### FAIR MAST Data Ingestion
 
-## Running on CSD3
-### Installation on CSD3
+## Project Structure
 
-After logging into your CSD3 account (on Icelake node), first load the correct Python module:
-
-```sh
-module load python/3.9.12/gcc/pdcqf4o5
+Below is a brief overview of the project structure
+```
+|-- campaign_shots      # CSV lists of shots for each MAST campaign
+|-- configs             # Config files for each level of ingestion
+|-- geometry            # Geometry data files for each diagnostic source 
+|-- jobs                # Job scripts for different HPC machines
+|-- mappings            # Mapping files for transforming units, names, dimensions, etc.
+|-- notebooks           # Notebooks for checking outputs
+|-- scripts             # Misc scripts for metadata curation
+|-- src                 # Source code for ingestion tools
+|   |-- core            # Core modules for ingestion, shared between all levels
+|   |-- level1          # Level1 data ingestion code
+|   |-- level2          # Level2 data ingestion code
+`-- tests               # Unit tests
+    |-- core            # Core module unit tests
+    |-- level1          # Level1 module unit tests
+    |-- level2          # Level2 module unit tests
 ```
 
+## Installation and Setup
+
 Clone the repository and fetch data files (Git LFS must be installed):
 
 ```sh
@@ -17,120 +31,82 @@ cd fair-mast-ingestion
 git lfs pull
 ```
 
-Create a virtual environment:
+Create a new python virtual environment:
 
 ```sh
-python -m venv fair-mast-ingestion
-source fair-mast-ingestion/bin/activate
+uv venv --python 3.12.6 
+source .venv/bin/activate
 ```
 
 Update pip and install required packages:
 
 ```sh
-python -m pip install -U pip
-python -m pip install -e .
+uv pip install git+ssh://[email protected]/MAST-U/mastcodes.git#subdirectory=uda/python
+uv pip install -e .
+uv pip install -e ".[dev]"
+uv pip install -e ".[mpi]"
 ```
 
-The final step to installation is to have mastcodes:
+If running on CSD3, we must also source the SSL certificate information by running the following command. Without this UDA cannot connect to the UKAEA network.
 
 ```sh
-git clone [email protected]:MAST-U/mastcodes.git
-cd mastcodes
-```
-
-Edit `uda/python/setup.py` and change the "version" to 1.3.9.
-
-```sh
-python -m pip install uda/python
-cd ..
 source ~/rds/rds-ukaea-ap002-mOlK9qn0PlQ/fairmast/uda-ssl.sh
 ```
 
-#### S3 Support (Optional)
-
-Finally, for uploading to S3 we need to install `s5cmd` and make sure it is on the path:
-
-```sh
-wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
-tar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz
-PATH=$PWD:$PATH
-```
-
-And add a config file for the bucket keys, by creating a file called `.s5cfg.stfc`:
+Finally, for uploading to S3 we need to create a local config file with the bucket keys. Create a file called `.s5cfg.stfc` with the following information:
 
 ```
 [default]
 aws_access_key_id=<access-key>
 aws_secret_access_key=<secret-key>
 ```
 
-You should now be able to run the following commands.
-
-### Submitting runs on CSD3
-
-#### First Run on CSD3
-
-This will ingest data into the test folder in S3. The small_ingest script allows you to put one file of shots into the ingestion.
-
-1. First submit a job to collect all the metadata:
 
-```sh
-sbatch ./jobs/metadata.csd3.slurm.sh
-```
+## Running Ingestion
 
-2. Then submit an ingestion job
+The following section details how to ingest data into a local folder with UDA. 
 
-Argument 1 (e.g. s3://mast/test/shots/) is where the data will ingest to, and argument 2 is the file of shots to ingest (e.g. campaign_shots/tiny_campaign.csv), arguments 3 and greater are the sources (e.g. amc)
+First you must edit both the config files in `./configs/` to point the writer `output_path` at a sensible location:
 
-```sh
-sbatch ./jobs/small_ingest.csd3.slurm.sh s3://mast/test/shots/ campaign_shots/tiny_campaign.csv amc
+```yaml
+...
+writer:
+  type: "zarr"
+  options:
+    zarr_format: 2
+    output_path: "/common/tmp/sjackson/upload-tmp/zarr/level1"
+...
 ```
 
-#### Ingesting All Shots
+### Level 1 Ingestion
 
-This ingestion job runs through all shots for the specified source (e.g. amc)
+Below gives an example of running a level 1 ingestion which will write `ayc` data for shot `30421` from MAST.
 
 ```sh
-sbatch ./jobs/small_ingest.csd3.slurm.sh s3://mast/test/shots/ amc
+mpirun -n 4 python3 -m src.level1.main -v --facility MAST --shot 30421 -i ayc
 ```
 
-## Manually Running Ingestor
-
-### Local Ingestion
-
-The following section details how to ingest data into a local folder on freia with UDA.
-
-1. Parse the metadata for all signals and sources for a list of shots with the following command
-
-```sh
-mpirun -n 16 python3 -m src.create_uda_metadata data/uda campaign_shots/tiny_campaign.csv 
-```
+### Level 2 Ingestion
 
+Below gives an example of running a level 2 ingestion which will write `thomson_scattering` data for shot `30421` from MAST.
 ```sh
-mpirun -np 16 python3 -m src.main data/local campaign_shots/tiny_campaign.csv --metadata_dir data/uda --source_names amc xsx --file_format nc
+mpirun -n 4 python3 -m src.level2.main mappings/level2/mast.yml -v --shot 30421 -i thomson_scattering
 ```
 
-Files will be output in the NetCDF format to `data/local`.
-
 ### Ingestion to S3
 
-The following section details how to ingest data into the s3 storage on freia with UDA.
-
-1. Parse the metadata for all signals and sources for a list of shots with the following command
-
-```sh
-mpirun -n 16 python3 -m src.create_uda_metadata data/uda campaign_shots/tiny_campaign.csv 
-```
-
-This will create the metadata for the tiny campaign. You may do the same for full campaigns such as `M9`.
+To ingest to S3 you must edit the config files in `./configs` to include the an upload entry specifying the endpoint and location to upload data to.
+For example the following config sets the base path and endpoint url for object storage at CSD3:
 
-2. Run the ingestion pipleline by submitting the following job:
-
-```sh
-mpirun -np 16 python3 -m src.main data/local campaign_shots/tiny_campaign.csv --bucket_path s3://mast/test/shots --source_names amc xsx --file_format zarr --upload --force
+```yaml
+upload:
+  base_path: "s3://mast/test/level1/shots"
+  mode: 's5cmd'
+  credentials_file: ".s5cfg.csd3"
+  endpoint_url: "https://object.arcus.openstack.hpc.cam.ac.uk"
 ```
 
-This will submit a job to the freia job queue that will ingest all of the shots in the tiny campaign and push them to the s3 bucket.
+Then simple rerun the commands as above.
 
 ## CPF Metadata
 

configs/level1.yml

@@ -0,0 +1,25 @@
+metadatabase_file: /common/tmp/sjackson/level1.ukaea.db
+readers:
+  uda:
+    type: 'uda'
+  fairmast:
+    type: 'zarr'
+    options:
+      base_path: 's3://mast/level1/shots'
+      protocol: 'simplecache'
+      target_protocol: "s3"
+      target_options:
+          anon: True 
+          endpoint_url: "https://s3.echo.stfc.ac.uk"
+
+writer:
+  type: "zarr"
+  options:
+    zarr_format: 2
+    output_path: "/common/tmp/sjackson/upload-tmp/zarr/level1"
+
+# upload:
+#   base_path: "s3://fairmast/mastu/level1/shots"
+#   mode: 's5cmd'
+#   credentials_file: "../fair-mast-ingestion/.s5cfg.ukaea"
+#   endpoint_url: "http://mon3.cepheus.hpc.l:8000"

configs/level2.yml

@@ -0,0 +1,27 @@
+metadatabase_file: /common/tmp/sjackson/level2.stfc.db
+readers:
+  uda:
+    type: 'uda'
+  fairmast:
+    type: 'zarr'
+    options:
+      base_path: 's3://mast/level1/shots'
+      protocol: 'simplecache'
+      target_protocol: "s3"
+      target_options:
+          anon: True 
+          endpoint_url: "https://s3.echo.stfc.ac.uk"
+
+writer:
+  type: "zarr"
+  options:
+    zarr_format: 2
+    # output_path: "/common/tmp/sjackson/upload-tmp/zarr/mast/level2/"
+    output_path: "./data-test"
+
+# upload:
+#   base_path: "s3://fairmast/mastu/level2/shots"
+#   mode: 's5cmd'
+#   credentials_file: "../fair-mast-ingestion/.s5cfg.ukaea"
+#   endpoint_url: "http://mon3.cepheus.hpc.l:8000"
+

jobs/consolidate_job.freia.qsub.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Choose parallel environment
+#$ -pe mpi 8
+
+# Specify the job name in the queue system
+#$ -N fairmast-consolidate
+
+# Start the script in the current working directory
+#$ -cwd
+
+# Time requirements
+#$ -l h_rt=48:00:00
+#$ -l s_rt=48:00:00
+
+source .venv/bin/activate
+
+num_workers=8
+bucket_path="s3://mast/level1/shots/"
+local_path="/common/tmp/sjackson/fair-mast/consolidate"
+
+python3 -m scripts.consolidate_s3 $bucket_path $local_path -n $num_workers --start-shot 11695 --end-shot 14830

jobs/consolidate_job.slurm.sh

@@ -12,6 +12,6 @@ num_workers=$SLURM_NTASKS
 
 bucket_path="s3://mast/level1/shots/"
 local_path="/rds/project/rds-mOlK9qn0PlQ/fairmast"
-S
+
 mpirun -n $num_workers \
-    python3 -m src.consolidate_s3 $bucket_path $local_path
+    python3 -m scripts.consolidate_s3 $bucket_path $local_path

jobs/cpf_metadata.freia.qsub

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# Choose parallel environment
+#$ -pe mpi 16
+
+# Specify the job name in the queue system
+#$ -N fairmast-cpf-writer
+
+# Start the script in the current working directory
+#$ -cwd
+
+# Time requirements
+#$ -l h_rt=48:00:00
+#$ -l s_rt=48:00:00
+
+source .venv/bin/activate
+
+# Run script
+python3 -m scripts.create_cpf_metadata mast --shot-min 11695 --shot-max 30475
+python3 -m scripts.create_cpf_metadata mastu --shot-min 41139 --shot-max 51056