Skip to content

Commit

Permalink
Renamed all occurences of containers/*.json to config/*.json
Browse files Browse the repository at this point in the history
  • Loading branch information
CodeGat committed Sep 28, 2023
1 parent 27601f4 commit 49390eb
Show file tree
Hide file tree
Showing 6 changed files with 19 additions and 19 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/dep-image-1-start.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ jobs:
id: get-model
run: |
if [[ "${{ inputs.model }}" = "all" ]]; then
echo "models=$(jq -cr 'keys' containers/models.json)" >> $GITHUB_OUTPUT
echo "models=$(jq -cr 'keys' config/models.json)" >> $GITHUB_OUTPUT
else
query=$(jq -cr '."${{ inputs.model }}"' containers/models.json)
query=$(jq -cr '."${{ inputs.model }}"' config/models.json)
if [ -z "$query" ]; then
exit 1
else
Expand All @@ -38,7 +38,7 @@ jobs:
fi
- name: Determine compilers to build
id: get-compiler
run: echo "compilers=$(jq -c . containers/compilers.json)" >> $GITHUB_OUTPUT
run: echo "compilers=$(jq -c . config/compilers.json)" >> $GITHUB_OUTPUT

dependency-image-workflow:
name: Create Dependency Image
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dep-image-3-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
steps:
- name: Get model components for ${{ inputs.model }}
id: get-model-components
run: echo "model-components=$(jq -cr '.["${{ inputs.model }}"]' containers/models.json)" >> $GITHUB_OUTPUT
run: echo "model-components=$(jq -cr '.["${{ inputs.model }}"]' config/models.json)" >> $GITHUB_OUTPUT

dependency-image:
name: Build build-${{ inputs.model }}-${{ inputs.compiler-name}}${{ inputs.compiler-version }}-${{ inputs.spack-packages-version }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/json-1-validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ jobs:
name: Validate
uses: access-nri/build-ci/.github/workflows/validate-json.yml@main
with:
src: "containers"
src: "config"
4 changes: 2 additions & 2 deletions .github/workflows/model-1-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,12 @@ jobs:
repository: access-nri/build-ci
- name: Get compilers to test
id: get-compilers
run: echo "compilers=$(jq -c . containers/compilers.json)" >> $GITHUB_OUTPUT
run: echo "compilers=$(jq -c . config/compilers.json)" >> $GITHUB_OUTPUT
- name: Get model to test
id: get-model
# model-components are associated with an overarching model (for example, cice5 is associated with access-om2), this uses models.json to find the associated model
run: |
model=$(jq -cr 'to_entries[] | select(.value | contains(["${{ needs.setup-model.outputs.package-name }}"])) | .key' containers/models.json)
model=$(jq -cr 'to_entries[] | select(.value | contains(["${{ needs.setup-model.outputs.package-name }}"])) | .key' config/models.json)
if [ -z "$model" ]; then
exit 1
else
Expand Down
22 changes: 11 additions & 11 deletions README-DEV.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ How they are used can be found in the [CI Run Through section](#ci-workflow-run-
This pipeline has explicit inputs, defined in the [`on.workflow_call.inputs` section](https://github.com/ACCESS-NRI/build-ci/blob/1b998192946c364fb75040e6807e7143c9123527/.github/workflows/dep-image-1-start.yml#L5-L15):

* `spack-packages-version`: A tag or branch of the `access-nri/spack_packages` [repository](https://github.com/ACCESS-NRI/spack_packages). This allows provenance of the build process of models.
* `model`: a coupled model name (such as `access-om2` or `access-om3`) or `all` if we want to build dependency images for all coupled models defined in `containers/models.json`.
* `model`: a coupled model name (such as `access-om2` or `access-om3`) or `all` if we want to build dependency images for all coupled models defined in `config/models.json`.

It also indirectly uses:

* [`containers/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/compilers.json): This is a data structure containing all the compilers we want to test against.
* [`containers/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json): This is a data structure containing all the coupled models (and their associated model components) that we want to test against.
* [`config/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/compilers.json): This is a data structure containing all the compilers we want to test against.
* [`config/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json): This is a data structure containing all the coupled models (and their associated model components) that we want to test against.
* [`containers/Dockefile.base-spack`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/Dockerfile.base-spack), [`containers/Dockerfile.dependency`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/Dockerfile.dependency) : uses these Dockerfiles to create the `base-spack` and `dependency` images.

#### Outputs
Expand All @@ -43,8 +43,8 @@ There are no explicit inputs to this workflow. The information required is infer
However, there are indirect inputs into this pipeline:

* Appropriate `dependency` Docker images of the form: `ghcr.io/access-nri/build-<coupled model>-<compiler name><compiler version>-<spack_packages version>:latest`.
* [`containers/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/compilers.json): This is a data structure containing all the compilers we want to test against.
* [`containers/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json): This is a data structure containing all the coupled models (and their associated model components) that we want to test against.
* [`config/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/compilers.json): This is a data structure containing all the compilers we want to test against.
* [`config/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json): This is a data structure containing all the coupled models (and their associated model components) that we want to test against.

#### Outputs

Expand All @@ -53,7 +53,7 @@ However, there are indirect inputs into this pipeline:

### JSON Validator Pipeline

This workflow finds all the [JSON Schema](https://json-schema.org/) files in the project, i.e. those with the '.schema.json' extension, and then runs [`jsonschema`](https://pypi.org/project/jsonschema/) on the matching json files. e.g. `containers/model.json` is validated against `containers/models.schema.json`.
This workflow finds all the [JSON Schema](https://json-schema.org/) files in the project, i.e. those with the '.schema.json' extension, and then runs [`jsonschema`](https://pypi.org/project/jsonschema/) on the matching json files. e.g. `config/model.json` is validated against `config/models.schema.json`.

#### Inputs

Expand All @@ -76,7 +76,7 @@ The rationale for this pipeline is the creation of a model-dependency docker ima

As an overview, this workflow, given a `access-nri/spack_packages` repo version and coupled model(s):

* Generates a staggered `compiler x model` matrix based on the [`compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/maine/containers/compilers.json) and [`models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json). This allows generation and testing of multiple different compiler and model image combinations in parallel.
* Generates a staggered `compiler x model` matrix based on the [`compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/maine/config/compilers.json) and [`models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json). This allows generation and testing of multiple different compiler and model image combinations in parallel.
* Uses an existing `base-spack` docker image (or creates it if it doesn't exist) that contains an install of spack, `access-nri/spack_packages` and a given compiler.
* Using the above `base-spack` image, creates a spack-based model-dependency docker image that separates each model (and it's components) into `spack env`s. This has all the dependencies of the model installed, but not the model itself.

Expand All @@ -103,7 +103,7 @@ This pipeline begins at [`dep-image-1-start.yml`](https://github.com/ACCESS-NRI/
dep-image-1-start.yml [compilers c1 c2]
```

This workflow is responsible for generating the matrix of compilers (from `containers/compilers.json`) and the information necessary for creating a matrix of coupled models for a future matrix (from `models.json`).
This workflow is responsible for generating the matrix of compilers (from `config/compilers.json`) and the information necessary for creating a matrix of coupled models for a future matrix (from `models.json`).

Rather than doing the `compiler x model` matrix at the beginning of the workflow, we do the model matrix later in a staggered approach. We do this because it removes duplication of effort and effectively uses the cache, rather than thrashing it. The differences between a `compiler x model` and a staggered `compiler` then `matrix` model strategy are explained below.

Expand Down Expand Up @@ -159,7 +159,7 @@ Finally, with the `base-spack` image created, and the models that need to be bui

In the `dep-image-3-build.yml` workflow, we:

* Get the associated model components of our coupled model from the [`containers/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json) file.
* Get the associated model components of our coupled model from the [`config/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json) file.
* Build and push the dependency image given the existing `base-spack` image as a base, and the list of model components from the previous job, using the reusable [`build-docker-image.yml`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/build-docker-image.yml) workflow.

This leads to a final pipeline looking like the following:
Expand Down Expand Up @@ -191,8 +191,8 @@ This workflow begins by inferring the 'appropriate dependency image' based on a
In order to find:

* `spack_packages version`: In the [`setup-spack-packages`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L8-L26) job, we take the latest tagged version of the `access-nri/spack_packages` repo.
* `coupled model`: In the [`setup-model`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L28-L37) and [`setup-build-ci`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L39-L63) jobs, we use the name of the calling repository (eg. `cice5`) and `build-ci`s [`containers/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json) to infer the overarching `coupled model` name.
* `compiler name`/`compiler version`: In the [`setup-build-ci`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L39-L63) job, we use all compilers from the [`containers/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/compilers.json) file. This would mean that another matrix strategy would be in order.
* `coupled model`: In the [`setup-model`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L28-L37) and [`setup-build-ci`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L39-L63) jobs, we use the name of the calling repository (eg. `cice5`) and `build-ci`s [`config/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json) to infer the overarching `coupled model` name.
* `compiler name`/`compiler version`: In the [`setup-build-ci`](https://github.com/ACCESS-NRI/build-ci/blob/main/.github/workflows/model-1-build.yml#L39-L63) job, we use all compilers from the [`config/compilers.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/compilers.json) file. This would mean that another matrix strategy would be in order.

Given those inferences, we are able to find the appropriate dependency images to use to build the modified models .

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ If you want to use the Model Test Pipeline go to the repo, then the `Actions` ta
Model must meet these requirements:

- Be [available as a spack package](https://github.com/ACCESS-NRI/spack_packages/tree/main/packages) in the [`access-nri/spack_packages` repo](https://github.com/ACCESS-NRI/spack_packages)
- Have an entry in [`containers/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/containers/models.json) in this repo
- Have an entry in [`config/models.json`](https://github.com/ACCESS-NRI/build-ci/blob/main/config/models.json) in this repo

### Create your own Dependency Images

Expand Down

0 comments on commit 49390eb

Please sign in to comment.