From 8338a26958c35da4cb5a0a0f61a95cbc52096688 Mon Sep 17 00:00:00 2001 From: Jingru Feng Date: Thu, 11 Apr 2024 09:20:04 +0200 Subject: [PATCH 1/4] add a page for ci --- docs/_quarto.yml | 1 + docs/contribute/ci.qmd | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 docs/contribute/ci.qmd diff --git a/docs/_quarto.yml b/docs/_quarto.yml index 4e3000ea8..e3cd0e3e5 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -46,6 +46,7 @@ website: - contribute/python.qmd - contribute/qgis.qmd - contribute/addnode.qmd + - contribute/ci.qmd - contribute/release.qmd format: diff --git a/docs/contribute/ci.qmd b/docs/contribute/ci.qmd new file mode 100644 index 000000000..4ad4e0cab --- /dev/null +++ b/docs/contribute/ci.qmd @@ -0,0 +1,38 @@ +--- +title: "Continuous integration" +--- + +Continuous integration is about commits being merged frequently and automatically. When a new commit is made, a series of tests will be done to make sure that this commit is error-free, appliance format, robust in difference environment and safe to be merged. This process drive each new development through building, testing, quality checking, it is like a pipeline. + +This page contains an extensive explanation on how the Ribasim pipeline works. + +# Github actions +When a new development is to be merged into the main branch, one can either make a pull request or commit to main branch directly. Before a commit is added to main branch, a series of GitHub action will be performed. Github actions includes following checks: + +* Julia Run Testmodels: This is to make sure all the test models can be run successfully +* Julia Tests: Run Julia unit tests on multiple platforms, depends on generated testmodels +* Docs: Executes our documentation examples +* Pre-commit: Checks code quality with linters, adjust the format +* Python codegen: Check that code generation wouldn't change any files +* Python lint (Run mypy on our Python codebase) TODO: Rename "Python Lint" to "Mypy" and "python_lint.yml" to "mypy.yml" +* Ribasim Python tests (Run Ribasim Python tests on multiple platforms and multiple Python versions, depends on ribasim_testmodels) +* QGIS Tests (Run QGIS tests, very limited at the moment) + +```{mermaid} +graph TD + A[New development]-->B[Github actions] + B-->C[Merge] +``` + +# TeamCity +Ribasim has another cloud-based CI/CD pipeline that is in the TeamCity. But TeamCity is not always triggered. TeamCity is only used in the following circumustance: + +* When it takes too long to run on Github Action, + +```{mermaid} +graph TD + A-->B + A-->C + B-->D + C-->D +``` From fa0ea420b29007f0aedfb6181051eec9daf4c205 Mon Sep 17 00:00:00 2001 From: Jingru Feng Date: Fri, 12 Apr 2024 15:12:16 +0200 Subject: [PATCH 2/4] Teamcity first paragraph --- docs/contribute/ci.qmd | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/contribute/ci.qmd b/docs/contribute/ci.qmd index 4ad4e0cab..307775ca8 100644 --- a/docs/contribute/ci.qmd +++ b/docs/contribute/ci.qmd @@ -13,10 +13,10 @@ When a new development is to be merged into the main branch, one can either make * Julia Tests: Run Julia unit tests on multiple platforms, depends on generated testmodels * Docs: Executes our documentation examples * Pre-commit: Checks code quality with linters, adjust the format -* Python codegen: Check that code generation wouldn't change any files -* Python lint (Run mypy on our Python codebase) TODO: Rename "Python Lint" to "Mypy" and "python_lint.yml" to "mypy.yml" -* Ribasim Python tests (Run Ribasim Python tests on multiple platforms and multiple Python versions, depends on ribasim_testmodels) -* QGIS Tests (Run QGIS tests, very limited at the moment) +* Python codegen: Check file difference and make sure that code generation wouldn't change any files +* Mypy: Run mypy on our Python codebase to check types +* Ribasim Python tests: Run Ribasim Python tests on multiple platforms and multiple Python versions, depends on ribasim_testmodels +* QGIS Tests: Run QGIS unit tests ```{mermaid} graph TD @@ -27,12 +27,11 @@ graph TD # TeamCity Ribasim has another cloud-based CI/CD pipeline that is in the TeamCity. But TeamCity is not always triggered. TeamCity is only used in the following circumustance: -* When it takes too long to run on Github Action, +* When a action takes too long to run on Github Action +* When the release depends on the artifects +* When other TeamCity projects depend on artifacts of Ribasim (e.g. the coupler) ```{mermaid} graph TD - A-->B - A-->C - B-->D - C-->D + A[TeamCity]-->B[Github]:monitoring ``` From 1733d18484f25f00610c7da18e60384f2909b24e Mon Sep 17 00:00:00 2001 From: Jingru Feng Date: Mon, 15 Apr 2024 14:38:15 +0200 Subject: [PATCH 3/4] TeamCity part finished --- docs/contribute/ci.qmd | 44 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/docs/contribute/ci.qmd b/docs/contribute/ci.qmd index 307775ca8..9e6afd6b9 100644 --- a/docs/contribute/ci.qmd +++ b/docs/contribute/ci.qmd @@ -19,19 +19,53 @@ When a new development is to be merged into the main branch, one can either make * QGIS Tests: Run QGIS unit tests ```{mermaid} -graph TD +graph LR A[New development]-->B[Github actions] B-->C[Merge] ``` # TeamCity -Ribasim has another cloud-based CI/CD pipeline that is in the TeamCity. But TeamCity is not always triggered. TeamCity is only used in the following circumustance: +Ribasim has another cloud-based CI/CD pipeline that is in the TeamCity. TeamCity closely monitors Github and manage the release process. + +```{mermaid} +graph LR + A[TeamCity]-->|Monitoring|B[Github] + A-->C(Release) +``` + +But TeamCity is not always triggered by every new development. TeamCity is only used or triggered in the following circumustances: * When a action takes too long to run on Github Action -* When the release depends on the artifects +* When the release depends on the artifects. For example, a new build needs to be publish + + TeamCity constantly monitors and has the information of every commit and when a commit is with a tag starts with `v20`, it trigger the release process. + * When other TeamCity projects depend on artifacts of Ribasim (e.g. the coupler) +In the release, we publish the generated testmodels, ribasim cli in Windows and Linux, Ribasim QGIS, and source code + +When TeamCity detects an release tag, the release process will be triggered. TeamCity will start to pack the artifects. + +We have the following pipeline to generate artifects for releasing: + +* Generate Testmodels: produces generated_testmodels artifect which is part of release. +* Make GitHub Release: uses artifacts and it makes release +* Build libribasim: builds binary of Ribasim in Linux and Windows. The artifact is tested in `Test ribasim_api` and used by iMOD Coupler +* Build ribasim_cli: builds command line interface in Linux and Windows,its artifact is tested in `Test ribasim_cli` and used by release +* Test ribasim_api: tests libribasim artifact in Linux and Windows +* Test ribasim_cli: tests ribasim_cli artifact in Linux and Windows + +::: {.callout-note} +Make Github Release does not publish Test libribasim and Test ribasim_cli. It only publish artifect of Build libribasim and Build ribasim_cli if they past their tests. +::: + ```{mermaid} -graph TD - A[TeamCity]-->B[Github]:monitoring +graph LR + A[Make Github Release]-->B(Release) + F[Generate Testmodels]-->A + G[Make QGIS plugin]-->A + H[Build libribasim]---D[Test libribasim] + D-->A + C[Build ribasim_cli] --- E[Test ribasim_cli] + E-->A ``` From 9e42fd448344e0d78e6881e17148382e4d2f5efe Mon Sep 17 00:00:00 2001 From: Jingru Feng Date: Tue, 16 Apr 2024 16:09:39 +0200 Subject: [PATCH 4/4] adopt comments --- docs/contribute/ci.qmd | 72 +++++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 33 deletions(-) diff --git a/docs/contribute/ci.qmd b/docs/contribute/ci.qmd index 9e6afd6b9..5754dd4ed 100644 --- a/docs/contribute/ci.qmd +++ b/docs/contribute/ci.qmd @@ -2,30 +2,38 @@ title: "Continuous integration" --- -Continuous integration is about commits being merged frequently and automatically. When a new commit is made, a series of tests will be done to make sure that this commit is error-free, appliance format, robust in difference environment and safe to be merged. This process drive each new development through building, testing, quality checking, it is like a pipeline. +Continuous integration (CI) is about commits being merged frequently. +When proposing new changes to the code base a [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) is opened. +When a new commit in that pull request, a series of tests will be done to make sure that this commit is error-free and robust in different environments. +This process drive each new development through building, testing, quality checking. -This page contains an extensive explanation on how the Ribasim pipeline works. +```{mermaid} +graph LR + A[New development]-->B[Continuous integration] + B-->C[Merge] +``` + + +This page contains an extensive explanation on how the Ribasim continuous integration works. # Github actions -When a new development is to be merged into the main branch, one can either make a pull request or commit to main branch directly. Before a commit is added to main branch, a series of GitHub action will be performed. Github actions includes following checks: + +With [GitHub Actions](https://docs.github.com/en/actions), Github provides their own continuous integration service. +They include the following checks: * Julia Run Testmodels: This is to make sure all the test models can be run successfully -* Julia Tests: Run Julia unit tests on multiple platforms, depends on generated testmodels -* Docs: Executes our documentation examples -* Pre-commit: Checks code quality with linters, adjust the format -* Python codegen: Check file difference and make sure that code generation wouldn't change any files -* Mypy: Run mypy on our Python codebase to check types -* Ribasim Python tests: Run Ribasim Python tests on multiple platforms and multiple Python versions, depends on ribasim_testmodels -* QGIS Tests: Run QGIS unit tests +* Julia Tests: Runs Julia unit tests on multiple platforms +* Docs: Builds the documentation and executes the examples +* Pre-commit: Checks Python code quality with linters and enforces formatting +* Python codegen: Makes sure code generation is still up to date by checking that code generation wouldn't change any files +* Mypy: Runs mypy on our Python codebase to check types +* Ribasim Python tests: Runs Ribasim Python tests on multiple platforms and multiple Python versions +* QGIS Tests: Runs QGIS unit tests -```{mermaid} -graph LR - A[New development]-->B[Github actions] - B-->C[Merge] -``` # TeamCity -Ribasim has another cloud-based CI/CD pipeline that is in the TeamCity. TeamCity closely monitors Github and manage the release process. +Ribasim has another cloud-based CI service based on [TeamCity](https://www.jetbrains.com/teamcity/). +TeamCity interacts with Github and manages the release process. ```{mermaid} graph LR @@ -33,30 +41,28 @@ graph LR A-->C(Release) ``` -But TeamCity is not always triggered by every new development. TeamCity is only used or triggered in the following circumustances: - -* When a action takes too long to run on Github Action -* When the release depends on the artifects. For example, a new build needs to be publish - - TeamCity constantly monitors and has the information of every commit and when a commit is with a tag starts with `v20`, it trigger the release process. - -* When other TeamCity projects depend on artifacts of Ribasim (e.g. the coupler) +## Conditions of using TeamCity +TeamCity is not always triggered by every new development. +For a certain workflow, TeamCity is only used under the following conditions: -In the release, we publish the generated testmodels, ribasim cli in Windows and Linux, Ribasim QGIS, and source code +* When the workflow takes too long to run on Github Action +* When the release depends on the artifacts of the workflow. +* When other TeamCity projects depend on artifacts of Ribasim (e.g. iMOD coupler) -When TeamCity detects an release tag, the release process will be triggered. TeamCity will start to pack the artifects. +## Release process +In the release, we include the generated testmodels, Ribasim CLI on Windows and Linux, Ribasim QGIS, and the source code. We have the following pipeline to generate artifects for releasing: -* Generate Testmodels: produces generated_testmodels artifect which is part of release. -* Make GitHub Release: uses artifacts and it makes release -* Build libribasim: builds binary of Ribasim in Linux and Windows. The artifact is tested in `Test ribasim_api` and used by iMOD Coupler -* Build ribasim_cli: builds command line interface in Linux and Windows,its artifact is tested in `Test ribasim_cli` and used by release -* Test ribasim_api: tests libribasim artifact in Linux and Windows -* Test ribasim_cli: tests ribasim_cli artifact in Linux and Windows +* Generate Testmodels: produces generated_testmodels artifact which is part of the release. +* Make GitHub Release: uses artifacts and makes the release. TeamCity constantly monitors the Github repository. When a tag starts with `v20` is added, it triggers the release process. +* Build libribasim: builds library of Ribasim on Linux and Windows. The artifact is tested in `Test ribasim_api` and used by iMOD Coupler +* Build ribasim_cli: builds CLI application on Linux and Windows, its artifact is tested in `Test ribasim_cli` and used by release +* Test ribasim_api: tests libribasim artifact on Linux and Windows +* Test ribasim_cli: tests ribasim_cli artifact on Linux and Windows ::: {.callout-note} -Make Github Release does not publish Test libribasim and Test ribasim_cli. It only publish artifect of Build libribasim and Build ribasim_cli if they past their tests. +Make Github Release does not publish artifacts of "Test libribasim" and "Test ribasim_cli". It only publishes artifacts of "Build libribasim" and "Build ribasim_cli" if the beforementioned tests pass. ::: ```{mermaid}