From 998c3f9f714e1917d0c435353bd96cdacda1a7bc Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 19 Nov 2024 10:19:52 +0100 Subject: [PATCH 1/4] refactor: make ref checkout for picasso scripts configurable This change makes the use of versions different than default possible, so the code available matches the version of the workflow used. --- .github/workflows/build.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index dab9f6c..c3e9ccc 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -34,6 +34,11 @@ on: required: false type: string default: '3.12' + PICASSO_VERSION: + description: 'Picasso version to use for the workflow scripts and utility functions. This should be a valid branch, tag or commit and it should match the version of the workflow used.' + required: false + type: string + default: 'v1' secrets: DOCKERHUB_USERNAME: description: 'DockerHub username for login' @@ -70,7 +75,7 @@ jobs: uses: actions/checkout@v4 with: repository: edunext/picasso - ref: v1 + ref: ${{ inputs.PICASSO_VERSION }} path: picasso - name: Checkout strains repository for build configurations From f186a2cb84b7d18b3d6759a141cdf2eb8220dc0a Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 25 Nov 2024 18:25:31 +0100 Subject: [PATCH 2/4] docs: update docs with latest inputs --- README.rst | 12 ++++++++++++ docs/source/how_tos/configure_workflow.rst | 6 ++++++ docs/source/topic_guides/reusable_build_workflow.rst | 5 +++-- 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 4a9dacc..90989d0 100644 --- a/README.rst +++ b/README.rst @@ -88,6 +88,18 @@ Before using the workflow, ensure that you have set up the following configurati - Enables limiting parallelism with buildkit to decrease resource consumption for those setups with low-powered machines. Default is ``true``. - boolean - Input + * - RUNNER_WORKFLOW_LABEL (Optional) + - The label of the runner workflow to use. Default is ``ubuntu-24.04``. + - string + - Input + * - PYTHON_VERSION (Optional) + - The Python version to use in the workflow. Default is ``3.12``. + - string + - Input + * - PICASSO_VERSION (Optional) + - Picasso version to use for the workflow scripts and utility functions. This should be a valid branch, tag or commit and it should match the version of the workflow used. Default is the latest release major version, e.g., ``v1``. + - string + - Input Usage ***** diff --git a/docs/source/how_tos/configure_workflow.rst b/docs/source/how_tos/configure_workflow.rst index 1ac8d6b..6d53e27 100644 --- a/docs/source/how_tos/configure_workflow.rst +++ b/docs/source/how_tos/configure_workflow.rst @@ -15,6 +15,9 @@ Inputs are the parameters that you can set to customize the behavior of the Pica * ``STRAIN_PATH (required)``: The path to the directory that contains the strain configuration file. This path should be relative to the root of the repository. * ``SERVICE (required)``: The name of the service that will be built. This service should be supported by Tutor or by a tutor plugin previously installed. * ``ENABLE_LIMIT_BUILDKIT_PARALLELISM (optional)``: If set to ``true``, the build process parallel steps will be limited by 3, which is the threshold found where both Open edX and MFE images are built without running out of resources in the Github Actions runner (please, see `PR #12`_ for more details) . If set to ``false``, the buildkit configuration default will be used. Default is ``true``. Set to ``false`` if you have a runner with more resources. +* ``RUNNER_WORKFLOW_LABEL (optional)``: This label should match the label of the runner that will execute the workflow. By default, the workflow will use the `Github Actions standard runners`_. +* ``PYTHON_VERSION (optional)``: The Python version to use in the workflow. By default, the workflow will use Python 3.12. +* ``PICASSO_VERSION (optional)``: Picasso version to use for the workflow scripts and utility functions. This should be a valid branch, tag or commit and it should match the version of the workflow used. By default, the workflow will use the latest release major version, e.g., ``v1``. These inputs can be set in the workflow file that calls the Picasso workflow using the ``with`` keyword, by manually setting them in the workflow file, or by using the ``workflow_dispatch`` event. For more details on how to set input values in Github Actions, please refer to the `Workflow syntax for GitHub Actions`_ documentation. @@ -34,6 +37,9 @@ Here is an example of how to use the Picasso workflow with the inputs set: STRAIN_PATH: redwood/base SERVICE: mfe ENABLE_LIMIT_BUILDKIT_PARALLELISM: false + RUNNER_WORKFLOW_LABEL: ubuntu-latest + PYTHON_VERSION: 3.10 + PICASSO_VERSION: main .. _`PR #12`: https://github.com/eduNEXT/picasso/pull/12 diff --git a/docs/source/topic_guides/reusable_build_workflow.rst b/docs/source/topic_guides/reusable_build_workflow.rst index 6ccac85..52d07da 100644 --- a/docs/source/topic_guides/reusable_build_workflow.rst +++ b/docs/source/topic_guides/reusable_build_workflow.rst @@ -22,14 +22,15 @@ The Picasso Workflow is designed to build Open edX Docker images for Tutor envir Key features of the Picasso Workflow include: -- **Runs on GitHub-hosted runners**: By default, the workflow uses ``ubuntu-latest`` runners to execute jobs. This can't be changed to self-hosted runners for the time being. +- **Runs on GitHub-hosted runners**: By default, the workflow uses GitHub hosted runners to execute jobs. This can't be changed to self-hosted runners for the time being. - **Builds and pushes Docker images**: The workflow pushes images to Dockerhub by default. This can be customized to push images to other registries. - **Supports multiple services**: You can specify the service to build (e.g., ``openedx``, ``mfe``, ``codejail``, etc.) using the ``SERVICE`` input. - **Customizable repository and strain**: The workflow allows for specifying the repository, branch, and path to the strain being built. This enables building images from different configurations. - **Configurable BuildKit parallelism**: By default, the workflow limits parallelism during the build process to optimize resource usage, although this can be changed using the ``ENABLE_LIMIT_BUILDKIT_PARALLELISM`` input. This is useful for low-powered machines, like `Github Actions standard runners`_. - **Private repository access**: SSH keys are used to clone private repositories securely. The SSH private key should be stored as a secret in the repository, and must have access to the repository specified in ``STRAIN_REPOSITORY``. +- **Configures docker registry**: The workflow sets up the Docker registry to push images to Dockerhub or AWS ECR. - **Extra commands**: The workflow allows running additional custom commands with ``tutor picasso run-extra-commands``. For details, refer to the `tutor-contrib-picasso`_ documentation. -- **Environment setup**: The workflow sets up and configures Tutor Virtual Environments (TVM), installs necessary plugins like ``tutor-contrib-picasso``, and prepares the environment to build and push Docker images using the `Tutor CLI`_. +- **Environment setup**: The workflow sets up installs necessary plugins like ``tutor-contrib-picasso``, and prepares the environment to build and push Docker images using the `Tutor CLI`_. .. _tutor-contrib-picasso: https://github.com/eduNEXT/tutor-contrib-picasso/ .. _Github Actions standard runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners From 199bb1aeaf0e79f72cfdcbb137e9c3cb3cdcab73 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 25 Nov 2024 18:30:35 +0100 Subject: [PATCH 3/4] fix: add missing reference to gh standard runners --- docs/source/how_tos/configure_workflow.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/how_tos/configure_workflow.rst b/docs/source/how_tos/configure_workflow.rst index 6d53e27..b7c0d78 100644 --- a/docs/source/how_tos/configure_workflow.rst +++ b/docs/source/how_tos/configure_workflow.rst @@ -15,7 +15,7 @@ Inputs are the parameters that you can set to customize the behavior of the Pica * ``STRAIN_PATH (required)``: The path to the directory that contains the strain configuration file. This path should be relative to the root of the repository. * ``SERVICE (required)``: The name of the service that will be built. This service should be supported by Tutor or by a tutor plugin previously installed. * ``ENABLE_LIMIT_BUILDKIT_PARALLELISM (optional)``: If set to ``true``, the build process parallel steps will be limited by 3, which is the threshold found where both Open edX and MFE images are built without running out of resources in the Github Actions runner (please, see `PR #12`_ for more details) . If set to ``false``, the buildkit configuration default will be used. Default is ``true``. Set to ``false`` if you have a runner with more resources. -* ``RUNNER_WORKFLOW_LABEL (optional)``: This label should match the label of the runner that will execute the workflow. By default, the workflow will use the `Github Actions standard runners`_. +* ``RUNNER_WORKFLOW_LABEL (optional)``: This label should match the label of the runner that will execute the workflow. By default, the workflow will use one of the `Github Actions standard runners`_, e.g., ``ubuntu-24.04``. * ``PYTHON_VERSION (optional)``: The Python version to use in the workflow. By default, the workflow will use Python 3.12. * ``PICASSO_VERSION (optional)``: Picasso version to use for the workflow scripts and utility functions. This should be a valid branch, tag or commit and it should match the version of the workflow used. By default, the workflow will use the latest release major version, e.g., ``v1``. @@ -99,3 +99,5 @@ Or using an AWS registry: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_REGION: ${{ secrets.AWS_REGION }} + +.. _`Github Actions standard runners`: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories From fd92f681c3507b96b70b48712a0ada43b7985f38 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Fri, 29 Nov 2024 19:24:03 +0100 Subject: [PATCH 4/4] refactor: address PR reviews --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 90989d0..13617cb 100644 --- a/README.rst +++ b/README.rst @@ -26,7 +26,7 @@ Key features of the Picasso Workflow include: - **Configurable BuildKit parallelism**: By default, the workflow limits parallelism during the build process to optimize resource usage, although this can be changed using the ``ENABLE_LIMIT_BUILDKIT_PARALLELISM`` input. This is useful for low-powered machines, like `Github Actions standard runners`_. - **Private repository access**: SSH keys are used to clone private repositories securely. The SSH private key should be stored as a secret in the repository, and must have access to the repository specified in ``STRAIN_REPOSITORY``. - **Extra commands**: The workflow allows running additional custom commands with ``tutor picasso run-extra-commands``. For details, refer to the `tutor-contrib-picasso`_ documentation. -- **Environment setup**: The workflow sets up and configures Tutor Virtual Environments (TVM), installs necessary plugins like ``tutor-contrib-picasso``, and prepares the environment to build and push Docker images using the `Tutor CLI`_. +- **Environment setup**: The workflow sets up installs necessary plugins like ``tutor-contrib-picasso``, and prepares the environment to build and push Docker images using the `Tutor CLI`_. .. _tutor-contrib-picasso: https://github.com/eduNEXT/tutor-contrib-picasso/ .. _Github Actions standard runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners