Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add CI job to push cloud images to ECR #51

Merged
merged 1 commit into from
Jan 26, 2024
Merged

add CI job to push cloud images to ECR #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 183 additions & 0 deletions .github/actions/build-and-push-to-quay/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
name: 'Build and push to Quay'
description: 'Builds a container image and pushes it to our Quay organization'
inputs:
image_name:
description: 'The name of the image, not including the registry or the tag, for example "postgres"'
required: true
registry:
description: 'The name of the image, not including the registry or the tag, for example "postgres"'
required: false
default: "quay.io/coredb"
registry_tembo:
description: 'The name of the image, not including the registry or the tag, for example "postgres"'
required: false
default: "quay.io/tembo"
docker_directory:
description: 'The relative path to a directory in which there is a Dockerfile'
required: false
default: '.'
quay_user:
required: true
description: "Quay 'robot user' user name"
quay_password:
required: true
description: "Quay 'robot user' access token"
quay_user_tembo:
required: true
description: "Quay 'robot user' user name for Tembo org"
quay_password_tembo:
required: true
description: "Quay 'robot user' access token for Tembo org"
publish_calver:
description: 'Should we tag with calendar versioning?'
required: false
default: false
calver_suffix:
description: 'Optional suffix to the calendar version'
required: false
default: ""
publish_latest:
description: "Should we tag with 'latest'?"
required: false
default: false
tag_cargo_version_if_present:
description: "Should we tag with the version found in Cargo.toml, if found?"
required: false
default: false
tags:
description: "Whitespace-separated tags, not including the registry, for example 'v1' or 'v1 release-1.0'. There are also some default tags provided, please see the other options of this action."
required: false
default: ""
gha_iam_role:
description: 'The AWS IAM Role to assume to push images to ECR'
required: true
aws_region:
description: 'The AWS Region to use for AWS Session Authentication'
required: false
default: us-east-1
ecr_registry:
description: 'The AWS ECR Registry ARN'
required: true
outputs: {}
runs:
using: "composite"
steps:
- name: Install TOML parser
shell: bash
run: |
set -xe
wget https://github.com/freshautomations/stoml/releases/download/v0.7.1/stoml_linux_amd64
mv stoml_linux_amd64 stoml
chmod +x stoml
sudo mv stoml /usr/local/bin/
- name: Create whitespace-separated tags list
shell: bash
id: tags
run: |
set -e

# input tags
TAGS='${{ inputs.tags }}'
SHORT_SHA=$(git rev-parse --short HEAD)

cd ${{ inputs.docker_directory }}

if [ "${{ inputs.tag_cargo_version_if_present }}" == "true" ] && test -f "Cargo.toml"; then
echo "Cargo file detected, adding to tags"
VERSION=$(stoml Cargo.toml package.version)-${SHORT_SHA}
TAGS="$TAGS $VERSION"
fi

# Calendar version
if [ "${{ inputs.publish_calver }}" == "true" ]; then
# A date without leading zeros, for example:
# 2023.1.26
CAL_VER=$(date '+%Y.%-m.%-d')
TAGS="$TAGS ${CAL_VER}${{ inputs.calver_suffix }}"
fi

# latest
if [ "${{ inputs.publish_latest }}" == "true" ]; then
TAGS="$TAGS latest"
fi

# Short Git hash
TAGS="$TAGS ${SHORT_SHA}"

echo "TAGS=$TAGS" >> $GITHUB_OUTPUT
- name: Run pre-build hooks
shell: bash
run: |
cd ${{ inputs.docker_directory }}
if [[ -f pre-build-hook.sh ]]; then
echo "detected pre-build hook, running"
/bin/bash pre-build-hook.sh
else
echo "no pre build hook detected"
fi
- name: Build image and tag
shell: bash
run: |
set -xe
# Build the image
docker build -t ${{ inputs.image_name }} ${{ inputs.docker_directory }}
# Tag with each tag in the comma-separate list
IFS=' ' read -ra TAG_ARRAY <<< "${{ steps.tags.outputs.TAGS }}"
for tag in "${TAG_ARRAY[@]}"; do
docker tag ${{ inputs.image_name }} ${{ inputs.image_name }}:$tag
done
- name: Login to CoreDB Quay
if: inputs.image_name != 'tembo-pg-cnpg'
uses: docker/login-action@v2
with:
registry: ${{ inputs.registry }}
username: ${{ inputs.quay_user }}
password: ${{ inputs.quay_password }}
- name: Push to Quay
if: inputs.image_name != 'tembo-pg-cnpg'
shell: bash
run: |
set -xe
IFS=' ' read -ra TAG_ARRAY <<< "${{ steps.tags.outputs.TAGS }}"
for tag in "${TAG_ARRAY[@]}"; do
docker tag ${{ inputs.image_name }}:$tag ${{ inputs.registry}}/${{ inputs.image_name }}:$tag
docker push ${{ inputs.registry}}/${{ inputs.image_name }}:$tag
done
- name: Login to Tembo Quay
if: inputs.image_name == 'tembo-pg-cnpg'
uses: docker/login-action@v2
with:
registry: ${{ inputs.registry_tembo }}
username: ${{ inputs.quay_user_tembo}}
password: ${{ inputs.quay_password_tembo }}
- name: Push to Quay
if: inputs.image_name == 'tembo-pg-cnpg'
shell: bash
run: |
set -xe
IFS=' ' read -ra TAG_ARRAY <<< "${{ steps.tags.outputs.TAGS }}"
for tag in "${TAG_ARRAY[@]}"; do
docker tag ${{ inputs.image_name }}:$tag ${{ inputs.registry_tembo}}/${{ inputs.image_name }}:$tag
docker push ${{ inputs.registry_tembo}}/${{ inputs.image_name }}:$tag
done
- name: Configure AWS credentials for ECR
if: inputs.image_name == 'standard-cnpg' || inputs.image_name == 'ml-cnpg' || inputs.image_name == 'dw-cnpg'
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.gha_iam_role }}
role-session-name: images-gha-docker-build-and-push
aws-region: ${{ inputs.aws_region }}
- name: Install awscli
if: inputs.image_name == 'standard-cnpg' || inputs.image_name == 'ml-cnpg' || inputs.image_name == 'dw-cnpg'
uses: unfor19/install-aws-cli-action@v1
- name: Push to ECR
if: inputs.image_name == 'standard-cnpg' || inputs.image_name == 'ml-cnpg' || inputs.image_name == 'dw-cnpg'
shell: bash
run: |
set -xe
IFS=' ' read -ra TAG_ARRAY <<< "${{ steps.tags.outputs.TAGS }}"
for tag in "${TAG_ARRAY[@]}"; do
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${{ inputs.ecr_registry }}/tembo-io/${{ inputs.image_name }}
docker tag ${{ inputs.image_name }}:$tag ${{ inputs.ecr_registry }}/tembo-io/${{ inputs.image_name }}:$tag
docker push ${{ inputs.ecr_registry }}/tembo-io/${{ inputs.image_name }}:$tag
done
15 changes: 7 additions & 8 deletions .github/workflows/build_images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:

build_and_push:
name: Build and push images
permissions:
id-token: write
contents: read
runs-on:
- self-hosted
- dind
Expand All @@ -63,12 +66,6 @@ jobs:
id: versions
run: |
echo "SHORT_SHA=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
- name: Check out the tembo repo to reuse some actions
uses: actions/checkout@v3
with:
repository: tembo-io/tembo
path: ./tembo
ref: 737713f5839bcd3f533644fe316540d890c611a8
- name: Determine which tags to publish
id: tags
run: |
Expand All @@ -84,7 +81,7 @@ jobs:
echo "tag_cargo=false" >> $GITHUB_OUTPUT
fi
- name: Build and upload image
uses: ./tembo/.github/actions/build-and-push-to-quay
uses: ./.github/actions/build-and-push-to-quay
with:
image_name: ${{ matrix.name }}
docker_directory: ${{ matrix.path }}
Expand All @@ -95,7 +92,9 @@ jobs:
quay_user: ${{ secrets.QUAY_USER_TEMBO }}
quay_password: ${{ secrets.QUAY_PASSWORD_TEMBO }}
quay_user_tembo: ${{ secrets.QUAY_USER_TEMBO }}
quay_password_tembo: ${{ secrets.QUAY_PASSWORD_TEMBO }}
quay_password_tembo: ${{ secrets.QUAY_PASSWORD_TEMBO }}
gha_iam_role: ${{ secrets.GHA_IAM_ROLE }}
ecr_registry: ${{ secrets.ECR_REGISTRY }}

build_and_push_pg_slim:
name: Build and push tembo-pg-slim
Expand Down
Loading