Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bed-4714): [WIP] Enhance GitHub Workflows #1000

Closed
wants to merge 54 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
83cf04f
refactor: remove workflow to be updated
neumachen Dec 6, 2024
e16bfaf
feat: add act secrets template
neumachen Dec 6, 2024
80f281a
feat: add act configuration
neumachen Dec 6, 2024
bc0afcc
feat: define static code analysis workflow
neumachen Dec 6, 2024
df55a85
feat: define container image builds workflow
neumachen Dec 6, 2024
763beb0
feat: define testing workflow
neumachen Dec 6, 2024
b6f0e76
feat: define ci workflow
neumachen Dec 6, 2024
d9ba386
feat: add composite action for OCI linter
neumachen Dec 6, 2024
88cd453
feat: add composite action for container registry authentication
neumachen Dec 6, 2024
706b423
feat: add composite action for generating container metadata
neumachen Dec 6, 2024
865d5c4
feat: add composite action for building container images
neumachen Dec 6, 2024
c5415f8
feat: add composite action for finding files
neumachen Dec 6, 2024
ae8a635
feat: add reusable workflow for building container images
neumachen Dec 6, 2024
3324ed7
feat: add reusable workflow for linting OCI files
neumachen Dec 6, 2024
1bd0618
refactor: use local reusable workflow
neumachen Dec 6, 2024
ea49632
fix: remove unreferenced inputs
neumachen Dec 6, 2024
77b3653
fix: yaml string interpolation
neumachen Dec 6, 2024
f3b83ce
chore: Set health-retries to 5 in testing workflow configuration
neumachen Dec 9, 2024
3d8ecc4
chore: remove deprecated static code analysis workflow file
neumachen Dec 9, 2024
52a3b6f
ci: Add reusable vulnerability scan workflow
neumachen Dec 9, 2024
96ce60e
docs: Improve workflow input descriptions for vulnerability scanning
neumachen Dec 9, 2024
020ad9e
chore: Update GitHub workflows and remove redundant vulnerability sca…
neumachen Dec 9, 2024
ea42a98
ci: Fix typos and add Trivy vulnerability scan workflow
neumachen Dec 9, 2024
2e313e4
refactor: Improve step names in static code analysis workflow
neumachen Dec 9, 2024
80cbceb
ci: Update Trivy vulnerability scan workflow configuration
neumachen Dec 9, 2024
5b1d18e
refactor: Improve workflow, job, and step names for clarity and consi…
neumachen Dec 9, 2024
38b6dba
refactor: Update step names for container registry authentication and…
neumachen Dec 9, 2024
f34fbb3
chore: Update GitHub Actions workflow and action description formatting
neumachen Dec 9, 2024
61eb602
refactor: Convert CLA workflow to reusable workflow with configurable…
neumachen Dec 9, 2024
aa3a2f7
chore: Simplify CLA workflow name by removing quotes
neumachen Dec 9, 2024
bd1283f
refactor: Rename `personal_access_token` to `gh_access_token` in work…
neumachen Dec 9, 2024
fa22a7c
ci: Add CLA check workflow and make it a prerequisite for other jobs
neumachen Dec 9, 2024
3bddf35
ci: Remove deprecated cla.yml workflow file
neumachen Dec 9, 2024
dd4809a
ci: Improve workflow, job, and step names for clarity
neumachen Dec 9, 2024
22c317c
docs: Improve input descriptions for reusable workflows to explain th…
neumachen Dec 9, 2024
553110a
docs: Improve workflow input descriptions for build-container-images.yml
neumachen Dec 9, 2024
35f04b2
docs: Enhance workflow input descriptions with strategic context and …
neumachen Dec 9, 2024
296190f
ci: Update GitHub Actions workflow with improved build configuration …
neumachen Dec 9, 2024
184b101
refactor: Improve workflow, job, and step names for container image b…
neumachen Dec 9, 2024
d354e18
refactor: Update workflow step names for clarity in build-container-i…
neumachen Dec 9, 2024
f1bbf6b
ci: Update CLA workflow secrets configuration
neumachen Dec 9, 2024
599720e
ci: Fix syntax errors in vulnerability scan workflow
neumachen Dec 9, 2024
ba975f8
chore: Upgrade Trivy Action to version 0.29.0
neumachen Dec 9, 2024
536cb3c
docs: Add Apache 2.0 license headers to GitHub workflow files
neumachen Dec 9, 2024
32c5909
chore: Remove find-files action and update bloodhound Dockerfile
neumachen Dec 9, 2024
f9be751
chore: Fix hadolint issues
neumachen Dec 9, 2024
76eecea
chore: Update build target and add golangci-lint to Dockerfile
neumachen Dec 9, 2024
e0fb4e2
feat: Add custom cache settings for BloodHound container image build
neumachen Dec 9, 2024
fff3094
chore: Optimize Dockerfiles and CI workflow for improved build effici…
neumachen Dec 9, 2024
aeda571
refactor: Optimize Dockerfile with multi-line RUN commands and wget p…
neumachen Dec 9, 2024
af02f3a
fix: Correct sha256sum and 7z extraction paths in Dockerfile
neumachen Dec 9, 2024
74c9291
ci: Update BloodHound workflow to use correct Go work command syntax
neumachen Dec 9, 2024
a7d21b6
ci: Update CLA workflow with improved job naming and configuration
neumachen Dec 9, 2024
9b3c4f4
refactor: Remove custom lint-container-image action and use hadolint …
neumachen Dec 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .act.secrets.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# vim:ft=env
DOCKERHUB_USERNAME=
DOCKERHUB_TOKEN=
GHCR_ACCOUNT=
GHCR_TOKEN=
GITHUB_TOKEN=

9 changes: 9 additions & 0 deletions .actrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
--rm
--reuse
--container-options=--privileged
--pull=false
--platform=ubuntu-latest=catthehacker/ubuntu:custom-20.04
--container-architecture=linux/amd64
--workflows=.github/workflows
--secret-file=./.act.secrets

2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,4 @@ INTEGRATION_POSTGRES_PORT=127.0.0.1:65432
# Integration Neo4j
INTEGRATION_NEO4J_AUTH=neo4j/bloodhoundcommunityedition
INTEGRATION_NEO4J_PORT=127.0.0.1:37687
INTEGRATION_NEO4J_WEB_PORT=127.0.0.1:37474
INTEGRATION_NEO4J_WEB_PORT=127.0.0.1:37474
189 changes: 189 additions & 0 deletions .github/actions/build-container-image/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# Copyright 2024 Specter Ops, Inc.
#
# Licensed under the Apache License, Version 2.0
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0

---
name: Build and Publish Container Image

description: |-
This composite action builds a container image based on the specified Dockerfile and metadata.
It can optionally push the built image to a container registry.
Ensure you have the necessary Docker and QEMU setup steps in your workflow for cross-platform builds.

inputs:
image_labels:
required: true
default: "false"
description: >
A JSON object containing image labels and metadata.
These labels help describe the image and can include information like version, author, and licenses.

There are given predefined values that can be overwritten:
org.opencontainers.image.authors=https://github.com/orgs/SpecterOps/teams/bloodhound-engineering
org.opencontainers.image.licenses=Commercial Standard License
org.opencontainers.image.vendor=SpecterOps
image_tags:
required: true
default: "false"
description: >
A comma-separated list of tags to assign to the built image.
These tags help identify different versions or variants of the image.
image_metadata_json:
required: true
default: "false"
description: |
The metadata required to build the container image.
This should be the JSON output from a previous 'generate-metadata' action.

See: https://github.com/docker/metadata-action#json-output-object
image_provenance:
required: true
default: "false"
description: >
Whether to include image provenance information in the image metadata.
Provenance information provides details about how the image was built and can be useful for auditing.
image_sbom:
required: true
default: "false"
description: >
Whether to include a Software Bill of Materials (SBOM) in the image metadata.
An SBOM lists all the software components used in the image, enhancing transparency and security.
dockerfile:
default: Dockerfile
description: >
The name of the Dockerfile used for building the container image.
If not specified, it defaults to 'Dockerfile' in the repository root.

Example: 'Dockerfile.prod' for a production-specific Dockerfile.
build_context:
default: "{{defaultContext}}"
description: |-
Build's context is the set of files located in the specified PATH or URL (default Git context)
build_target:
description: >
The build stage target for multi-stage Docker builds, if applicable.
Specify this if your Dockerfile has multiple stages, and you want to build a specific one.

Example: 'production' for a multi-stage Dockerfile with a 'production' stage.
build_args:
description: >
Additional build arguments to pass to the Docker build process.
These arguments can be used to customize the build based on your requirements.

Predefined values may already be present, and any inputs provided here will be appended.

Example: 'MY_VARIABLE=value' to set an environment variable during the build.
build_contexts:
description: >
Additional build contexts to pass to Docker build process.

Define additional build context with specified contents. In Dockerfile the
context can be accessed when FROM name or --from=name is used. When
Dockerfile defines a stage with the same name it is overwritten.

Example: 'name=path'
build_outputs:
description: >
Set build additional build outputs.
cache_from:
description: >
The source image repository from which to cache layers during the build.
This can help improve build speed by reusing layers from a previously built image.

Default: type=gha

Example: 'docker.io/my-app:cache' to cache from a specific image.
default: |-
type=gha
cache_to:
description: >
The destination image cache settings to optimize the caching strategy during the build.
This input specifies where to store cached layers and how they are scoped.

Default: type=gha,mode=max

Example: "type=gha,mode=max,scope=\$\{\{ github.workflow \}\}"
default: |-
type=gha,mode=max
push_image:
required: true
default: "false"
description: >
Whether to push the built container image to the registry after building.
Set this to 'true' if you want to automatically push the image.

Example: 'true' to push the image to the registry, 'false' to skip pushing.

runs:
using: composite
steps:
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: linux/amd64,linux/arm64

- name: Set up buildx
id: buildx
uses: docker/setup-buildx-action@v3
with:
platforms: linux/amd64,linux/arm64

- name: Build
uses: docker/build-push-action@v6
with:
context: ${{ inputs.build_context }}
file: ${{ inputs.dockerfile }}
target: ${{ inputs.build_target }}
build-args: |
BUILDTIME=${{ fromJSON(inputs.image_metadata_json).labels['org.opencontainers.image.created'] }}
VERSION=${{ fromJSON(inputs.image_metadata_json).labels['org.opencontainers.image.version'] }}
REVISION=${{ fromJSON(inputs.image_metadata_json).labels['org.opencontainers.image.revision'] }}
${{ inputs.build_args }}
labels: |
org.opencontainers.image.authors=https://github.com/orgs/SpecterOps/teams/bloodhound-engineering
org.opencontainers.image.licenses=Commercial Standard License
org.opencontainers.image.vendor=SpecterOps
${{ inputs.image_labels }}
build-contexts: |
${{ inputs.build_contexts }}
provenance: ${{ inputs.image_provenance }}
sbom: ${{ inputs.image_sbom }}
cache-from: |
${{ inputs.cache_from }}
cache-to: |
${{ inputs.cache_to }}
tags: |
${{ inputs.image_tags }}
outputs: |
type=docker
${{ inputs.build_outputs }}

- name: Push Images to Registry
if: ${{ contains(inputs.push_image, 'true') }}
shell: bash
run: |
echo "::group::Push Images"
# Read each line as an image tag and push it
while IFS= read -r tag; do
# Remove leading/trailing whitespace if any
tag="$(echo -e "${tag}" | tr -d '[:space:]')"

notice_title="Pushing ${tag}"

echo "::notice title=$notice_title"
docker push "${tag}"
echo "::notice Finished pushing image"
done <<< "${{ inputs.image_tags }}"
echo "::endgroup::"
123 changes: 123 additions & 0 deletions .github/actions/container-image-metadata/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Copyright 2024 Specter Ops, Inc.
#
# Licensed under the Apache License, Version 2.0
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0

---
name: Generate Container Image Metadata

description: |
This composite action generates metadata for a container image and extracts useful information,
such as JSON data, tags, labels, revision, version, and the container image reference.
The metadata is essential for tracking and managing container images effectively.

inputs:
container_registry:
default: docker.io
required: true
description: >
The name of the container image registry where the image will be hosted.

Example: 'docker.io' for Docker Hub or a custom registry URL.

image_repository:
required: true
description: >
The name of the repository where the container image will be stored within the registry.

Example: 'my-app' or 'username/my-app' if it's a user-specific repository.

image_flavor:
description: >
See: https://github.com/docker/metadata-action#flavor-input

image_tags:
description: >
See: https://github.com/docker/metadata-action#tags-input

outputs:
json:
value: ${{ steps.generate-metadata.outputs.json }}
description: >
The JSON metadata for the container image, including details about the image and its layers.

tags:
value: ${{ steps.generate-metadata.outputs.tags }}
description: >
A list of tags associated with the container image, which may include version and branch information.

labels:
value: ${{ steps.generate-metadata.outputs.labels }}
description: >
Custom labels associated with the container image, providing additional information and metadata.

revision:
value: ${{ fromJSON(steps.generate-metadata.outputs.json).labels['org.opencontainers.image.revision'] }}
description: >
The revision of the container image, if available, typically extracted from metadata labels.

version:
value: ${{ steps.generate-metadata.outputs.version }}
description: >
The version of the container image, often derived from tags or other versioning patterns.

image_reference:
value: ${{ steps.set-image-reference.outputs.image_reference }}
description: >
The full image reference, including registry, repository, and tag.
This fully qualified name is used to pull or locate a specific image
from a particular registry (e.g., docker.io/myrepo/myimage:tag).

image_name:
value: ${{ steps.set-image-reference.outputs.image_name }}
description: >
The image name with repository and tag, typically used locally or
with the default registry. This shorthand version omits the registry
and assumes the default registry if unspecified (e.g., myimage:tag).

runs:
using: composite
steps:
- name: Downcase container registry name.
id: lower-container-registry
uses: ASzc/change-string-case-action@v6
with:
string: ${{ inputs.container_registry }}

- name: Downcase repository name
id: lower-repo
uses: ASzc/change-string-case-action@v6
with:
string: ${{ inputs.image_repository }}

- name: Generate metadata
id: generate-metadata
uses: docker/metadata-action@v5
with:
images:
${{ steps.lower-container-registry.outputs.lowercase }}/${{ steps.lower-repo.outputs.lowercase }}
flavor: ${{ inputs.image_flavor }}
tags: |-
type=raw,value=sha-{{sha}}-{{date 'YYYYMMDD-HHmmss'}},priority=1500
type=sha,format=long,priority=1450
${{ inputs.image_tags }}

- name: Set Container Image Reference
id: set-image-reference
shell: bash
run: |-
image_reference=${{ fromJSON(steps.generate-metadata.outputs.json).tags[0] }}
echo "image_reference=$image_reference" >> $GITHUB_OUTPUT
image_name=$(echo "$image_reference" | sed 's/.*\///')
echo "image_name=$image_name" >> $GITHUB_OUTPUT
52 changes: 52 additions & 0 deletions .github/actions/container-registry-authentication/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Copyright 2024 Specter Ops, Inc.
#
# Licensed under the Apache License, Version 2.0
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0

---
name: Container Registry Authentication

description: |
This composite action authenticates your GitHub Actions workflow with a container registry. It
allows your workflow to securely push container images to the registry using the provided registry
account and token.

inputs:
registry_account:
required: true
description: |
The container registry account or username.

Example: 'myusername' or 'myorg'.
registry_token:
required: true
description: |
The personal access token (PAT) or secret token for
authenticating with the GitHub Container Registry.
Ensure this token has the necessary permissions
to push container images to the specified registry.
container_registry:
required: false
description: |
The container registry that will be authenticated against.

runs:
using: composite
steps:
- name: Authenticate To Registry
uses: docker/login-action@v3
with:
registry: ${{ inputs.container_registry }}
username: ${{ inputs.registry_account }}
password: ${{ inputs.registry_token }}
Loading
Loading