Skip to content

Adding additional test suites. [skip-ci] #66

Adding additional test suites. [skip-ci]

Adding additional test suites. [skip-ci] #66

# --------------------------------------------------------------------
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed
# with this work for additional information regarding copyright
# ownership. The ASF licenses this file to You under the Apache
# License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the
# License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License.
#
# --------------------------------------------------------------------
# GitHub Actions Workflow: Apache Cloudberry Build Pipeline
# --------------------------------------------------------------------
# Description:
#
# This workflow builds, tests, and packages Apache Cloudberry on
# Rocky Linux 9. It ensures artifact integrity, performs installation
# tests, validates key operations, and provides detailed test reports,
# including handling for ignored test cases.
#
# Workflow Overview:
# 1. **Check Skip**:
# - Dynamically determines if the workflow should run based on CI skip flags.
# - Evaluates the following fields for skip flags:
# - **Pull Request Events**: PR title and PR body.
# - **Push Events**: Commit message of the head commit.
# - Supports the following skip patterns (case-insensitive):
# - `[skip ci]`
# - `[ci skip]`
# - `[no ci]`
# - **Example Usage**:
# - Add `[skip ci]` to a commit message, PR title, or body to skip the workflow.
#
# 2. **Build Job**:
# - Configures and builds Apache Cloudberry.
# - Runs unit tests and verifies build artifacts.
# - Creates RPM packages, source tarballs, and logs.
# - **Key Artifacts**: RPM package, source tarball, build logs.
#
# 3. **RPM Install Test Job**:
# - Verifies RPM integrity and installs Cloudberry.
# - Validates successful installation.
# - **Key Artifacts**: Installation logs, verification results.
#
# 4. **Test Job (Matrix)**:
# - Executes a test matrix to validate different scenarios.
# - Creates a demo cluster and runs installcheck tests.
# - Parses and reports test results, including failed and ignored tests.
# - **Key Features**:
# - Regression diffs are displayed if found, aiding quick debugging.
# - Both failed and ignored test names are logged and reported.
# - **Key Artifacts**: Test logs, regression files, test summaries.
#
# **Matrix Tests**:
# - `ic-good-opt-off`: Basic regression tests with optimizer disabled.
# - `ic-expandshrink`: Cluster expansion/shrinking tests.
# - `ic-singlenode_isolation`: Isolation tests on a single-node setup.
# - `ic-singlenode_isolation2`: Additional isolation tests on single-node.
# - `ic-resgroup-v2`: Resource group version 2 tests.
#
# 5. **Report Job**:
# - Aggregates job results into a final report.
# - Sends failure notifications if any step fails.
#
# Execution Environment:
# - **Runs On**: Ubuntu-latest with Rocky Linux 9 containers.
# - **Resource Requirements**:
# - Disk: Minimum 20GB free space.
# - Memory: Minimum 8GB RAM.
# - CPU: Recommended 4+ cores.
#
# Triggers:
# - Push to `build-devel` branch.
# - Pull requests to `build-devel` branch.
# - Manual workflow dispatch.
#
# Container Images:
# - **Build**: `apache/incubator-cloudberry:cbdb-build-rocky9-latest`
# - **Test**: `apache/incubator-cloudberry:cbdb-test-rocky9-latest`
#
# Artifacts:
# - RPM Package (retention: ${{ env.LOG_RETENTION_DAYS }} days).
# - Source Tarball (retention: ${{ env.LOG_RETENTION_DAYS }} days).
# - Logs and Test Results (retention: ${{ env.LOG_RETENTION_DAYS }} days).
# - Regression Diffs (retention: ${{ env.LOG_RETENTION_DAYS }} days).
#
# Notes:
# - Supports concurrent job execution.
# - Includes robust skip logic for pull requests and pushes.
# - Handles ignored test cases, ensuring results are comprehensive.
# - Provides detailed logs and error handling for failed and ignored tests.
# --------------------------------------------------------------------
name: Apache Cloudberry Build
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
# Note: Step details, logs, and artifacts require users to be logged into GitHub
# even for public repositories. This is a GitHub security feature and cannot
# be overridden by permissions.
permissions:
# READ permissions allow viewing repository contents
contents: read # Required for checking out code and reading repository files
# READ permissions for packages (Container registry, etc)
packages: read # Allows reading from GitHub package registry
# WRITE permissions for actions includes read access to:
# - Workflow runs
# - Artifacts (requires GitHub login)
# - Logs (requires GitHub login)
actions: write
# READ permissions for checks API:
# - Step details visibility (requires GitHub login)
# - Check run status and details
checks: read
# READ permissions for pull request metadata:
# - PR status
# - Associated checks
# - Review states
pull-requests: read
env:
LOG_RETENTION_DAYS: 7
jobs:
## ======================================================================
## Job: check-skip
## ======================================================================
check-skip:
runs-on: ubuntu-latest
outputs:
should_skip: ${{ steps.skip-check.outputs.should_skip }}
steps:
- id: skip-check
shell: bash
env:
EVENT_NAME: ${{ github.event_name }}
PR_TITLE: ${{ github.event.pull_request.title || '' }}
PR_BODY: ${{ github.event.pull_request.body || '' }}
run: |
# Default to not skipping
echo "should_skip=false" >> "$GITHUB_OUTPUT"
# Apply skip logic only for pull_request events
if [[ "$EVENT_NAME" == "pull_request" ]]; then
# Combine PR title and body for skip check
MESSAGE="${PR_TITLE}\n${PR_BODY}"
# Escape special characters using printf %s
ESCAPED_MESSAGE=$(printf "%s" "$MESSAGE")
echo "Checking PR title and body (escaped): $ESCAPED_MESSAGE"
# Check for skip patterns
if echo -e "$ESCAPED_MESSAGE" | grep -qEi '\[skip[ -]ci\]|\[ci[ -]skip\]|\[no[ -]ci\]'; then
echo "should_skip=true" >> "$GITHUB_OUTPUT"
fi
else
echo "Skip logic is not applied for $EVENT_NAME events."
fi
- name: Report Skip Status
if: steps.skip-check.outputs.should_skip == 'true'
run: |
echo "CI Skip flag detected in PR - skipping all checks."
exit 0
## ======================================================================
## Job: build
## ======================================================================
build:
name: Build Apache Cloudberry
env:
JOB_TYPE: build
needs: check-skip
runs-on: ubuntu-latest
timeout-minutes: 120
outputs:
build_timestamp: ${{ steps.set_timestamp.outputs.timestamp }}
container:
image: apache/incubator-cloudberry:cbdb-build-rocky9-latest
options: >-
--user root
-h cdw
steps:
- name: Skip Check
if: needs.check-skip.outputs.should_skip == 'true'
run: |
echo "Build skipped via CI skip flag" >> "$GITHUB_STEP_SUMMARY"
exit 0
- name: Set build timestamp
if: needs.check-skip.outputs.should_skip != 'true'
id: set_timestamp # Add an ID to reference this step
run: |
timestamp=$(date +'%Y%m%d_%H%M%S')
echo "timestamp=$timestamp" | tee -a "$GITHUB_OUTPUT" # Use GITHUB_OUTPUT for job outputs
echo "BUILD_TIMESTAMP=$timestamp" | tee -a "$GITHUB_ENV" # Also set as environment variable
- name: Checkout Apache Cloudberry (apache/cloudberry)
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/checkout@v4
with:
repository: apache/cloudberry
fetch-depth: 1
- name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/checkout@v4
with:
repository: apache/cloudberry-devops-release
ref: main
path: cloudberry-devops-release
- name: Move cloudberry-devops-release directory
if: needs.check-skip.outputs.should_skip != 'true'
run: |
set -eo pipefail
if ! mv "${GITHUB_WORKSPACE}"/cloudberry-devops-release "${GITHUB_WORKSPACE}"/..; then
echo "::error::Container initialization failed"
exit 1
fi
- name: Cloudberry Environment Initialization
if: needs.check-skip.outputs.should_skip != 'true'
env:
LOGS_DIR: build-logs
run: |
set -eo pipefail
if ! su - gpadmin -c "/tmp/init_system.sh"; then
echo "::error::Container initialization failed"
exit 1
fi
mkdir -p "${LOGS_DIR}/details"
chown -R gpadmin:gpadmin .
chmod -R 755 .
chmod 777 "${LOGS_DIR}"
df -kh /
rm -rf /__t/*
df -kh /
df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
{
echo "=== Environment Information ==="
uname -a
df -h
free -h
env
} | tee -a "${LOGS_DIR}/details/environment.log"
echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
- name: Generate Build Job Summary Start
if: needs.check-skip.outputs.should_skip != 'true'
run: |
{
echo "# Build Job Summary"
echo "## Environment"
echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
echo "- OS Version: $(cat /etc/redhat-release)"
echo "- GCC Version: $(gcc --version | head -n1)"
} >> "$GITHUB_STEP_SUMMARY"
- name: Run Apache Cloudberry configure script
if: needs.check-skip.outputs.should_skip != 'true'
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh
if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR} ${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/configure-cloudberry.sh"; then
echo "::error::Configure script failed"
exit 1
fi
- name: Run Apache Cloudberry build script
if: needs.check-skip.outputs.should_skip != 'true'
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh
if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR} ${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/build-cloudberry.sh"; then
echo "::error::Build script failed"
exit 1
fi
- name: Verify build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
run: |
set -eo pipefail
echo "Verifying build artifacts..."
{
echo "=== Build Artifacts Verification ==="
echo "Timestamp: $(date -u)"
if [ ! -d "/usr/local/cloudberry-db" ]; then
echo "::error::Build artifacts directory not found"
exit 1
fi
# Verify critical binaries
critical_binaries=(
"/usr/local/cloudberry-db/bin/postgres"
"/usr/local/cloudberry-db/bin/psql"
)
echo "Checking critical binaries..."
for binary in "${critical_binaries[@]}"; do
if [ ! -f "$binary" ]; then
echo "::error::Critical binary missing: $binary"
exit 1
fi
if [ ! -x "$binary" ]; then
echo "::error::Binary not executable: $binary"
exit 1
fi
echo "Binary verified: $binary"
ls -l "$binary"
done
# Test binary execution
echo "Testing binary execution..."
if ! /usr/local/cloudberry-db/bin/postgres --version; then
echo "::error::postgres binary verification failed"
exit 1
fi
if ! /usr/local/cloudberry-db/bin/psql --version; then
echo "::error::psql binary verification failed"
exit 1
fi
echo "All build artifacts verified successfully"
} 2>&1 | tee -a build-logs/details/build-verification.log
- name: Create Source tarball, create RPM and verify artifacts
if: needs.check-skip.outputs.should_skip != 'true'
env:
CBDB_VERSION: 99.0.0
BUILD_NUMBER: 1
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
{
echo "=== Artifact Creation Log ==="
echo "Timestamp: $(date -u)"
# Create source tarball
echo "Creating source tarball..."
tar czf "${SRC_DIR}"/../apache-cloudberry-incubating-src.tgz -C "${SRC_DIR}"/.. ./cloudberry
mv "${SRC_DIR}"/../apache-cloudberry-incubating-src.tgz "${SRC_DIR}"
# Verify tarball contents
echo "Verifying source tarball contents..."
if ! tar tzf "${SRC_DIR}"/apache-cloudberry-incubating-src.tgz > /dev/null; then
echo "::error::Source tarball verification failed"
exit 1
fi
# Create RPM
echo "Creating RPM package..."
rpmdev-setuptree
ln -s "${SRC_DIR}"/../cloudberry-devops-release/packaging/rpm/el/SPECS/apache-cloudberry-db-incubating.spec "${HOME}"/rpmbuild/SPECS/apache-cloudberry-db-incubating.spec
cp "${SRC_DIR}"/LICENSE /usr/local/cloudberry-db
"${SRC_DIR}"/../cloudberry-devops-release/scripts/build-rpm.sh --version "${CBDB_VERSION}" --release "${BUILD_NUMBER}"
# Get OS version and move RPM
os_version=$(grep -oP '(?<=^VERSION_ID=")[0-9]' /etc/os-release)
RPM_FILE="${HOME}"/rpmbuild/RPMS/x86_64/apache-cloudberry-db-incubating-"${CBDB_VERSION}"-"${BUILD_NUMBER}".el"${os_version}".x86_64.rpm
cp "${RPM_FILE}" "${SRC_DIR}"
# Get package information
echo "Package Information:"
rpm -qip "${RPM_FILE}"
# Verify critical files in RPM
echo "Verifying critical files in RPM..."
for binary in "bin/postgres" "bin/psql"; do
if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
echo "::error::Critical binary '${binary}' not found in RPM"
exit 1
fi
done
# Record checksums
echo "Calculating checksums..."
sha256sum "${RPM_FILE}" | tee -a build-logs/details/checksums.log
sha256sum "${SRC_DIR}"/apache-cloudberry-incubating-src.tgz | tee -a build-logs/details/checksums.log
echo "Artifacts created and verified successfully"
} 2>&1 | tee -a build-logs/details/artifact-creation.log
- name: Run Apache Cloudberry unittest script
if: needs.check-skip.outputs.should_skip != 'true'
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh
if ! time su - gpadmin -c "cd ${SRC_DIR} && SRC_DIR=${SRC_DIR} ${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/unittest-cloudberry.sh"; then
echo "::error::Unittest script failed"
exit 1
fi
- name: Generate Build Job Summary End
if: always()
run: |
{
echo "## Build Results"
echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload build logs
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/upload-artifact@v4
with:
name: build-logs-${{ env.BUILD_TIMESTAMP }}
path: |
build-logs/
retention-days: ${{ env.LOG_RETENTION_DAYS }}
- name: Upload Cloudberry RPM build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/upload-artifact@v4
with:
name: apache-cloudberry-db-incubating-rpm-build-artifacts
retention-days: ${{ env.LOG_RETENTION_DAYS }}
if-no-files-found: error
path: |
*.rpm
- name: Upload Cloudberry source build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/upload-artifact@v4
with:
name: apache-cloudberry-db-incubating-source-build-artifacts
retention-days: ${{ env.LOG_RETENTION_DAYS }}
if-no-files-found: error
path: |
apache-cloudberry-incubating-src.tgz
## ======================================================================
## Job: rpm-install-test
## ======================================================================
rpm-install-test:
name: RPM Install Test Apache Cloudberry
needs: [check-skip, build]
runs-on: ubuntu-latest
timeout-minutes: 120
container:
image: apache/incubator-cloudberry:cbdb-test-rocky9-latest
options: >-
--user root
-h cdw
steps:
- name: Skip Check
if: needs.check-skip.outputs.should_skip == 'true'
run: |
echo "RPM install test skipped via CI skip flag" >> "$GITHUB_STEP_SUMMARY"
exit 0
- name: Download Cloudberry RPM build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/download-artifact@v4
with:
name: apache-cloudberry-db-incubating-rpm-build-artifacts
path: ${{ github.workspace }}/rpm_build_artifacts
merge-multiple: false
- name: Cloudberry Environment Initialization
if: needs.check-skip.outputs.should_skip != 'true'
env:
LOGS_DIR: install-logs
run: |
set -eo pipefail
if ! su - gpadmin -c "/tmp/init_system.sh"; then
echo "::error::Container initialization failed"
exit 1
fi
mkdir -p "${LOGS_DIR}/details"
chown -R gpadmin:gpadmin .
chmod -R 755 .
chmod 777 "${LOGS_DIR}"
df -kh /
rm -rf /__t/*
df -kh /
df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
{
echo "=== Environment Information ==="
uname -a
df -h
free -h
env
} | tee -a "${LOGS_DIR}/details/environment.log"
echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
- name: Verify RPM artifacts
if: needs.check-skip.outputs.should_skip != 'true'
id: verify-artifacts
run: |
set -eo pipefail
RPM_FILE=$(ls "${GITHUB_WORKSPACE}"/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
if [ ! -f "${RPM_FILE}" ]; then
echo "::error::RPM file not found"
exit 1
fi
echo "rpm_file=${RPM_FILE}" >> "$GITHUB_OUTPUT"
echo "Verifying RPM artifacts..."
{
echo "=== RPM Verification Summary ==="
echo "Timestamp: $(date -u)"
echo "RPM File: ${RPM_FILE}"
# Get RPM metadata and verify contents
echo "Package Information:"
rpm -qip "${RPM_FILE}"
# Get key RPM attributes for verification
RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" "${RPM_FILE}")
RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" "${RPM_FILE}")
echo "version=${RPM_VERSION}" >> "$GITHUB_OUTPUT"
echo "release=${RPM_RELEASE}" >> "$GITHUB_OUTPUT"
# Verify expected binaries are in the RPM
echo "Verifying critical files in RPM..."
for binary in "bin/postgres" "bin/psql"; do
if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
echo "::error::Critical binary '${binary}' not found in RPM"
exit 1
fi
done
echo "RPM Details:"
echo "- Version: ${RPM_VERSION}"
echo "- Release: ${RPM_RELEASE}"
# Calculate and store checksum
echo "Checksum:"
sha256sum "${RPM_FILE}"
} 2>&1 | tee -a install-logs/details/rpm-verification.log
- name: Install Cloudberry RPM
if: success() && needs.check-skip.outputs.should_skip != 'true'
env:
RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
run: |
set -eo pipefail
if [ -z "${RPM_FILE}" ]; then
echo "::error::RPM_FILE environment variable is not set"
exit 1
fi
{
echo "=== RPM Installation Log ==="
echo "Timestamp: $(date -u)"
echo "RPM File: ${RPM_FILE}"
echo "Version: ${RPM_VERSION}"
echo "Release: ${RPM_RELEASE}"
# Clean install location
rm -rf /usr/local/cloudberry-db
# Install RPM
echo "Starting installation..."
if ! time dnf install -y "${RPM_FILE}"; then
echo "::error::RPM installation failed"
exit 1
fi
echo "Installation completed successfully"
rpm -qi apache-cloudberry-db-incubating
echo "Installed files:"
rpm -ql apache-cloudberry-db-incubating
} 2>&1 | tee -a install-logs/details/rpm-installation.log
- name: Upload install logs
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/upload-artifact@v4
with:
name: install-logs-${{ needs.build.outputs.build_timestamp }}
path: |
install-logs/
retention-days: ${{ env.LOG_RETENTION_DAYS }}
- name: Generate Install Test Job Summary End
if: always()
shell: bash {0}
run: |
{
echo "# Installed Package Summary"
echo "\`\`\`"
rpm -qi apache-cloudberry-db-incubating
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY" || true
## ======================================================================
## Job: test
## ======================================================================
test:
name: ${{ matrix.test }}
needs: [check-skip, build]
runs-on: ubuntu-latest
timeout-minutes: 120
strategy:
fail-fast: false # Continue with other tests if one fails
matrix:
test:
- ic-good-opt-off
- ic-expandshrink
- ic-singlenode_isolation
- ic-singlenode_isolation2
- ic-resgroup-v2
include:
- test: ic-good-opt-off
make_target: installcheck-good
make_directory: --directory=src/test/regress
num_primary_mirror_pairs: 3
pg_settings:
optimizer: "off"
- test: ic-expandshrink
make_target: installcheck-expandshrink
make_directory: --directory=src/test/isolation2
num_primary_mirror_pairs: 3
- test: ic-singlenode_isolation
make_target: installcheck-singlenode
make_directory: --directory=src/test/isolation
num_primary_mirror_pairs: 0
- test: ic-singlenode_isolation2
make_target: installcheck-singlenode
make_directory: --directory=src/test/singlenode_isolation2
num_primary_mirror_pairs: 0
- test: ic-resgroup-v2
make_target: installcheck-resgroup-v2
make_directory: --directory=src/test/isolation2
num_primary_mirror_pairs: 3
container:
image: apache/incubator-cloudberry:cbdb-build-rocky9-latest
options: >-
--privileged
--user root
--hostname cdw
--shm-size=2gb
--cgroupns=host
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
steps:
- name: Skip Check
if: needs.check-skip.outputs.should_skip == 'true'
run: |
echo "Test ${{ matrix.test }} skipped via CI skip flag" >> "$GITHUB_STEP_SUMMARY"
exit 0
- name: Use timestamp from previous job
if: needs.check-skip.outputs.should_skip != 'true'
run: |
echo "Timestamp from output: ${{ needs.build.outputs.build_timestamp }}"
- name: Checkout CI Build/Test Scripts (apache/cloudberry-devops-release)
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/checkout@v4
with:
repository: apache/cloudberry-devops-release
ref: main
path: cloudberry-devops-release
- name: Move cloudberry-devops-release directory
if: needs.check-skip.outputs.should_skip != 'true'
run: |
set -eo pipefail
if ! mv "${GITHUB_WORKSPACE}"/cloudberry-devops-release "${GITHUB_WORKSPACE}"/..; then
echo "::error::Container initialization failed"
exit 1
fi
- name: Cloudberry Environment Initialization
env:
LOGS_DIR: build-logs
run: |
set -eo pipefail
if ! su - gpadmin -c "/tmp/init_system.sh"; then
echo "::error::Container initialization failed"
exit 1
fi
mkdir -p "${LOGS_DIR}/details"
chown -R gpadmin:gpadmin .
chmod -R 755 .
chmod 777 "${LOGS_DIR}"
df -kh /
rm -rf /__t/*
df -kh /
df -h | tee -a "${LOGS_DIR}/details/disk-usage.log"
free -h | tee -a "${LOGS_DIR}/details/memory-usage.log"
{
echo "=== Environment Information ==="
uname -a
df -h
free -h
env
} | tee -a "${LOGS_DIR}/details/environment.log"
echo "SRC_DIR=${GITHUB_WORKSPACE}" | tee -a "$GITHUB_ENV"
- name: Setup cgroups
if: needs.check-skip.outputs.should_skip != 'true'
shell: bash
run: |
set -uxo pipefail
echo "Current mounts:"
mount | grep cgroup
CGROUP_BASEDIR=/sys/fs/cgroup
# 1. Basic setup with permissions
sudo chmod -R 777 ${CGROUP_BASEDIR}/
sudo mkdir -p ${CGROUP_BASEDIR}/gpdb
sudo chmod -R 777 ${CGROUP_BASEDIR}/gpdb
sudo chown -R gpadmin:gpadmin ${CGROUP_BASEDIR}/gpdb
# 2. Enable controllers
sudo bash -c "echo '+cpu +cpuset +memory +io' > ${CGROUP_BASEDIR}/cgroup.subtree_control" || true
sudo bash -c "echo '+cpu +cpuset +memory +io' > ${CGROUP_BASEDIR}/gpdb/cgroup.subtree_control" || true
# 3. CPU settings
sudo bash -c "echo 'max 100000' > ${CGROUP_BASEDIR}/gpdb/cpu.max" || true
sudo bash -c "echo '100' > ${CGROUP_BASEDIR}/gpdb/cpu.weight" || true
sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/cpu.weight.nice" || true
sudo bash -c "echo 0-$(( $(nproc) - 1 )) > ${CGROUP_BASEDIR}/gpdb/cpuset.cpus" || true
sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/cpuset.mems" || true
# 4. Memory settings
sudo bash -c "echo 'max' > ${CGROUP_BASEDIR}/gpdb/memory.max" || true
sudo bash -c "echo '0' > ${CGROUP_BASEDIR}/gpdb/memory.min" || true
sudo bash -c "echo 'max' > ${CGROUP_BASEDIR}/gpdb/memory.high" || true
# 5. IO settings
echo "Available block devices:"
lsblk
sudo bash -c "
if [ -f \${CGROUP_BASEDIR}/gpdb/io.stat ]; then
echo 'Detected IO devices:'
cat \${CGROUP_BASEDIR}/gpdb/io.stat
fi
echo '' > \${CGROUP_BASEDIR}/gpdb/io.max || true
"
# 6. Fix permissions again after all writes
sudo chmod -R 777 ${CGROUP_BASEDIR}/gpdb
sudo chown -R gpadmin:gpadmin ${CGROUP_BASEDIR}/gpdb
# 7. Check required files
echo "Checking required files:"
required_files=(
"cgroup.procs"
"cpu.max"
"cpu.pressure"
"cpu.weight"
"cpu.weight.nice"
"cpu.stat"
"cpuset.cpus"
"cpuset.mems"
"cpuset.cpus.effective"
"cpuset.mems.effective"
"memory.current"
"io.max"
)
for file in "${required_files[@]}"; do
if [ -f "${CGROUP_BASEDIR}/gpdb/$file" ]; then
echo "✓ $file exists"
ls -l "${CGROUP_BASEDIR}/gpdb/$file"
else
echo "✗ $file missing"
fi
done
# 8. Test subdirectory creation
echo "Testing subdirectory creation..."
sudo -u gpadmin bash -c "
TEST_DIR=\${CGROUP_BASEDIR}/gpdb/test6448
if mkdir -p \$TEST_DIR; then
echo 'Created test directory'
sudo chmod -R 777 \$TEST_DIR
if echo \$\$ > \$TEST_DIR/cgroup.procs; then
echo 'Successfully wrote to cgroup.procs'
cat \$TEST_DIR/cgroup.procs
# Move processes back to parent before cleanup
echo \$\$ > \${CGROUP_BASEDIR}/gpdb/cgroup.procs
else
echo 'Failed to write to cgroup.procs'
ls -la \$TEST_DIR/cgroup.procs
fi
ls -la \$TEST_DIR/
rmdir \$TEST_DIR || {
echo 'Moving all processes to parent before cleanup'
cat \$TEST_DIR/cgroup.procs | while read pid; do
echo \$pid > \${CGROUP_BASEDIR}/gpdb/cgroup.procs 2>/dev/null || true
done
rmdir \$TEST_DIR
}
else
echo 'Failed to create test directory'
fi
"
# 9. Verify setup as gpadmin user
echo "Testing cgroup access as gpadmin..."
sudo -u gpadmin bash -c "
echo 'Checking mounts...'
mount | grep cgroup
echo 'Checking /proc/self/mounts...'
cat /proc/self/mounts | grep cgroup
if ! grep -q cgroup2 /proc/self/mounts; then
echo 'ERROR: cgroup2 mount NOT visible to gpadmin'
exit 1
fi
echo 'SUCCESS: cgroup2 mount visible to gpadmin'
if ! [ -w ${CGROUP_BASEDIR}/gpdb ]; then
echo 'ERROR: gpadmin cannot write to gpdb cgroup'
exit 1
fi
echo 'SUCCESS: gpadmin can write to gpdb cgroup'
echo 'Verifying key files content:'
echo 'cpu.max:'
cat ${CGROUP_BASEDIR}/gpdb/cpu.max || echo 'Failed to read cpu.max'
echo 'cpuset.cpus:'
cat ${CGROUP_BASEDIR}/gpdb/cpuset.cpus || echo 'Failed to read cpuset.cpus'
echo 'cgroup.subtree_control:'
cat ${CGROUP_BASEDIR}/gpdb/cgroup.subtree_control || echo 'Failed to read cgroup.subtree_control'
"
# 10. Show final state
echo "Final cgroup state:"
ls -la ${CGROUP_BASEDIR}/gpdb/
echo "Cgroup setup completed successfully"
- name: "Generate Test Job Summary Start: ${{ matrix.test }}"
if: always()
run: |
{
echo "# Test Job Summary: ${{ matrix.test }}"
echo "## Environment"
echo "- Start Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]]; then
echo "## Skip Status"
echo "✓ Test execution skipped via CI skip flag"
else
echo "- OS Version: $(cat /etc/redhat-release)"
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Download Cloudberry RPM build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/download-artifact@v4
with:
name: apache-cloudberry-db-incubating-rpm-build-artifacts
path: ${{ github.workspace }}/rpm_build_artifacts
merge-multiple: false
- name: Download Cloudberry Source build artifacts
if: needs.check-skip.outputs.should_skip != 'true'
uses: actions/download-artifact@v4
with:
name: apache-cloudberry-db-incubating-source-build-artifacts
path: ${{ github.workspace }}/source_build_artifacts
merge-multiple: false
- name: Verify downloaded artifacts
if: needs.check-skip.outputs.should_skip != 'true'
id: verify-artifacts
run: |
set -eo pipefail
SRC_TARBALL_FILE=$(ls "${GITHUB_WORKSPACE}"/source_build_artifacts/apache-cloudberry-incubating-src.tgz)
if [ ! -f "${SRC_TARBALL_FILE}" ]; then
echo "::error::SRC TARBALL file not found"
exit 1
fi
echo "src_tarball_file=${SRC_TARBALL_FILE}" >> "$GITHUB_OUTPUT"
echo "Verifying SRC TARBALL artifacts..."
{
echo "=== SRC TARBALL Verification Summary ==="
echo "Timestamp: $(date -u)"
echo "SRC TARBALL File: ${SRC_TARBALL_FILE}"
# Calculate and store checksum
echo "Checksum:"
sha256sum "${SRC_TARBALL_FILE}"
} 2>&1 | tee -a build-logs/details/src-tarball-verification.log
RPM_FILE=$(ls "${GITHUB_WORKSPACE}"/rpm_build_artifacts/apache-cloudberry-db-incubating*.rpm)
if [ ! -f "${RPM_FILE}" ]; then
echo "::error::RPM file not found"
exit 1
fi
echo "rpm_file=${RPM_FILE}" >> "$GITHUB_OUTPUT"
echo "Verifying RPM artifacts..."
{
echo "=== RPM Verification Summary ==="
echo "Timestamp: $(date -u)"
echo "RPM File: ${RPM_FILE}"
# Get RPM metadata and verify contents
echo "Package Information:"
rpm -qip "${RPM_FILE}"
# Get key RPM attributes for verification
RPM_VERSION=$(rpm -qp --queryformat "%{VERSION}" "${RPM_FILE}")
RPM_RELEASE=$(rpm -qp --queryformat "%{RELEASE}" "${RPM_FILE}")
echo "version=${RPM_VERSION}" >> "$GITHUB_OUTPUT"
echo "release=${RPM_RELEASE}" >> "$GITHUB_OUTPUT"
# Verify expected binaries are in the RPM
echo "Verifying critical files in RPM..."
for binary in "bin/postgres" "bin/psql"; do
if ! rpm -qlp "${RPM_FILE}" | grep -q "${binary}$"; then
echo "::error::Critical binary '${binary}' not found in RPM"
exit 1
fi
done
echo "RPM Details:"
echo "- Version: ${RPM_VERSION}"
echo "- Release: ${RPM_RELEASE}"
# Calculate and store checksum
echo "Checksum:"
sha256sum "${RPM_FILE}"
} 2>&1 | tee -a build-logs/details/rpm-verification.log
- name: Install Cloudberry RPM
if: success() && needs.check-skip.outputs.should_skip != 'true'
env:
RPM_FILE: ${{ steps.verify-artifacts.outputs.rpm_file }}
RPM_VERSION: ${{ steps.verify-artifacts.outputs.version }}
RPM_RELEASE: ${{ steps.verify-artifacts.outputs.release }}
run: |
set -eo pipefail
if [ -z "${RPM_FILE}" ]; then
echo "::error::RPM_FILE environment variable is not set"
exit 1
fi
{
echo "=== RPM Installation Log ==="
echo "Timestamp: $(date -u)"
echo "RPM File: ${RPM_FILE}"
echo "Version: ${RPM_VERSION}"
echo "Release: ${RPM_RELEASE}"
# Clean install location
rm -rf /usr/local/cloudberry-db
# Install RPM
echo "Starting installation..."
if ! time dnf install -y "${RPM_FILE}"; then
echo "::error::RPM installation failed"
exit 1
fi
echo "Installation completed successfully"
rpm -qi apache-cloudberry-db-incubating
} 2>&1 | tee -a build-logs/details/rpm-installation.log
- name: Extract source tarball
if: success() && needs.check-skip.outputs.should_skip != 'true'
env:
SRC_TARBALL_FILE: ${{ steps.verify-artifacts.outputs.src_tarball_file }}
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
{
echo "=== Source Extraction Log ==="
echo "Timestamp: $(date -u)"
echo "Starting extraction..."
if ! time tar zxf "${SRC_TARBALL_FILE}" -C "${SRC_DIR}"/.. ; then
echo "::error::Source extraction failed"
exit 1
fi
echo "Extraction completed successfully"
echo "Extracted contents:"
ls -la "${SRC_DIR}/../cloudberry"
echo "Directory size:"
du -sh "${SRC_DIR}/../cloudberry"
} 2>&1 | tee -a build-logs/details/source-extraction.log
- name: Create Apache Cloudberry demo cluster
if: success() && needs.check-skip.outputs.should_skip != 'true'
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -eo pipefail
{
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
if ! time su - gpadmin -c "cd ${SRC_DIR} && NUM_PRIMARY_MIRROR_PAIRS='${{ matrix.num_primary_mirror_pairs }}' SRC_DIR=${SRC_DIR} ${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh"; then
echo "::error::Demo cluster creation failed"
exit 1
fi
} 2>&1 | tee -a build-logs/details/create-cloudberry-demo-cluster.log
- name: "Run Tests: ${{ matrix.test }}"
if: success() && needs.check-skip.outputs.should_skip != 'true'
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -o pipefail
{
# Construct PGOPTIONS from pg_settings
PG_OPTS=""
if [[ -n "${{ matrix.pg_settings.optimizer }}" ]]; then
PG_OPTS="$PG_OPTS -c optimizer=${{ matrix.pg_settings.optimizer }}"
fi
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh
if ! time su - gpadmin -c "cd ${SRC_DIR} && \
MAKE_NAME='${{ matrix.test }}' \
MAKE_TARGET='${{ matrix.make_target }}' \
MAKE_DIRECTORY='${{ matrix.make_directory }}' \
PGOPTIONS='${PG_OPTS}' \
SRC_DIR='${SRC_DIR}' \
${SRC_DIR}/../cloudberry-devops-release/build_automation/cloudberry/scripts/test-cloudberry.sh"; then
echo "::error::Test execution failed"
df -kh /
exit 1
fi
} 2>&1 | tee -a build-logs/details/make-${{ matrix.test }}.log
- name: Parse Test Results
id: test-results
if: always()
shell: bash {0}
env:
SRC_DIR: ${{ github.workspace }}
run: |
set -o pipefail
{
# Check if the workflow was cancelled
if [[ "${{ job.status }}" == "cancelled" ]]; then
echo "Workflow was cancelled - recording cancelled status"
{
echo "status=cancelled"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
echo "Cancellation status recorded" | tee -a build-logs/details/parse-test-results-${{ matrix.test }}.log
exit 0
fi
# Check for skipped tests
if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]]; then
echo "Tests skipped - setting default values"
{
echo "status=skipped"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
echo "Skip status recorded" | tee -a build-logs/details/parse-test-results-${{ matrix.test }}.log
exit 0
fi
# Ensure log file exists and has content
LOG_FILE="build-logs/details/make-${{ matrix.test }}.log"
if [[ ! -f "$LOG_FILE" ]]; then
echo "::warning::Test log file not found, using default values"
{
echo "status=no_log"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
exit 0
fi
# Check if log file is empty or contains only whitespace
if [[ ! -s "$LOG_FILE" ]] || ! grep -q '[^[:space:]]' "$LOG_FILE"; then
echo "::warning::Test log file is empty, using default values"
{
echo "status=empty_log"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
exit 0
fi
# Attempt to parse results
chmod +x "${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh
MAKE_NAME='${{ matrix.test }}' \
"${SRC_DIR}"/../cloudberry-devops-release/build_automation/cloudberry/scripts/parse-test-results.sh \
"$LOG_FILE"
status_code=$?
case $status_code in
0) # All tests passed
echo "All tests passed successfully"
;;
1) # Tests failed but parsed successfully
echo "Test failures detected but properly parsed"
# Don't exit here - let the workflow continue
;;
2) # Parse error or missing file
echo "::warning::Could not parse test results properly"
{
echo "status=parse_error"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
exit 0
;;
*) # Unexpected error
echo "::warning::Unexpected error during test results parsing"
{
echo "status=unknown_error"
echo "total_tests=0"
echo "failed_tests=0"
echo "passed_tests=0"
echo "ignored_tests=0"
} >> "$GITHUB_OUTPUT"
exit 0
;;
esac
} 2>&1 | tee -a build-logs/details/parse-test-results-${{ matrix.test }}.log
- name: Check and Display Regression Diffs
if: always()
run: |
# Search for regression.diffs recursively
found_file=$(find . -type f -name "regression.diffs" | head -n 1)
if [[ -n "$found_file" ]]; then
echo "Found regression.diffs at: $found_file"
cat "$found_file"
else
echo "No regression.diffs file found in the hierarchy."
fi
- name: "Generate Test Job Summary End: ${{ matrix.test }}"
if: always()
shell: bash {0}
run: |
{
if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]]; then
echo "## Test Results - SKIPPED"
echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
exit 0
fi
echo "## Test Results"
echo "- End Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
# Check if job was cancelled
if [[ "${{ job.status }}" == "cancelled" ]]; then
echo "### Test Status"
echo "🚫 Test execution was cancelled"
echo ""
echo "### Execution Summary"
echo "Test run was interrupted and did not complete. No test results are available."
exit 0
fi
# Add test execution results
echo "### Test Execution Summary"
echo "| Metric | Count |"
echo "|--------|-------|"
echo "| Total Tests | ${{ steps.test-results.outputs.total_tests }} |"
echo "| Passed Tests | ${{ steps.test-results.outputs.passed_tests }} |"
echo "| Failed Tests | ${{ steps.test-results.outputs.failed_tests }} |"
echo "| Ignored Tests | ${{ steps.test-results.outputs.ignored_tests }} |"
# Test status with emoji based on results
echo "### Test Status"
if [[ "${{ steps.test-results.outputs.status }}" == "passed" ]]; then
echo "✅ All ${{ steps.test-results.outputs.total_tests }} tests passed successfully"
else
echo "⚠️ ${{ steps.test-results.outputs.failed_tests }} of ${{ steps.test-results.outputs.total_tests }} tests failed"
# Add failed tests as a bullet list if there are failures
if [[ -n "${{ steps.test-results.outputs.failed_test_names }}" ]]; then
echo ""
echo "### Failed Tests"
echo "The following tests failed:"
echo "${{ steps.test-results.outputs.failed_test_names }}" | tr ',' '\n' | while read -r test; do
if [[ -n "$test" ]]; then
echo "* \`${test}\`"
fi
done
fi
fi
# Add ignored tests as a separate section
if [[ -n "${{ steps.test-results.outputs.ignored_test_names }}" ]]; then
echo ""
echo "### Ignored Tests"
echo "The following tests were ignored (results not critical):"
echo "${{ steps.test-results.outputs.ignored_test_names }}" | tr ',' '\n' | while read -r test; do
if [[ -n "$test" ]]; then
echo "* \`${test}\`"
fi
done
fi
} >> "$GITHUB_STEP_SUMMARY" || true
- name: Upload test logs
if: always()
uses: actions/upload-artifact@v4
with:
name: test-logs-${{ matrix.test }}-${{ needs.build.outputs.build_timestamp }}
path: |
build-logs/
retention-days: ${{ env.LOG_RETENTION_DAYS }}
- name: Upload test regression logs
if: >-
always() && (
steps.test-results.outputs.status == 'failed' ||
job.status == 'cancelled' ||
steps.test-results.outputs.status == 'cancelled'
)
uses: actions/upload-artifact@v4
with:
name: regression-logs-${{ matrix.test }}-${{ needs.build.outputs.build_timestamp }}
path: |
src/test/**/regression.out
src/test/**/regression.diffs
src/test/**/results/
gpAux/gpdemo/datadirs/standby/log/
gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/
gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/log/
gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/log/
gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/log/
gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/log/
gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/log/
gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/log/
retention-days: ${{ env.LOG_RETENTION_DAYS }}
report:
name: Generate Apache Cloudberry Build Report
needs: [check-skip, build, rpm-install-test, test]
if: always()
runs-on: ubuntu-latest
steps:
- name: Generate Final Report
run: |
{
echo "# Apache Cloudberry Build Pipeline Report"
if [[ "${{ needs.check-skip.outputs.should_skip }}" == "true" ]]; then
echo "## CI Skip Status"
echo "✅ CI checks skipped via skip flag"
echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
else
echo "## Job Status"
echo "- Build Job: ${{ needs.build.result }}"
echo "- Test Job: ${{ needs.test.result }}"
echo "- Completion Time: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
if [[ "${{ needs.build.result }}" == "success" && "${{ needs.test.result }}" == "success" ]]; then
echo "✅ Pipeline completed successfully"
else
echo "⚠️ Pipeline completed with failures"
if [[ "${{ needs.build.result }}" != "success" ]]; then
echo "### Build Job Failure"
echo "Check build logs for details"
fi
if [[ "${{ needs.test.result }}" != "success" ]]; then
echo "### Test Job Failure"
echo "Check test logs and regression files for details"
fi
fi
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Notify on failure
if: |
needs.check-skip.outputs.should_skip != 'true' &&
(needs.build.result != 'success' || needs.test.result != 'success')
run: |
echo "::error::Build/Test pipeline failed! Check job summaries and logs for details"
echo "Timestamp: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
echo "Build Result: ${{ needs.build.result }}"
echo "Test Result: ${{ needs.test.result }}"