Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s390x CI support #85

Open
wants to merge 3 commits into
base: trunk
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions build-scripts/cassandra-artifacts_s390x.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
#!/bin/bash -xe
Copy link
Member

@michaelsembwever michaelsembwever Nov 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's unclear to me how this file is different to the existing build-scripts/cassandra-artifacts.sh ?

it is just the deb and rpm packaging that has been removed?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes..deb and rpm packaging is removed for s390x for now as apache cassandra needs few code changes to fix test failures on s390x and we have already provided code changes through https://issues.apache.org/jira/browse/CASSANDRA-17723 . Also, chronicle-bytes code used in Apache Cassandra needs s390x support hence we are going ahead with this workaround for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Is the plan to fix those failures as part of 17723 or 18072 ?

If that is the case then this PR should be marked as a draft, as the intention is never to merge it.

If you want to delay fixing the deb and rpm packing, then I suggest adding a conditional around that in the original script. Something like DEP_PACKAGING_SKIP and RPM_PACKAGING_SKIP.

Copy link
Author

@Nayana-ibm Nayana-ibm Nov 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Is the plan to fix those failures as part of 17723 or 18072 ?

Issue-17723 will fix 6 tests out of 13 tests which we are skipping for s390x.
Issue-18072 is a part of CI for s390x. I raised different issue as repository was different.

If that is the case then this PR should be marked as a draft, as the intention is never to merge it.

If you want to delay fixing the deb and rpm packing, then I suggest adding a conditional around that in the original script. Something like DEP_PACKAGING_SKIP and RPM_PACKAGING_SKIP.

I think we can go ahead with .deb and .rpm packaging for s390x also. Lets not delay.


################################
#
# Prep
#
################################

# variables, with defaults
[ "x${cassandra_builds_dir}" != "x" ] || cassandra_builds_dir="cassandra-builds"

# pre-conditions
command -v ant >/dev/null 2>&1 || { echo >&2 "ant needs to be installed"; exit 1; }
command -v pip >/dev/null 2>&1 || { echo >&2 "pip needs to be installed"; exit 1; }
command -v virtualenv >/dev/null 2>&1 || { echo >&2 "virtualenv needs to be installed"; exit 1; }
command -v docker >/dev/null 2>&1 || { echo >&2 "docker needs to be installed"; exit 1; }
(docker info >/dev/null 2>&1) || { echo >&2 "docker needs to running"; exit 1; }
[ -f "build.xml" ] || { echo >&2 "build.xml must exist"; exit 1; }
[ -d "${cassandra_builds_dir}" ] || { echo >&2 "cassandra-builds directory must exist"; exit 1; }

# print debug information on versions
ant -version
pip --version
virtualenv --version
docker --version

# Sphinx is needed for the gen-doc target
virtualenv venv
source venv/bin/activate
# setuptools 45.0.0 requires python 3.5+
python -m pip install "setuptools<45" Sphinx sphinx_rtd_theme

################################
#
# Main
#
################################

# Setup JDK
java_version=$(java -version 2>&1 | awk -F '"' '/version/ {print $2}' | awk -F. '{print $1}')
if [ "$java_version" -ge 11 ]; then
java_version="11"
export CASSANDRA_USE_JDK11=true
if ! grep -q CASSANDRA_USE_JDK11 build.xml ; then
echo "Skipping build. JDK11 not supported against $(grep 'property\s*name=\"base.version\"' build.xml |sed -ne 's/.*value=\"\([^"]*\)\".*/\1/p')"
exit 0
fi
fi

# Loop to prevent failure due to maven-ant-tasks not downloading a jar..
set +e # disable immediate exit from this point

ARTIFACTS_BUILD_RUN=0
ECLIPSE_WARNINGS_RUN=0
HAS_DEPENDENCY_CHECK_TARGET=0
# versions starting from 6.4.1 contain "rate limiter" functionality to make builds more stable
# https://github.com/jeremylong/DependencyCheck/pull/3725
DEPENDENCY_CHECK_VERSION=6.4.1

for x in $(seq 1 3); do
if [ "${ARTIFACTS_BUILD_RUN}" -eq "0" ]; then
ant clean artifacts
RETURN="$?"
fi
if [ "${RETURN}" -eq "0" ]; then
ARTIFACTS_BUILD_RUN=1
if [ "${ECLIPSE_WARNINGS_RUN}" -eq "0" ]; then
# Run eclipse-warnings if build was successful
ant eclipse-warnings
RETURN="$?"
fi
if [ "${RETURN}" -eq "0" ]; then
ECLIPSE_WARNINGS_RUN=1
if [ "${HAS_DEPENDENCY_CHECK_TARGET}" -eq "1" ]; then
ant -Ddependency-check.version=${DEPENDENCY_CHECK_VERSION} -Ddependency-check.home=/tmp/dependency-check-${DEPENDENCY_CHECK_VERSION} dependency-check
RETURN="$?"
else
RETURN="0"
fi
if [ ! "${RETURN}" -eq "0" ]; then
if [ -f /tmp/dependency-check-${DEPENDENCY_CHECK_VERSION}/dependency-check-ant/dependency-check-ant.jar ]; then
# Break the build here only in case dep zip was downloaded (hence JAR was extracted) just fine
# but the check itself has failed. If JAR does not exist, it is probably
# because the network was down so the ant target did not download the zip in the first place.
echo "Failing the build on OWASP dependency check. Run 'ant dependency-check' locally and consult build/dependency-check-report.html to see the details."
break
else
# sleep here to give the net the chance to resume after probable partition
sleep 10
continue
fi
fi
set -e
fi
break
fi
# sleep here to give the net the chance to resume after probable partition
sleep 10
done

################################
#
# Clean
#
################################

# /virtualenv
deactivate

exit "${RETURN}"
29 changes: 28 additions & 1 deletion build-scripts/cassandra-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ command -v git >/dev/null 2>&1 || { echo >&2 "git needs to be installed"; exit 1
ant -version
git --version

# check arch
arch=`uname -m`

# lists all tests for the specific test type
_list_tests() {
local -r classlistprefix="$1"
Expand Down Expand Up @@ -102,7 +105,30 @@ _main() {

export TMP_DIR="$(pwd)/tmp"
mkdir -p ${TMP_DIR}


if [ "$arch" == "s390x" ]
then
case $target in
"stress-test")
# hard fail on test compilation, but dont fail the test run as unstable test reports are processed
ant clean jar stress-build-test
ant $target -Dtmp.dir="$(pwd)/tmp" || echo "failed $target"
;;
"test")
_run_testlist "unit" "testclasslist-s390x" "${split_chunk}" "$(_timeout_for 'test.timeout')"
;;
"test-cdc")
_run_testlist "unit" "testclasslist-cdc-s390x" "${split_chunk}" "$(_timeout_for 'test.timeout')"
;;
"test-compression")
_run_testlist "unit" "testclasslist-compression-s390x" "${split_chunk}" "$(_timeout_for 'test.timeout')"
;;
*)
echo "unregconised \"$target\""
exit 1
;;
esac
else
case $target in
"stress-test")
# hard fail on test compilation, but dont fail the test run as unstable test reports are processed
Expand Down Expand Up @@ -160,6 +186,7 @@ _main() {
exit 1
;;
esac
fi
}

_main "$@"