Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support external tempest plugins #103

Merged
merged 1 commit into from
Dec 20, 2023
Merged

Support external tempest plugins #103

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
205 changes: 180 additions & 25 deletions container-images/tcib/base/os/tempest/run_tempest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,96 @@
#!/bin/sh
# run_tempest.sh
# ==============
#
# This script is executed inside the tempest containers defined in the tcib
# repository (tempest, tempest-all and tempest-extra). The main purpose of this
# script is executing the tempest command with correct arguments and preparing
# the tempest.conf file. The execution of the script can be influenced by
# setting values for environment variables which match the TEMPEST_* or
# TEMPESTCONF_* regex.
#
#
# TEMPESTCONF_* environment variables
# -----------------------------------
#
# These variables define with which arguments should be the
# discover-tempest-config command executed.
#
# Supported boolean arguments:
# --create, --insecure, --collect-timing, --no-default-deployer, --debug
# --verbose, --no-rng, --non-admin, --retry-image, --convert-to-raw
#
# Supported string arguments:
# --timeout, --out, --deployer-input, --test-accounts, --create-accounts-file
# --profile, --generate-profile, --image-disk-format, --image,
# --flavor-min-mem, --flavor-min-disk, --disk, --network-id
#
# If you want discover-tempest-config to be executed with any of the boolean arguments
# mentioned above, then set corresponding environment variable to 'true'. The
# name of the variable consists of the prefix 'TEMPESTCONF_' plus the name
# of the argument. For example, if you want to run discover-tempest-config with
# --non-admin argument then set TEMPESTCONF_NON_ADMIN=true. Similarly, in the case
# of string arguments, when you want to run discover-tempest-config with
# --timeout 600 then set TEMPESTCONF_TIMEOUT=600.
#
# To override values in tempest.conf generated by discover-tempest-config please
# use TEMPESTCONF_OVERRIDES.
#
# TEMPEST_* environment ariables
# ------------------------------
#
# These variables define behaviour of the part of the script which is
# responsible for the tempest tests execution.
#
# Supported boolean arguments:
# --smoke, --parallel, --serial
#
# Supported string arguments:
# --include-list, --exclude-list, --concurrency, --worker-file,
#
# If you want the tempest command to be executed with any of the boolean
# arguments mentioned above, then set the corresponding environment variable to
# 'true'. The name of the variable consists of the prefix 'TEMPEST_' plus the
# name of the argument. For example, if you want to run tempest with --smoke
# argument then set TEMPEST_SMOKE=true. Similarly, in the case of string
# arguments, when you want to run tempest witn --include-list=/path/to/list.txt
# then set TEMPEST_INCLUDE_LIST=/path/to/list.txt
#
# The script, putting the above-mentioned tempest environment variables aside,
# offers also the following extra variables: TEMPEST_EXTERNAL_PLUGIN_GIT_URL,
# TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL, TEMPEST_EXTERNAL_PLUGIN_REFSPEC. These
# can be used to specify extra tempest plugins you want to be installed within
# the container plus patches you want to be applied on top of the installed
# plugins.
#
# For example, by setting the environment variables like in the example below
# you ensure that the script downloads barbican-tempest-plugin and
# neutron-tempest-plugin and installs neutron-tempest-plugin with
# "refs/changes/97/896397/2" change from gerrit:
#
#
# TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL=\
# "https://opendev.org/openstack/barbican-tempest-plugin.git,"\
# "https://opendev.org/openstack/neutron-tempest-plugin.git"
#
# TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL=\
# "-,"\
# "https://review.opendev.org/openstack/neutron-tempest-plugin"
#
# TEMPEST_EXTERNAL_PLUGIN_REFSPEC=\
# "-,"\
# "refs/changes/97/896397/2"
#
# Note, that if you specify these variables then tempest will have access only
# to plugins which were specified by these variables. Meaning, if you consider
# the example above, then tempest won't be able to execute tests from
# octavia-tempest-plugin for example.

set -x

HOMEDIR=/var/lib/tempest
TEMPEST_DIR=$HOMEDIR/openshift
TEMPEST_PATH=$HOMEDIR/
lpiwowar marked this conversation as resolved.
Show resolved Hide resolved
TEMPEST_DIR=$HOMEDIR/openshift
lpiwowar marked this conversation as resolved.
Show resolved Hide resolved
CONCURRENCY="${CONCURRENCY:-}"
TEMPESTCONF_ARGS=""
TEMPEST_ARGS=""
Expand Down Expand Up @@ -34,7 +121,22 @@ TEMPEST_ARGS=""
[[ ! -z ${TEMPESTCONF_FLAVOR_MIN_DISK} ]] && TEMPESTCONF_ARGS+="--flavor-min-disk ${TEMPESTCONF_FLAVOR_MIN_DISK} "
[[ ! -z ${TEMPESTCONF_NETWORK_ID} ]] && TEMPESTCONF_ARGS+="--network-id ${TEMPESTCONF_NETWORK_ID} "

TEMPESTCONF_OVERRIDES="$(echo ${TEMPESTCONF_OVERRIDES} | tr '\n' ' ') identity.v3_endpoint_type public"
kopecmartin marked this conversation as resolved.
Show resolved Hide resolved

# Tempest arguments
TEMPEST_EXTERNAL_PLUGIN_GIT_URL="${TEMPEST_EXTERNAL_PLUGIN_GIT_URL:-}"
TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL="${TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL:-}"
TEMPEST_EXTERNAL_PLUGIN_REFSPEC="${TEMPEST_EXTERNAL_PLUGIN_REFSPEC:-}"
TEMPEST_EXTERNAL_PLUGIN_DIR=/var/lib/tempest/external-plugins

# Convert comma separated lists to arrays
OLD_IFS=$IFS
kopecmartin marked this conversation as resolved.
Show resolved Hide resolved
IFS=","
read -ra TEMPEST_EXTERNAL_PLUGIN_GIT_URL <<< $TEMPEST_EXTERNAL_PLUGIN_GIT_URL
read -ra TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL <<< $TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL
read -ra TEMPEST_EXTERNAL_PLUGIN_REFSPEC <<< $TEMPEST_EXTERNAL_PLUGIN_REFSPEC
IFS=$OLD_IFS

[[ ${TEMPEST_SMOKE} == true ]] && TEMPEST_ARGS+="--smoke "
[[ ${TEMPEST_PARALLEL:=true} == true ]] && TEMPEST_ARGS+="--parallel "
[[ ${TEMPEST_SERIAL} == true ]] && TEMPEST_ARGS+="--serial "
Expand All @@ -58,13 +160,82 @@ if [[ ! -z ${TEMPESTCONF_REMOVE} ]]; then
done <<< "$TEMPESTCONF_REMOVE"
fi

TEMPESTCONF_OVERRIDES="$(echo ${TEMPESTCONF_OVERRIDES} | tr '\n' ' ') identity.v3_endpoint_type public"

if [ -n "$CONCURRENCY" ] && [ -z ${TEMPEST_CONCURRENCY} ]; then
TEMPEST_ARGS+="--concurrency ${CONCURRENCY} "
fi

pushd $HOMEDIR
function run_git_tempest {
mkdir -p $TEMPEST_EXTERNAL_PLUGIN_DIR
pushd $TEMPEST_EXTERNAL_PLUGIN_DIR

python3 -m venv .venv
source ./.venv/bin/activate

for plugin_index in "${!TEMPEST_EXTERNAL_PLUGIN_GIT_URL[@]}"; do
git_url=${TEMPEST_EXTERNAL_PLUGIN_GIT_URL[plugin_index]}
change_url=${TEMPEST_EXTERNAL_PLUGIN_CHANGE_URL[plugin_index]}
refspec=${TEMPEST_EXTERNAL_PLUGIN_REFSPEC[plugin_index]}
plugin_name=$(basename -s .git $git_url)

git clone $git_url

if [[ ! -z $change_url ]] && [[ ! -z $refspec ]] || \
[[ $change_url != "-" ]] && [[ $refspec != "-" ]]; then
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a comment would help but i'm trying to imagine what $change_url looks like here and why we need to guard for != "-" (and failing to imagine it :))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained the usage of the variables used here in the comment at the beginning of the script. I can imagine that the "-" is a little bit confusing. I think that it would be used only in edge cases when someone would want to download multiple changes.

I think that the conversation here openstack-k8s-operators/test-operator#12 can be helpful as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got it. Not sure if there is a better way to implement it, but I see that you need "-" because your loop gets values based on indexes.
A good documentation was added to the script itself. Is there another place that we can add this? So everybody would know how to use this?

pushd $plugin_name
git fetch $change_url $refspec
git checkout FETCH_HEAD
popd
fi

pip install ./$plugin_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip/pip3? does it make a difference? I guess pip points to pip3 so it doesn't matter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that using pip should be ok here. We are inside a virtual environment.

done

pushd $HOMEDIR
tempest init openshift
pushd $TEMPEST_DIR

discover-tempest-config ${TEMPESTCONF_ARGS} ${TEMPESTCONF_OVERRIDES}
tempest run ${TEMPEST_ARGS}
kopecmartin marked this conversation as resolved.
Show resolved Hide resolved
RETURN_VALUE=$?

deactivate

popd
popd
popd
}

function run_rpm_tempest {
pushd $HOMEDIR
tempest init openshift
pushd $TEMPEST_DIR

discover-tempest-config ${TEMPESTCONF_ARGS} ${TEMPESTCONF_OVERRIDES}
tempest run ${TEMPEST_ARGS}
RETURN_VALUE=$?

popd
popd
}

function generate_test_results {
pushd $TEMPEST_DIR

echo "Generate subunit"
stestr last --subunit > ${TEMPEST_PATH}testrepository.subunit || true

echo "Generate subunit xml file"
subunit2junitxml ${TEMPEST_PATH}testrepository.subunit > ${TEMPEST_PATH}tempest_results.xml || true

echo "Generate html result"
subunit2html ${TEMPEST_PATH}testrepository.subunit ${TEMPEST_PATH}stestr_results.html || true

echo Copying logs file
cp -rf ${TEMPEST_DIR}/* ${TEMPEST_PATH}

popd
}

export OS_CLOUD=default

Expand All @@ -85,28 +256,12 @@ if [ ! -f ${TEMPEST_PATH}exclude.txt ] && [ -z ${TEMPEST_EXCLUDE_LIST} ]; then
touch ${TEMPEST_PATH}exclude.txt
fi

tempest init openshift

pushd $TEMPEST_DIR
discover-tempest-config ${TEMPESTCONF_ARGS} ${TEMPESTCONF_OVERRIDES}

tempest run ${TEMPEST_ARGS}

RETURN_VALUE=$?

echo "Generate subunit"
stestr last --subunit > ${TEMPEST_PATH}testrepository.subunit || true

echo "Generate subunit xml file"
subunit2junitxml ${TEMPEST_PATH}testrepository.subunit > ${TEMPEST_PATH}tempest_results.xml || true

echo "Generate html result"
subunit2html ${TEMPEST_PATH}testrepository.subunit ${TEMPEST_PATH}stestr_results.html || true
if [ -z $TEMPEST_EXTERNAL_PLUGIN_GIT_URL ]; then
run_rpm_tempest
else
run_git_tempest
fi

echo Copying logs file
cp -rf ${TEMPEST_DIR}/* ${TEMPEST_PATH}
generate_test_results

exit ${RETURN_VALUE}

popd
popd