Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Spark version for regression tests #168

Merged
merged 3 commits into from
Aug 22, 2024
Merged

Update Spark version for regression tests #168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
19511bb
Update Spark version
Aug 21, 2024
85fef99
setup.sh won't work independently
Aug 21, 2024
2bc8333
Set the env variables in run_spark_sql.sh
Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions regtests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,13 +74,6 @@ AZURE_BLOB_TEST_BASE=abfss://<container-name>@<storage-account-name>.blob.core.w
into the `credentials` folder. Then specify the name of the file in your .env file - do not change the
path, as `/tmp/credentials` is the folder on the container where the credentials file will be mounted.

## Setup without running tests

Setup is idempotent.

```
./setup.sh
```
Copy link
Contributor Author

@flyrain flyrain Aug 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't work independently locally, but I guess it's fine to remove it.


## Experiment with failed test

Expand Down
5 changes: 4 additions & 1 deletion regtests/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,11 @@
#
# Run without args to run all tests, or single arg for single test.

export SPARK_VERSION=spark-3.5.2
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3-scala2.13

if [ -z "${SPARK_HOME}"]; then
export SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"

Expand Down
5 changes: 4 additions & 1 deletion regtests/run_spark_sql.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,13 @@
REGTEST_HOME=$(dirname $(realpath $0))
cd ${REGTEST_HOME}

export SPARK_VERSION=spark-3.5.2
export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3-scala2.13

./setup.sh

if [ -z "${SPARK_HOME}"]; then
export SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi

SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
Expand Down
18 changes: 11 additions & 7 deletions regtests/setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ set -x
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

if [ -z "${SPARK_HOME}" ]; then
SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
fi
SPARK_CONF="${SPARK_HOME}/conf/spark-defaults.conf"
export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"
Expand All @@ -33,22 +33,26 @@ export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-
echo 'Verifying Spark binaries...'
if ! [ -f ${SPARK_HOME}/bin/spark-sql ]; then
echo 'Setting up Spark...'
if ! [ -f ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz ]; then
if [ -z "${SPARK_VERSION}" ] || [ -z "${SPARK_DISTRIBUTION}" ]; then
echo 'SPARK_VERSION or SPARK_DISTRIBUTION not set. Please set SPARK_VERSION and SPARK_DISTRIBUTION to the desired version.'
exit 1
fi
if ! [ -f ~/${SPARK_DISTRIBUTION}.tgz ]; then
echo 'Downloading spark distro...'
wget -O ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz
if ! [ -f ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz ]; then
wget -O ~/${SPARK_DISTRIBUTION}.tgz https://dlcdn.apache.org/spark/${SPARK_VERSION}/${SPARK_DISTRIBUTION}.tgz
if ! [ -f ~/${SPARK_DISTRIBUTION}.tgz ]; then
if [[ "${OSTYPE}" == "darwin"* ]]; then
echo "Detected OS: mac. Running 'brew install wget' to try again."
brew install wget
wget -O ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz
wget -O ~/${SPARK_DISTRIBUTION}.tgz https://dlcdn.apache.org/spark/${SPARK_VERSION}/${SPARK_DISTRIBUTION}.tgz
fi
fi
else
echo 'Found existing Spark tarball'
fi
tar xzvf ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz -C ~
tar xzvf ~/${SPARK_DISTRIBUTION}.tgz -C ~
echo 'Done!'
SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
SPARK_CONF="${SPARK_HOME}/conf/spark-defaults.conf"
else
echo 'Verified Spark distro already installed.'
Expand Down
Loading