Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixes spark 2.2.1 bug #369

Merged
merged 2 commits into from
Feb 21, 2018
Merged

fixes spark 2.2.1 bug #369

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aee47dd
fixes spark 2.2.1 bug
akmorrow13 Feb 21, 2018
59a6419
update spark version in jenkins test
akmorrow13 Feb 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions mango-cli/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,6 @@
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version.prefix}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
Expand All @@ -131,7 +130,6 @@
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version.prefix}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.bdgenomics.adam</groupId>
Expand Down
2 changes: 0 additions & 2 deletions mango-core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,10 @@
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version.prefix}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version.prefix}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.bdgenomics.adam</groupId>
Expand Down
8 changes: 7 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
<scala.version>2.11.12</scala.version>
<scala.version.prefix>2.11</scala.version.prefix>
<scalatra.version>2.4.1</scalatra.version>
<spark.version>2.1.0</spark.version>
<spark.version>2.2.1</spark.version>
<spark.version.prefix>-spark2_</spark.version.prefix>
<snappy.version>1.0.5</snappy.version>
<scoverage.plugin.version>1.1.1</scoverage.plugin.version>
Expand Down Expand Up @@ -438,6 +438,12 @@
Spark 2.2.0 release notes.
-->
<version>1.8.1</version>
<exclusions>
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
Expand Down
11 changes: 7 additions & 4 deletions scripts/jenkins-test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ fi

set -e

# are we testing for spark 2.1.0? if so, we need to rewrite our poms first
if [ ${SPARK_VERSION} == 2.1.0 ];
# are we testing for spark 2.2.1? if so, we need to rewrite our poms first
if [ ${SPARK_VERSION} == 2.2.1 ];
then

# shouldn't be able to move to spark 2 twice
Expand Down Expand Up @@ -116,7 +116,7 @@ find . -name "*.bak" -exec rm -f {} \;

# run integration tests
# prebuilt spark distributions are scala 2.11 for spark 2.x
if [[ ${SPARK_VERSION} == 2.1.0 && ${SCALAVER} == 2.11 ]];
if [[ ${SPARK_VERSION} == 2.2.1 && ${SCALAVER} == 2.11 ]];
then

# make a temp directory
Expand Down Expand Up @@ -158,7 +158,10 @@ then
SPARK=spark-${SPARK_VERSION}

# download prepackaged spark assembly
wget -q http://d3kbcqa49mib13.cloudfront.net/${SPARK}-bin-${HADOOP}.tgz
curl \
-L "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/${SPARK}/${SPARK}-bin-${HADOOP}.tgz" \
-o ${SPARK}-bin-${HADOOP}.tgz

tar xzvf ${SPARK}-bin-${HADOOP}.tgz
export SPARK_HOME=${MANGO_TMP_DIR}/${SPARK}-bin-${HADOOP}

Expand Down