Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

pomadchin · 2020-09-16T22:09:02Z

Overview

TODO:

Check other libraries: mb not only spire requires an upgrade
- Upgrade Accumulo up to 2.x #3381
- Update Cassandra up to 4.x #3382
Test on AWS EMR 6.2 with and without GDAL
- Test GeoTrellis with Spark 3 and Hadoop 3 on EMR 6.2+ #3380

Checklist

./CHANGELOG.md updated, if necessary. Link to the issue if closed, otherwise the PR.

Closes #2893
Closes #3218
Closes #3358

cassandra/src/main/scala/geotrellis/store/cassandra/CassandraInstance.scala

hbase-spark/src/main/scala/org/apache/spark/deploy/GTSparkHadoopUtil.scala

project/Dependencies.scala

spark/src/test/scala/geotrellis/spark/summary/StatsTileRDDMethodsSpec.scala

hbase-spark/src/test/resources/log4j.properties

echeipesh · 2021-04-12T16:00:42Z

publish/publish-to-sonatype.sh

@@ -1,3 +1,3 @@
 #!/bin/bash

-./publish/publish-to-sonatype-212.sh


2.13 not published

@echeipesh hm do you think we want to publish 3.6.0 on MavenCentral with the Spark 3.2.0-SNAPSHOT dep?

I think we should publish 3.6.0-SNAPSHOT with spark 3.2.0-SNAPSHOT dep, which would happen. Our release process is not setup for a split release though ... maybe the problem will solve itself with release of spark 3.2.0 before we actually have to cut 3.6.0 release. I don't think there is anything forcing our release to happen soon?

Yea, that is why 2.13 is not published, these scripts are to publish release artifacts.
Check out locationtech scripts - they contain 2.13 artifacts publish.

raster/src/main/scala/geotrellis/raster/ArrayMultibandTile.scala

raster/src/main/scala/geotrellis/raster/render/RGBA.scala

pomadchin · 2021-04-12T20:45:41Z

proj4/src/test/scala/geotrellis/proj4/io/wkt/WKTTest.scala

+  it("should return the EPSG code(3857) of the passed WKT string with proj4 extension") {
+    val comparisonCode = "EPSG:3857"


This test properly works now!

metasim · 2021-04-13T16:17:14Z

Nicely done guys!! Much appreciated! ❤️

pomadchin added the needs cq label Sep 16, 2020

pomadchin self-assigned this Sep 16, 2020

pomadchin commented Sep 17, 2020

View reviewed changes

cassandra/src/main/scala/geotrellis/store/cassandra/CassandraInstance.scala Show resolved Hide resolved

pomadchin commented Sep 17, 2020

View reviewed changes

hbase-spark/src/main/scala/org/apache/spark/deploy/GTSparkHadoopUtil.scala Outdated Show resolved Hide resolved

pomadchin commented Sep 17, 2020

View reviewed changes

project/Dependencies.scala Outdated Show resolved Hide resolved

pomadchin commented Sep 17, 2020

View reviewed changes

spark/src/test/scala/geotrellis/spark/summary/StatsTileRDDMethodsSpec.scala Show resolved Hide resolved

pomadchin mentioned this pull request Mar 8, 2021

WIP: Cross-compile with 2.13 #3346

Closed

5 tasks

pomadchin added 3 commits April 8, 2021 09:59

Upd Spark and Spire, drop Scala 2.11

6077b89

Proxy SparkHadoopUtil

8d35578

Workaround Cassandra JXM issue and commentout a failing test

04748ff

pomadchin force-pushed the upd/spark-3 branch from 4cfad09 to 04748ff Compare April 8, 2021 13:59

Add Scala 2.13 crosscompilation

c1d54b8

pomadchin mentioned this pull request Apr 8, 2021

Consider releasing a subset of packages that don't depend on Spark against 2.13 #3358

Closed

pomadchin added the in progress label Apr 8, 2021

pomadchin changed the title ~~[WIP] Spark 3 support~~ Spark 3 support Apr 8, 2021

pomadchin force-pushed the upd/spark-3 branch from 31dc664 to 0c91881 Compare April 8, 2021 19:57

Adjust Raster and Spark tests

b36631e

pomadchin force-pushed the upd/spark-3 branch from 0c91881 to b36631e Compare April 8, 2021 20:03

Fix Raster tests

7518fbc

pomadchin force-pushed the upd/spark-3 branch from 0513c20 to 7518fbc Compare April 8, 2021 21:23

pomadchin changed the title ~~Spark 3 support~~ Spark 3 support and Scala 2.13 crosscompilation Apr 8, 2021

pomadchin changed the title ~~Spark 3 support and Scala 2.13 crosscompilation~~ Spark 3 support and Scala 2.13 cross compilation Apr 8, 2021

pomadchin mentioned this pull request Apr 8, 2021

WIP: Cross-compile 213 #3363

Closed

5 tasks

Fix GDAL tests

2d033b0

pomadchin force-pushed the upd/spark-3 branch 3 times, most recently from 295f384 to d4c1062 Compare April 9, 2021 13:54

pomadchin commented Apr 9, 2021

View reviewed changes

hbase-spark/src/test/resources/log4j.properties Show resolved Hide resolved

Fix HBase tests

9c9b5e1

pomadchin force-pushed the upd/spark-3 branch from d4c1062 to 9c9b5e1 Compare April 9, 2021 14:19

pomadchin changed the title ~~Spark 3 support and Scala 2.13 cross compilation~~ Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation Apr 9, 2021

Upd up to Hadoop 3.2.1 and HBase 2.4.2

744d88e

pomadchin force-pushed the upd/spark-3 branch from 8813533 to 744d88e Compare April 9, 2021 21:29

pomadchin mentioned this pull request Apr 9, 2021

Investigate RDD.sample behavior change in Spark 3 #3379

Open

pomadchin added 2 commits April 9, 2021 18:16

Downgrade HBase server

c7404a9

Move Accumulo into a separate executor

2ab8c7b

pomadchin removed the in progress label Apr 9, 2021

pomadchin requested a review from echeipesh April 9, 2021 23:15

pomadchin added 4 commits April 10, 2021 20:37

Supply more empty argument lists explicitly

bb028a8

Upd SBT up to 1.5.0

35ab1d7

Bump spire up to 0.17.0

25ac4f2

Rm SparkHadoopUtilShim object

f8e6111

echeipesh reviewed Apr 12, 2021

View reviewed changes

raster/src/main/scala/geotrellis/raster/ArrayMultibandTile.scala Show resolved Hide resolved

Move new Hadoop3 deps into the Provided scope

a9f5598

pomadchin force-pushed the upd/spark-3 branch from 8f95f6b to a9f5598 Compare April 12, 2021 16:33

echeipesh reviewed Apr 12, 2021

View reviewed changes

raster/src/main/scala/geotrellis/raster/render/RGBA.scala Outdated Show resolved Hide resolved

pomadchin added 2 commits April 12, 2021 14:02

fromRGBA(double) => fromRGBAPct

6fa727c

Bump proj4j version up

f37c7c1

pomadchin force-pushed the upd/spark-3 branch from 7659d24 to f37c7c1 Compare April 12, 2021 20:44

pomadchin commented Apr 12, 2021

View reviewed changes

echeipesh approved these changes Apr 13, 2021

View reviewed changes

pomadchin merged commit b071b33 into locationtech:master Apr 13, 2021

pomadchin deleted the upd/spark-3 branch April 13, 2021 15:09

pomadchin mentioned this pull request Apr 13, 2021

Fix deprecated API usage to support JDK11 compilation #3383

Merged

pomadchin removed the needs cq label Apr 20, 2021

pomadchin mentioned this pull request May 18, 2021

mutable vs. immutable Seq handling in 2.13 upgrade #3359

Closed

jessewebb mentioned this pull request Jul 8, 2021

Scala 2.13 cross compilation #2893

Closed

10 tasks

pomadchin mentioned this pull request Jul 10, 2022

WebMercator is reported as deprecated EPSG:3785 instead of EPSG:3857 in metadata #3247

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

pomadchin commented Sep 16, 2020 •

edited

Loading

echeipesh Apr 12, 2021

pomadchin Apr 12, 2021

echeipesh Apr 12, 2021

pomadchin Apr 12, 2021 •

edited

Loading

pomadchin Apr 12, 2021

metasim commented Apr 13, 2021

		@@ -1,3 +1,3 @@
		#!/bin/bash

		./publish/publish-to-sonatype-212.sh

		it("should return the EPSG code(3857) of the passed WKT string with proj4 extension") {
		val comparisonCode = "EPSG:3857"

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

Conversation

pomadchin commented Sep 16, 2020 • edited Loading

Overview

Checklist

echeipesh Apr 12, 2021

Choose a reason for hiding this comment

pomadchin Apr 12, 2021

Choose a reason for hiding this comment

echeipesh Apr 12, 2021

Choose a reason for hiding this comment

pomadchin Apr 12, 2021 • edited Loading

Choose a reason for hiding this comment

pomadchin Apr 12, 2021

Choose a reason for hiding this comment

metasim commented Apr 13, 2021

pomadchin commented Sep 16, 2020 •

edited

Loading

pomadchin Apr 12, 2021 •

edited

Loading