Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

Merged
merged 22 commits into from
Apr 13, 2021
Merged

Spark 3 & Hadoop 3 support, Scala 2.13 cross compilation #3294

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6077b89
Upd Spark and Spire, drop Scala 2.11
pomadchin Sep 16, 2020
8d35578
Proxy SparkHadoopUtil
pomadchin Sep 16, 2020
04748ff
Workaround Cassandra JXM issue and commentout a failing test
pomadchin Sep 17, 2020
c1d54b8
Add Scala 2.13 crosscompilation
pomadchin Apr 8, 2021
b36631e
Adjust Raster and Spark tests
pomadchin Apr 8, 2021
7518fbc
Fix Raster tests
pomadchin Apr 8, 2021
2d033b0
Fix GDAL tests
pomadchin Apr 9, 2021
9c9b5e1
Fix HBase tests
pomadchin Apr 9, 2021
6874f33
Generate mdoc in a sep SBT process
pomadchin Apr 9, 2021
ffe87c9
Accumulo tests run in a separate process
pomadchin Apr 9, 2021
f28e6b7
Increase CircleCI no output timeout
pomadchin Apr 9, 2021
da70ac9
Supply the empty argument list explicitly, it is deprecated since Sca…
pomadchin Apr 9, 2021
744d88e
Upd up to Hadoop 3.2.1 and HBase 2.4.2
pomadchin Apr 9, 2021
c7404a9
Downgrade HBase server
pomadchin Apr 9, 2021
2ab8c7b
Move Accumulo into a separate executor
pomadchin Apr 9, 2021
bb028a8
Supply more empty argument lists explicitly
pomadchin Apr 11, 2021
35ab1d7
Upd SBT up to 1.5.0
pomadchin Apr 11, 2021
25ac4f2
Bump spire up to 0.17.0
pomadchin Apr 11, 2021
f8e6111
Rm SparkHadoopUtilShim object
pomadchin Apr 12, 2021
a9f5598
Move new Hadoop3 deps into the Provided scope
pomadchin Apr 12, 2021
6fa727c
fromRGBA(double) => fromRGBAPct
pomadchin Apr 12, 2021
f37c7c1
Bump proj4j version up
pomadchin Apr 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions proj4/src/test/scala/geotrellis/proj4/CoordinateTransformTest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,18 +168,19 @@ class CoordinateTransformTest extends AnyFunSuite with BaseCoordinateTransformTe
}

test("EPSG_4326") {
// Adjusted with the Proj4j 1.1.2 update, see https://github.com/locationtech/proj4j/pull/71
checkTransformAndInverse(
"EPSG:4326", -126.54, 54.15,
"EPSG:3005", 964813.103719, 1016486.305862,
0.0001, 0.2 * APPROX_METRE_IN_DEGREES)

checkTransformAndInverse(
"EPSG:32633", 249032.839239894, 7183612.30572229,
"EPSG:4326", 9.735465995810884, 64.68347938257097,
"EPSG:4326", 9.735465995870696, 64.68347938261206,
0.000001, 0.3 * APPROX_METRE_IN_DEGREES )

checkTransformAndInverse(
"EPSG:32636", 500000, 4649776.22482,
"EPSG:32636", 500000, 4649776.224819178,
"EPSG:4326", 33, 42,
0.000001, 20 * APPROX_METRE_IN_DEGREES )
}
Expand Down
5 changes: 2 additions & 3 deletions proj4/src/test/scala/geotrellis/proj4/io/wkt/WKTTest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,8 @@ class WKTTest extends AnyFunSpec {
assert(comparisonCode == epsgCodeOfWKT)
}

// https://github.com/locationtech/proj4j/issues/61
it("should return the EPSG code(3785) of the passed WKT string with proj4 extension") {
val comparisonCode = "EPSG:3785" // "EPSG:3857" // https://github.com/locationtech/proj4j/issues/61
it("should return the EPSG code(3857) of the passed WKT string with proj4 extension") {
val comparisonCode = "EPSG:3857"
Comment on lines +67 to +68
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test properly works now!


val epsgCodeOfWKT = WKT.getEpsgStringCode("PROJCS[\"WGS 84 / Pseudo-Mercator\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Mercator_1SP\"],PARAMETER[\"central_meridian\",0],PARAMETER[\"scale_factor\",1],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],EXTENSION[\"PROJ4\",\"+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs\"],AUTHORITY[\"EPSG\",\"3857\"]]").get
assert(comparisonCode == epsgCodeOfWKT)
Expand Down
2 changes: 1 addition & 1 deletion project/Dependencies.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ object Dependencies {
val scalacheck = "org.scalacheck" %% "scalacheck" % "1.15.2"
val scalaXml = "org.scala-lang.modules" %% "scala-xml" % "1.3.0"
val jts = "org.locationtech.jts" % "jts-core" % "1.17.1"
val proj4j = "org.locationtech.proj4j" % "proj4j" % "1.1.1"
val proj4j = "org.locationtech.proj4j" % "proj4j" % "1.1.2"
val openCSV = "com.opencsv" % "opencsv" % "5.3"
val spire = "org.typelevel" %% "spire" % Version.spire
val spireMacro = "org.typelevel" %% "spire-macros" % Version.spire
Expand Down