Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Spark 2.4.3 #238

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Support for Spark 2.4.3 #238

wants to merge 5 commits into from

Conversation

mscnop
Copy link

@mscnop mscnop commented May 20, 2019

Hi, this is my first ever PR to anything, so please bear with me if I have done something terribly illegal/bad practice :)

I have updated a range of the dependencies to allow for Magellan compatibility with Spark 2.4.3 and Scala 2.11.12. In order to make it work with the new versions of the dependencies I had to change a few things regarding class/object creation.

All tests are passing, when I run "sbt test".

Best Mads

@codecov-io
Copy link

codecov-io commented May 20, 2019

Codecov Report

Merging #238 into master will decrease coverage by 0.16%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #238      +/-   ##
==========================================
- Coverage   90.12%   89.96%   -0.17%     
==========================================
  Files          52       52              
  Lines        1782     1783       +1     
  Branches      125      124       -1     
==========================================
- Hits         1606     1604       -2     
- Misses        176      179       +3
Impacted Files Coverage Δ
...in/scala/magellan/mapreduce/ShapeInputFormat.scala 90.62% <50%> (ø) ⬆️
src/main/scala/magellan/catalyst/SpatialJoin.scala 93.02% <75%> (-2.22%) ⬇️
src/main/scala/magellan/DoubleArrayData.scala 16.66% <0%> (-4.17%) ⬇️
src/main/scala/magellan/IntegerArrayData.scala 16.66% <0%> (-4.17%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 402df2f...de1ca19. Read the comment docs.

@khajaasmath786
Copy link

I am also looking for spark 2.4 support. getting below exceptions with 1.5 version of magellan.

Exception in thread "main" java.lang.VerifyError: class magellan.catalyst.Spatia lJoinHint overrides final method canonicalized.()Lorg/apache/spark/sql/catalyst/ plans/QueryPlan;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14 2)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

@khajaasmath786
Copy link

I was able to get this working with https://github.com/mscno/magellan for spark2.4 .

@harsha2010
Copy link
Owner

Does this preserve backward compatibility with Spark 2.3?

@khajaasmath786
Copy link

Does this preserve backward compatibility with Spark 2.3?

I just tested it out for spark2.1 and it doesn't work for it.

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.cat alyst.plans.logical.LogicalPlan.transformUp(Lscala/PartialFunction;)Lorg/apache/ spark/sql/catalyst/plans/logical/LogicalPlan;
at magellan.catalyst.SpatialJoin.apply(SpatialJoin.scala:36)
at magellan.catalyst.SpatialJoin.apply(SpatialJoin.scala:27)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$ anonfun$apply$1.apply(RuleExecutor.scala:85)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$ anonfun$apply$1.apply(RuleExecutor.scala:82)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized .scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.a pply(RuleExecutor.scala:82)

@mscnop
Copy link
Author

mscnop commented May 29, 2019

Hi,

When i did the changes, I was able to make it work for both Spark 2.3.x and Spark 2.4.x. And both versions compile and test without errors in the CI travis builds: CI Results.
I was not able to maintain backwards compatability with SPark 2.1.x or Spark 2.2.x. /Mads

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants