-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Spark 2.4.3 #238
base: master
Are you sure you want to change the base?
Conversation
Scala 2.11.12 Spark 2.4.3 Jackson XML: 2.9.8 Fastparse 0.4.4 Scalatest 3.0.5 Guava 27-1 Commons-io 2.6 Sbt Assembly 0.14.9 Sbt 0.13.18
Codecov Report
@@ Coverage Diff @@
## master #238 +/- ##
==========================================
- Coverage 90.12% 89.96% -0.17%
==========================================
Files 52 52
Lines 1782 1783 +1
Branches 125 124 -1
==========================================
- Hits 1606 1604 -2
- Misses 176 179 +3
Continue to review full report at Codecov.
|
I am also looking for spark 2.4 support. getting below exceptions with 1.5 version of magellan. Exception in thread "main" java.lang.VerifyError: class magellan.catalyst.Spatia lJoinHint overrides final method canonicalized.()Lorg/apache/spark/sql/catalyst/ plans/QueryPlan; |
I was able to get this working with https://github.com/mscno/magellan for spark2.4 . |
Does this preserve backward compatibility with Spark 2.3? |
I just tested it out for spark2.1 and it doesn't work for it. Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.cat alyst.plans.logical.LogicalPlan.transformUp(Lscala/PartialFunction;)Lorg/apache/ spark/sql/catalyst/plans/logical/LogicalPlan; |
Hi, When i did the changes, I was able to make it work for both Spark 2.3.x and Spark 2.4.x. And both versions compile and test without errors in the CI travis builds: CI Results. |
Hi, this is my first ever PR to anything, so please bear with me if I have done something terribly illegal/bad practice :)
I have updated a range of the dependencies to allow for Magellan compatibility with Spark 2.4.3 and Scala 2.11.12. In order to make it work with the new versions of the dependencies I had to change a few things regarding class/object creation.
All tests are passing, when I run "sbt test".
Best Mads