Apache Spark

[marijaselakovic]

About

Would be interesting to try out in a similar fashion: https://questdb.io/docs/third-party-tools/spark/

Outcome

Tutorial on Apache Spark and CrateDB: Getting started with Apache Spark and CrateDB: A step by step tutorial

What's Inside

• Install pyspark
• Reading data from CrateDB
• Writing data to CrateDB
• If first tests succeed, try writing more complex types
• Document in confluence
• Write a tutorial
• Publish tutorial on community

[amotl]

Status

About Apache Spark specifically, we don't present much, most probably because connecting to Spark using vanilla pgJDBC works reasonably.

About advertisement matters, it's still very poor, so we should improve dearly. 🍀

Needs

I think the need for integrating optimally with Apache Spark is high, so we should elevate our attention.

/cc @surister, @zolbatar, @kneth, @ckurze

202x-

• Dataframes: Apache Spark #67
• https://github.com/crate/roadmap/issues/23
• https://github.com/crate/roadmap/issues/53
• https://github.com/crate/tech-content/issues/71
• https://github.com/crate/support/issues/482

2017-

• Requesting Open Source Features crate#5835
• Writing a spark dataframe to crate using overwrite mode generates type conflict exception and gets the table deleted crate#7118
• User profile table, with 200 columns, upsert data using spark concurrency, please advise if the scheme is feasible?What's the catch? crate#8116
• Crate-JDBC runing on Zeppelin (Spark 2.1.0) Issue crate-jdbc#228
• https://github.com/crate/support/issues/27
• https://github.com/crate/c-apporchid/issues/5
• https://github.com/crate/c-apporchid/issues/12
• https://github.com/crate/c-apporchid/issues/14
• https://github.com/crate/p-roosi/issues/3
• https://github.com/crate/p-roosi/issues/5

Initiatives

I hope we will have a dedicated Spark track within upcoming initiatives.

• https://github.com/crate/roadmap/issues/91
• https://github.com/crate/roadmap/issues/81
• https://github.com/crate/writing/issues/24