.NET Apache Spark write to ElasticSearch

[markookram]

Does .NET for Apache Spark supports writing to Elastic Search?

If not, do you have a hint when that might be available?

Thank you

[GoEddie]

I haven't used it so can't confirm if this works but if you can get the elastic search hadoop jar and use this to write then it should work:

df.Write
  .Format("org.elasticsearch.spark.sql")
  .Option("es.nodes.wan.only","true")
  .Option("es.port","443")
  .Option("es.net.ssl","true")
  .Option("es.nodes", esURL)
  .Mode("Overwrite")
  .Save("index/dogs")

Download:

https://www.elastic.co/downloads/hadoop

Inside the download is "elasticsearch-spark-20_2.11-7.10.0.jar" pass that to your Apache Spark job when you submit your .NET for Apache Spark application using "--jars /path/to/elasticsearch-spark-20_2.11-7.10.0.jar"

The code was adopted from https://docs.databricks.com/data/data-sources/elasticsearch.html

The other examples for writing to ES all seem to use the RDD API which isn't available today.

[markookram]

@GoEddie tnx.
I'll try, and will share experience.
:):)