Reading/Writing to Nebula Graph through PySpark

[xs2tarunkukreja]

I need to develop ETL Pipeline in PySpark to write data in Nebula Graph. I have seen example of PySpark with Nebula Spark Connector. But code sample is using metaAddress only.

We have deployed Nebula Graph in K8S in different cluster. We have exposed only nebula-graphd-svc. Is there any way to access Nebula Graph through nebula-graphd-svc only?

Please also share the code for reference as well.

[wey-gu]

It's actually already supported, if you look into the nebula spark connector repo.

[xs2tarunkukreja]

In repo, I can see only one example of reading through PySpark -
df = spark.read.format(
"com.vesoft.nebula.connector.NebulaDataSource").option(
"type", "vertex").option(
"spaceName", "basketballplayer").option(
"label", "player").option(
"returnCols", "name,age").option(
"metaAddress", "metad0:9559").option(
"partitionNumber", 1).load()

Which is through meta address. Please share the code reference by using graphd-svc only.

[xs2tarunkukreja]

@wey-gu - So, as per your reply we can connect to Nebula by using Spark-Connector by using only graphd-svc without using meta address. Is my understanding correct? Please share code reference with meta address in PySpark

[wey-gu]

Dear @xs2tarunkukreja ,

sorry I was replying you via mobile client of github, so didnt point you the example.

see:

• https://github.com/vesoft-inc/nebula-spark-connector/pull/84/files the example
• support ngql reader nebula-spark-connector#78 the impl.
• https://github.com/vesoft-inc/nebula-algorithm/pull/64/files#diff-591b2c8ca6b9187bd03b9465ed4f291c705aab682b5ad662c377b954c65a740e how nebula-algorithm leveraged it

But for now, it seems only supports queries that returning edges as e in query as I see from the impl.

[wey-gu]

Also, it's highly recommended to leverage meta/storage to scan the data instead of doing so in query based way. It'll be more scalable/performant in large scale of data touched.

In k8s, this could be done by put spark in same namespace of nebulagraph.

[xs2tarunkukreja]

ok.. In your sample https://github.com/vesoft-inc/nebula-algorithm/pull/64/files#diff-591b2c8ca6b9187bd03b9465ed4f291c705aab682b5ad662c377b954c65a740e, we are using meta address.

val config =
NebulaConnectionConfig
.builder()
.withMetaAddress(metaAddress)
.withGraphAddress(graphAddress)
.withConenctionRetry(2)
.build()

[xs2tarunkukreja]

@wey-gu - You mentioned - "In k8s, this could be done by put spark in same namespace of nebulagraph". But we are running ETL Pipeline and Nebula Graph DB on different Cluster.