use native Tensorflow in SparkNLP

[cometta]

Is there any example where we can use the native tensorflow in SparkNLP library to train our model in distributed mode in Spark ? Any such example available?

[maziyarpanahi]

Hi,

All the trainable annotators are listed here: https://sparknlp.org/docs/en/training

We use TensorFlow as a backend/engine rather than making it available for any use of TF on Spark. For that, there are other libraries including the tensorflow-java

That said, there is no distributed training in tensorflow-java nor in most TF on Spark libraries. (the easiest way is to just use 1 single machine with a GPU and train the models which in case of Spark NLP it is more than enough. We optimized our trainable-models to not require high resources)