Best practice for speeding up NER?

[santoshbs]

I am using spark-nlp for identifying named entities and tagging the text. I wanted to understand what is the best approach to speed up this process.

My text data is around 20 GB with each sentence in a separate line. I am calling spark-nlp with chunks of this data. Regardless of the chunk size, the speed is pretty slow compared to Spacy NER. The latter is at least 20 times faster.

This makes me believe that I might not be using spark-nlp in the best possible way. Are there any best practices to increase the speed when using CPU? Please advise.

[Smitspeed29]

I am new to spark ml pipelines. I am facing similar issue of speed wile doing inference. I also found spacy 20 times faster than spark nlp. So to speed up spark nlp should I run it on a bigger spark cluster to distribute tasks better.

[maziyarpanahi]

Hi,
We need actual info on the environment, hardware, number of rows, avg. length of documents, code snippets to reproduce the exact finidings, Spark config including memory and cores, etc.

Without them, it's really hard to compare and give any advice.