John Snow Labs Spark-NLP 3.4.1: TF session warmup, a new F1 metric to track to save the best model in NerDL, new T5 models like WikiSQL or grammar corrector, other new multi-lingual state-of-the-art models, and bug fixes!

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 3.4.1! This release comes with a TF session warmup in 3 annotators where the first inference was slower than the rest, adding a new param to choose which F1 to track to save the best model when training a NerDL model, new T5 models such as text to SQL or grammar correction, new multi-lingual state-of-the-art models, and other bug fixes!

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features & Enhancements

• Implement TF Session warmup for MarianTransformer, T5Transformer, and GPT2Transformer annotators. The first inference for these annotators used to take between 15-20 seconds, now with the warmup session all the inferences including the first time will be the same Implement TF Session warmup for some TF-based annotators #6773
• Add bestModelMetric param to choose between Micro-average or Macro-average for best model Add bestModelMetric param to choose between Micro-average or Macro-average for best model #6749
• Add trimWhitespace and preservePosition params to RegexTokenizer Feature/regex tok trim #6806
• Add a new setSentenceMatch param to EntityRuler to match entities across documents/sentences and not just tokens Adding sentence match parameter to EntityRuler #6841
• Add support spark32 and real_time_output flags in sparknlp.start() function at the same time Feature/init spark32 realtime output #6822
• Allow users to set tasks in the T5Transformer annotator

---

Bug Fixes

• Fix random NullPointerException when using TensorFlow models without Kyro serialization Revert back to none optional session for more reliable serializations #6741
• Fix RecursiveTokenizerModel not being readable in a saved Pipeline Fix RecursiveTokenizerModel not being readable in a saved Pipeline #6748
• Fix ContextSpellCheckerApproach not being trained on Databricks Serialization context spell #6750
• Fix ContextSpellCheckerModel wrong order of tokens it's used with Sentence Detectors fix tokens order based in ContextSpellCheckerModel when there are sentences #6799
• Fix GraphExtraction when fullAnnotate and document are used at the same time Fix GraphExtraction crashing when fullAnnotate and document are used at the same time #6845
• Fix Word2VecModel being cast to Doc2VecModel by mistake Fix Word2VecModel being cast to Doc2VecModel by mistake #6849
• Fix broken sentence indexing in BertEmbeddings that impacted SentenceEmbeddings for text classification Fix broken BertEmbeddings wrong sentence indexing #6867
• Fix missing setExceotionsPath param in Tokenizer when it's used in Python Fix broken setExceptionsPath in Tokenizer in Python #6868
• Fix the wrong metrics being mentioned when useBestModel was enabled. The documentation said Micro-averaged F1 but in fact, it was Macro-average F1 (the option to choose which metric to be tracked is now available as well)
• Update broken slow unit tests Update slow unit tests #6767

---

Models

New state-of-the-art models in English, French, Vietnamese, Dutch, and Indian (Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu)

Featured Pretrained Models

Model	Name	Lang
T5Transformer	t5_informal_to_formal_styletransfer	en
T5Transformer	t5_formal_to_informal_styletransfer	en
T5Transformer	t5_passive_to_active_styletransfer	en
T5Transformer	t5_active_to_passive_styletransfer	en
T5Transformer	t5_grammar_error_corrector	en
T5Transformer	t5_small_wikiSQL	en
LongformerEmbeddings	clinical_longformer	en
AlbertEmbeddings	albert_indic	xx
DistilBertEmbeddings	distilbert_base_cased	vi
BertForSequenceClassification	bert_sequence_classifier_news_sentiment	de
BertForSequenceClassification	bert_sequence_classifier_emotion	en
DistilBertForTokenClassification	distilbert_token_classifier_typo_detector	en
DistilBertForTokenClassification	distilbert_base_token_classifier_masakhaner	xx
WordEmbeddingsModel	word2vec_wiki_1000	fr
WordEmbeddingsModel	word2vec_wac_200	fr
WordEmbeddingsModel	w2v_cc_300d	fr

---

Documentation

• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP in Action
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.4.1

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.4.1

spark-nlp on Apache Spark 3.2.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark32_2.12:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark32_2.12:3.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark32_2.12:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark32_2.12:3.4.1

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.4.1

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.4.1

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp on Apache Spark 3.2.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark32_2.12</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark32_2.12</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.4.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.4.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.4.1.jar

• GPU on Apache Spark 3.0.x/3.1.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.4.1.jar

• CPU on Apache Spark 3.2.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark32-assembly-3.4.1.jar

• GPU on Apache Spark 3.2.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark32-assembly-3.4.1.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.4.1.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.4.1.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.4.1.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.4.1.jar

What's Changed

• 2022-01-06-bert_token_classifier_ner_chemicals_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_chemicals_en #6731
• 2022-01-06-bert_token_classifier_ner_anatomy_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_anatomy_en #6727
• Update 2022-01-05-re_drugprot_clinical_en.md by @josejuanmartinez in Update 2022-01-05-re_drugprot_clinical_en.md #6725
• fix indentation again by @murat-gunay in fix indentation again #6717
• 2022-01-06-bert_token_classifier_ner_cellular_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_cellular_en #6728
• 2022-01-06-ner_deid_generic_de by @jsl-models in 2022-01-06-ner_deid_generic_de #6730
• 2022-01-06-ner_deid_subentity_de by @jsl-models in 2022-01-06-ner_deid_subentity_de #6729
• 2022-01-06-bert_token_classifier_ner_jsl_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_jsl_en #6738
• 2022-01-06-bert_token_classifier_ner_jsl_slim_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_jsl_slim_en #6737
• 2022-01-06-bert_token_classifier_ner_clinical_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_clinical_en #6736
• 2022-01-06-bert_token_classifier_ner_drugs_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_drugs_en #6735
• 2022-01-06-bert_token_classifier_ner_deid_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_deid_en #6734
• 2022-01-06-bert_token_classifier_ner_chemprot_en by @jsl-models in 2022-01-06-bert_token_classifier_ner_chemprot_en #6732
• 2022-01-06-clinical_deidentification_de by @jsl-models in 2022-01-06-clinical_deidentification_de #6739
• 2022-01-07-bert_token_classifier_ner_bacteria_en by @jsl-models in 2022-01-07-bert_token_classifier_ner_bacteria_en #6740
• Fix true to True for python sample by @xyutech in Fix true to True for python sample #6583
• md files updated by @Cabir40 in md files updated #6742
• 2022-01-08-clinical_deidentification_de by @jsl-models in 2022-01-08-clinical_deidentification_de #6744
• Update licensed_release_notes.md by @muhammetsnts in Update licensed_release_notes.md #6745
• Models hub internal by @muhammetsnts in Models hub internal #6746
• Added doc part for VisualDocumentNERv2 by @xyutech in Added doc part for VisualDocumentNERv2 #6690
• [skip ci] Create PR 3.4.0-healthcare-docs-0af2adc5b515ab1cb8058c2d25e5ddda984fd676-7 by @jsl-builder in [skip ci] Create PR 3.4.0-healthcare-docs-0af2adc5b515ab1cb8058c2d25e5ddda984fd676-7 #6743
• Docs/alab2.6.0 by @diatrambitas in Docs/alab2.6.0 #6757
• added documentation for Annotation Page Configuration and Rules Page. by @diatrambitas in added documentation for Annotation Page Configuration and Rules Page. #6761
• Adds Embeddings dimension info to roberta_base_biomedical_es by @josejuanmartinez in Adds Embeddings dimension info to roberta_base_biomedical_es #6765
• 2022-01-11-pos_ud_bokmaal_nb by @jsl-models in 2022-01-11-pos_ud_bokmaal_nb #6752
• 2022-01-12-t5_small_wikiSQL_en by @jsl-models in 2022-01-12-t5_small_wikiSQL_en #6758
• 2022-01-12-t5_grammar_error_corrector_en by @jsl-models in 2022-01-12-t5_grammar_error_corrector_en #6759
• 2022-01-12-t5_formal_to_informal_styletransfer_en by @jsl-models in 2022-01-12-t5_formal_to_informal_styletransfer_en #6760
• 2022-01-12-t5_informal_to_formal_styletransfer_en by @jsl-models in 2022-01-12-t5_informal_to_formal_styletransfer_en #6762
• 2022-01-12-t5_active_to_passive_styletransfer_en by @jsl-models in 2022-01-12-t5_active_to_passive_styletransfer_en #6763
• 2022-01-12-t5_passive_to_active_styletransfer_en by @jsl-models in 2022-01-12-t5_passive_to_active_styletransfer_en #6764
• 2022-01-13-distilbert_base_cased_vi by @jsl-models in 2022-01-13-distilbert_base_cased_vi #6766
• 2022-01-13-roberta_base_biomedical_es by @jsl-models in 2022-01-13-roberta_base_biomedical_es #6768
• Models hub by @maziyarpanahi in Models hub #6769
• Add new docs articles by @agsfer in Add new docs articles #6770
• Update best_practices_pretrained_models.md by @muhammetsnts in Update best_practices_pretrained_models.md #6774
• 2022-01-14-bert_sequence_classifier_emotion_en by @jsl-models in 2022-01-14-bert_sequence_classifier_emotion_en #6771
• 2022-01-18-bert_sequence_classifier_sentiment_da by @jsl-models in 2022-01-18-bert_sequence_classifier_sentiment_da #6782
• 2022-01-18-bert_sequence_classifier_news_sentiment_de by @jsl-models in 2022-01-18-bert_sequence_classifier_news_sentiment_de #6783
• Updates wrong name of the model - by @C-K-Loan by @josejuanmartinez in Updates wrong name of the model - by @C-K-Loan #6794
• Update 2021-12-03-xlm_roberta_large_token_classification_ner_id.md by @josejuanmartinez in Update 2021-12-03-xlm_roberta_large_token_classification_ner_id.md #6795
• Hide broken links in Learn section by @agsfer in Hide broken links in Learn section #6784
• Allow T5 models to change task by @DevinTDHa in Allow T5 models to change task #6781
• Update evaluation.md by @muhammetsnts in Update evaluation.md #6809
• 2022-01-23-ner_ncbi_glove_100d_en by @jsl-models in 2022-01-23-ner_ncbi_glove_100d_en #6807
• Revert back to none optional session for more reliable serializations by @maziyarpanahi in Revert back to none optional session for more reliable serializations #6741
• Fix RecursiveTokenizerModel not being readable in a saved Pipeline by @maziyarpanahi in Fix RecursiveTokenizerModel not being readable in a saved Pipeline #6748
• Add bestModelMetric param to choose between Micro-average or Macro-average for best model by @maziyarpanahi in Add bestModelMetric param to choose between Micro-average or Macro-average for best model #6749
• Serialization context spell by @albertoandreottiATgmail in Serialization context spell #6750
• Update slow unit tests by @maziyarpanahi in Update slow unit tests #6767
• Implement TF Session warmup for some TF-based annotators by @maziyarpanahi in Implement TF Session warmup for some TF-based annotators #6773
• fix tokens order based in ContextSpellCheckerModel when there are sentences by @xusliebana in fix tokens order based in ContextSpellCheckerModel when there are sentences #6799
• Feature/regex tok trim by @wolliq in Feature/regex tok trim #6806
• Documentation spark nlp internal 3.4.0 by @xusliebana in Documentation spark nlp internal 3.4.0 #6823
• FEATURE NMH-58: Add sort by views and date [skip-test] by @KshitizGIT in FEATURE NMH-58: Add sort by views and date [skip-test] #6800
• Update Annotation_tool_Json_reader.md by @HashamUlHaq in Update Annotation_tool_Json_reader.md #6816
• Add an example of using an external volume with the NLP Server [skip test] by @pabla in Add an example of using an external volume with the NLP Server [skip test] #6814
• 2022-01-19-distilbert_token_classifier_typo_detector_en by @jsl-models in 2022-01-19-distilbert_token_classifier_typo_detector_en #6796
• 2022-01-18-distilbert_base_token_classifier_masakhaner_xx by @jsl-models in 2022-01-18-distilbert_base_token_classifier_masakhaner_xx #6793
• 2022-01-19-distilbert_token_classifier_typo_detector_is by @jsl-models in 2022-01-19-distilbert_token_classifier_typo_detector_is #6824
• Doc/alab pagination by @rpranab in Doc/alab pagination #6830
• Fixing entitiy section to be compatible with ALab by @josejuanmartinez in Fixing entitiy section to be compatible with ALab #6747
• doesn't work properly, removing model by @Meryem1425 in doesn't work properly, removing model #6828
• Feature/init spark32 realtime output by @maziyarpanahi in Feature/init spark32 realtime output #6822
• Adding sentence match parameter to EntityRuler by @danilojsl in Adding sentence match parameter to EntityRuler #6841
• 2022-02-01-word2vec_wac_200_fr by @jsl-models in 2022-02-01-word2vec_wac_200_fr #6843
• Scala fixes by @josejuanmartinez in Scala fixes #6844
• 2022-01-26-word2vec_wiki_1000_fr by @jsl-models in 2022-01-26-word2vec_wiki_1000_fr #6825
• Fix GraphExtraction crashing when fullAnnotate and document are used at the same time by @maziyarpanahi in Fix GraphExtraction crashing when fullAnnotate and document are used at the same time #6845
• Add new learn docs from PRP page by @agsfer in Add new learn docs from PRP page #6815
• 2022-02-03-w2v_cc_300d_fr by @jsl-models in 2022-02-03-w2v_cc_300d_fr #6848
• updated for ner_traffic md file by @Damla-Gurbaz in updated for ner_traffic md file #6847
• Fix of several md files with typos by @egenc in Fix of several md files with typos #6864
• models hub existing licensed models mardown Scala example code fixes by @luca-martial in models hub existing licensed models mardown Scala example code fixes #6865
• Fix Word2VecModel being cast to Doc2VecModel by mistake by @maziyarpanahi in Fix Word2VecModel being cast to Doc2VecModel by mistake #6849
• 2022-01-26-albert_indic_xx by @jsl-models in 2022-01-26-albert_indic_xx #6826
• Fix broken BertEmbeddings wrong sentence indexing by @maziyarpanahi in Fix broken BertEmbeddings wrong sentence indexing #6867
• Fix broken setExceptionsPath in Tokenizer in Python by @maziyarpanahi in Fix broken setExceptionsPath in Tokenizer in Python #6868
• Update benchmark.md by @muhammetsnts in Update benchmark.md #6871
• Fixing benchmark and move Indian languages by @agsfer in Fixing benchmark and move Indian languages #6872
• 2022-02-08-clinical_longformer_en by @jsl-models in 2022-02-08-clinical_longformer_en #6875
• Release/341 release candidate by @maziyarpanahi in Release/341 release candidate #6819
• Models hub by @maziyarpanahi in Models hub #6881

New Contributors

• @Cabir40 made their first contribution in modelhub md files updated #6685
• @rpranab made their first contribution in Doc/alab pagination #6830
• @Meryem1425 made their first contribution in doesn't work properly, removing model #6828
• @Damla-Gurbaz made their first contribution in updated for ner_traffic md file #6847

Full Changelog: 3.4.0...3.4.1

---

This discussion was created from the release John Snow Labs Spark-NLP 3.4.1: TF session warmup, a new F1 metric to track to save the best model in NerDL, new T5 models like WikiSQL or grammar corrector, other new multi-lingual state-of-the-art models, and bug fixes!.