John Snow Labs Spark-NLP 3.3.3: New DistilBERT for Sequence Classification, new trainable and distributed Doc2Vec, BERT improvements on GPU, new state-of-the-art DistilBERT models for topic and sentiment detection, enhancements, and bug fixes!

[maziyarpanahi]

---

Overview

(knock, knock, knock) Penny? Yes, this is a very special release if you are obsessed with the number 3 as much as we are! So we are pleased to announce Spark NLP 🚀 3.3.3 release! 🎉 🎊 🎈

This release comes with a new DistilBertForSequenceClassification annotator for existing or fine-tuned DistilBERT models for Text Classification on HuggingFace, new distributed and trainable Doc2Vec annotator based on Word2Vec implementation in Spark ML, improving BertEmbeddings and BertSentenceEmbeddings on a single machine on a GPU device where the DataFrame has 1 sentence per row or input column is set to document, new state-of-the-art fine-tuned DistilBERT models for Sequence Classification, enhancements, bug fixes, and more!

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features and Enhancements

• NEW: Introducing DistilBertForSequenceClassification annotator in Spark NLP 🚀. DistilBertForSequenceClassification DistilBertForSequenceClassification can load DistilBERT Models with sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for multi-class document classification tasks. This annotator is compatible with all the models trained/fine-tuned by using DistilBertForSequenceClassification or TFDistilBertForSequenceClassification in HuggingFace 🤗
• NEW: Introducing trainable and distributed Doc2Vec annotators based on Word2Vec in Spark ML
• Improving BertEmbeddings for single document/sentence DataFrame per row on a single machine with a GPU device
• Improving BertSentenceEmbeddings for single document/sentence DataFrame per row on a single machine with a GPU device
• Add a new feature to the CoNLL() class, allowing it to read multiple CoNLL files at the same time into a single DataFrame
• Add support for Long type in label column for ClassifierDLApproach and SentimentDLApproach
• Add script to setup AWS SageMaker thanks to @xegulon
• Add instructions to setup Amazon Linux 2

---

Bug Fixes

• Improve models and pipelines resolutions in Spark NLP when wrong models/pipelines are downloaded regardless of their Apache Spark version
• Fix MarianTransformer bug on empty sequences
• Fix TFInvalidArgumentException in MarianTransformer for sequences longer than 512
• Fix MarianTransformer multi-lingual models and pipelines such as opus_mt_mul_en and opus_mt_mul_en
• Fix a bug in DateMatcher and MultiDateMatcher when detecting month from subwords by mistake
• Add the missing lemma_antbnc model to Models Hub
• Add the missing sentiment_vivekn model to Models Hub
• Add the missing spellcheck_norvig model to Models Hub

---

Models

New state-of-the-art fine-tuned DistilBERT models for Sequence Classification:

Featured Pretrained Models

Model	Name	Build	Lang
DistilBertForSequenceClassification	distilbert_sequence_classifier_sst2	en	3.3.3
DistilBertForSequenceClassification	distilbert_sequence_classifier_policy	en	3.3.3
DistilBertForSequenceClassification	distilbert_sequence_classifier_industry	en	3.3.3
DistilBertForSequenceClassification	distilbert_sequence_classifier_emotion	en	3.3.3
DistilBertForSequenceClassification	distilbert_sequence_classifier_banking77	``	3.3.3
DistilBertForSequenceClassification	distilbert_multilingual_sequence_classifier_allocine	fr	3.3.3
DistilBertForSequenceClassification	distilbert_base_sequence_classifier_imdb	ur	3.3.3
DistilBertForSequenceClassification	distilbert_base_sequence_classifier_imdb	en	3.3.3
DistilBertForSequenceClassification	distilbert_base_sequence_classifier_amazon_polarity	en	3.3.3
DistilBertForSequenceClassification	distilbert_base_sequence_classifier_ag_news	en	3.3.3
Doc2VecModel	doc2vec_gigaword_300	en	3.3.3
Doc2VecModel	doc2vec_gigaword_wiki_300	en	3.3.3

The complete list of all 4000+ models & pipelines in 200+ languages is available on Models Hub.

New Notebooks

Spark NLP	Notebooks	Colab
DistilBertForSequenceClassification	HuggingFace in Spark NLP - DistilBertForSequenceClassification
Doc2Vec	Train Doc2Vec for Text Classification

---

Documentation

• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP in Action
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.3.3

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.3.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.3.3

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.3.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.3.3

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.3.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.3.3

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.3.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.3.3</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.3.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.3.3</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.3.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.3.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.3.3.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.3.3.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.3.3.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.3.3.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.3.3.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.3.3.jar

What's Changed

• 2021-11-03-bert_token_classifier_ner_cellular_en by @jsl-models in 2021-11-03-bert_token_classifier_ner_cellular_en #6378
• Fix Roberta model by @josejuanmartinez in Fix roberta model #6384
• 2021-11-03-robertaresolve_snomed_es by @jsl-models in 2021-11-03-robertaresolve_snomed_es #6372
• 2021-11-03-bert_token_classifier_ner_bionlp_en by @jsl-models in 2021-11-03-bert_token_classifier_ner_bionlp_en #6385
• update and fix tags by @murat-gunay in update and fix tags #6388
• 2021-11-03-ner_profiling_biobert_en by @jsl-models in 2021-11-03-ner_profiling_biobert_en #6391
• Update 2021-09-23-ner_profiling_biobert_en.md by @muhammetsnts in Update 2021-09-23-ner_profiling_biobert_en.md #6392
• Update 2021-09-24-ner_profiling_clinical_en.md by @muhammetsnts in Update 2021-09-24-ner_profiling_clinical_en.md #6390
• 2021-11-03-ner_profiling_clinical_en by @jsl-models in 2021-11-03-ner_profiling_clinical_en #6389
• update and fix scala section&tag section by @murat-gunay in update and fix scala section&tag section #6387
• 2021-11-04-sbiobertresolve_rxnorm_augmented_en by @jsl-models in 2021-11-04-sbiobertresolve_rxnorm_augmented_en #6398
• Docs: Fixed some issues with Comet documentation by @DevinTDHa in Docs: Fixed some issues with Comet documentation #6397
• Changes Roberta es embeddings link from clinical to the public by @josejuanmartinez in Changes Roberta es embeddings link from clinical to public #6404
• Removes "licensed" tag from Roberta biomedical by @josejuanmartinez in Removes "licensed" tag from roberta biomedical #6406
• Updating Roberta embeddings in snomed example by @josejuanmartinez in Updating roberta embeddings in snomed example #6405
• 2021-11-05-bert_sequence_classifier_question_statement_clinical_en by @jsl-models in 2021-11-05-bert_sequence_classifier_question_statement_clinical_en #6403
• Changes bad column header name by @josejuanmartinez in Changes bad column header name #6408
• 2021-11-05-ner_covid_trials_en by @jsl-models in 2021-11-05-ner_covid_trials_en #6402
• 2021-11-04-ner_chemd_clinical_en by @jsl-models in 2021-11-04-ner_chemd_clinical_en #6400
• 2021-11-04-bert_sequence_classifier_question_statement_en by @jsl-models in 2021-11-04-bert_sequence_classifier_question_statement_en #6399
• 2021-11-04-roberta_ner_diag_proc_es by @jsl-models in 2021-11-04-roberta_ner_diag_proc_es #6395
• Removes extra quotes in some code examples by @josejuanmartinez in Removes extra quotes in some code examples #6412
• Simplifies pipeline in the Spanish snomed example by @josejuanmartinez in Simplifies pipeline in the Spanish snomed example #6410
• Changes ner token-based results by chunk-based by @josejuanmartinez in Changes ner token-based results by chunk-based #6409
• Added release notes for healthcare 3.3.2 by @albertoandreottiATgmail in added release notes for healthcare 3.3.2 #6416
• version compat table by @albertoandreottiATgmail in version compat table #6417
• Update licensed_release_notes.md by @muhammetsnts in Update licensed_release_notes.md #6418
• general updates to alab instructions by @diatrambitas in general updates to alab instructions #6420
• [skip ci] Create PR 3.3.2-healthcare-docs-5fa00652053023311a991be6ed9aadf5289f0a41-3 by @jsl-builder in [skip ci] Create PR 3.3.2-healthcare-docs-5fa00652053023311a991be6ed9aadf5289f0a41-3 #6414
• 2021-11-06-bert_sequence_classifier_antisemitism_en by @jsl-models in 2021-11-06-bert_sequence_classifier_antisemitism_en #6415
• 2021-11-06-bert_sequence_classifier_hatexplain_en by @jsl-models in 2021-11-06-bert_sequence_classifier_hatexplain_en #6419
• 2021-11-04-classifierdl_use_sentiment_pipeline_tr by @jsl-models in 2021-11-04-classifierdl_use_sentiment_pipeline_tr #6394
• 2021-11-06-bert_sequence_classifier_trec_coarse_en by @jsl-models in 2021-11-06-bert_sequence_classifier_trec_coarse_en #6421
• 2021-11-07-bert_sequence_classifier_banking77_en by @jsl-models in 2021-11-07-bert_sequence_classifier_banking77_en #6422
• Docs/libs install by @diatrambitas in Docs/libs install #6433
• added the NLP libs install script by @diatrambitas in added the NLP libs install script #6435
• updated installation instructions by @diatrambitas in updated installation instructions #6437
• Removes extra quotes from Question vs Statement (Opensource) by @josejuanmartinez in Removes extra quotes from Question vs Statement (Opensource) #6413
• 2021-11-07-bert_sequence_classifier_song_lyrics_en by @jsl-models in 2021-11-07-bert_sequence_classifier_song_lyrics_en #6423
• 2021-11-07-bert_sequence_classifier_age_news_en by @jsl-models in 2021-11-07-bert_sequence_classifier_age_news_en #6424
• 2021-11-07-bert_sequence_classifier_sms_spam_en by @jsl-models in 2021-11-07-bert_sequence_classifier_sms_spam_en #6425
• fixing scala script by @murat-gunay in fixing scala script #6427
• urgent fix of entity typo by @murat-gunay in urgent fix of entity typo #6438
• 2021-11-09-sbiobertresolve_rxnorm_augmented_en by @jsl-models in 2021-11-09-sbiobertresolve_rxnorm_augmented_en #6441
• Updating models hub cards description by @gadde5300 in Updating models hub cards description #6444
• Adds models md to Models Hub for 3.3.2 Healthcare release by @josejuanmartinez in Adds models md to Models Hub for 3.3.2 Healthcare release #6434
• Fixing colon in titles by @josejuanmartinez in Fixing colon in titles #6445
• Updated 2021-06-01-redl_date_clinical_biobert_en.md by @gadde5300 in Updated 2021-06-01-redl_date_clinical_biobert_en.md #6448
• 2021-11-11-sbiobertresolve_snomed_procedures_measurements_en by @jsl-models in 2021-11-11-sbiobertresolve_snomed_procedures_measurements_en #6450
• 2021-11-11-classifierdl_bertwiki_finance_sentiment_pipeline_en by @jsl-models in 2021-11-11-classifierdl_bertwiki_finance_sentiment_pipeline_en #6451
• installation image is added by @egenc in installation image is added #6454
• Fixing scala code section typos for several models by @josejuanmartinez in Fixing scala code section typos for several models #6465
• Docs/alab2.3.0 by @diatrambitas in Docs/alab2.3.0 #6467
• Fixes robertaresolve_snomed name in code by @josejuanmartinez in Fixes robertaresolve_snomed name in code #6468
• Models hub m11.1 (fixes) by @gadde5300 in Models hub m11.1 (fixes) #6457
• Added Docs for Doc2Vec by @DevinTDHa in Added Docs for Doc2Vec #6471
• 2021-11-14-sbiobertresolve_mesh_en by @jsl-models in 2021-11-14-sbiobertresolve_mesh_en #6463
• Update 2021-11-14-sbiobertresolve_mesh_en.md by @muhammetsnts in Update 2021-11-14-sbiobertresolve_mesh_en.md #6478
• Adding timestamp in logging file's name by @danilojsl in Adding timestamp in logging file's name #6439
• Multi columns approach by @xusliebana in Multicolumns approach #6446
• feature: script for setting up spark and spark nlp on a sagemaker kernel by @xegulon in feature: script for setting up spark and spark nlp on a sagemaker kernel #6449
• amazon linux 2 installation by @egenc in amazon linux 2 installation #6455
• Modifies Models Resolution by @danilojsl in Modifies Models Resolution #6458
• Add support to labels as LongType in ClassifierDL and SentimentDL by @maziyarpanahi in Add support to labels as LongType in ClassifierDL and SentimentDL #6460
• Fix MarianTransformer crashing on both empty sequence or longer than 512 by @maziyarpanahi in Fix MarianTransformer crashing on both empty sequence or longer than 512 #6461
• Optimized batch processing for Bert sentence and word embeddings by @vankov in Optimized batch processing for Bert sentence and word embeddings #6462
• Introducing Distributed and Trainable Doc2Vec annotators by @maziyarpanahi in Introducing Distributed and Trainable Doc2Vec annotators #6474
• Introducing a DistilBertForSequenceClassification annotator by @maziyarpanahi in Introducing a DistilBertForSequenceClassification annotator #6475
• A new param to control which tokens are ignored in transformers' decoder output. by @vankov in A new param to control which tokens are ignored in transformers' decoder output. #6476
• Optimized Conll Reader by @albertoandreottiATgmail in Optimized Conll Reader #6482
• Updated docs for release 333 by @DevinTDHa in Updated docs for release 333 #6481
• Add new demos by @agsfer in Add new demos #6452
• added aws emr docs by @albertoandreottiATgmail in added aws emr docs #6484
• fixed links by @albertoandreottiATgmail in fixed links #6485
• Bugfix/date relaxed matcher subword month by @wolliq in Bugfix/date relaxed matcher subword month #6487
• 2021-11-21-doc2vec_gigaword_300_en by @jsl-models in 2021-11-21-doc2vec_gigaword_300_en #6489
• 2021-11-21-doc2vec_gigaword_wiki_300_en by @jsl-models in 2021-11-21-doc2vec_gigaword_wiki_300_en #6490
• 2021-11-21-distilbert_sequence_classifier_policy_en by @jsl-models in 2021-11-21-distilbert_sequence_classifier_policy_en #6491
• 2021-11-21-distilbert_sequence_classifier_industry_en by @jsl-models in 2021-11-21-distilbert_sequence_classifier_industry_en #6492
• 2021-11-21-distilbert_sequence_classifier_sst2_en by @jsl-models in 2021-11-21-distilbert_sequence_classifier_sst2_en #6493
• 2021-11-21-distilbert_sequence_classifier_emotion_en by @jsl-models in 2021-11-21-distilbert_sequence_classifier_emotion_en #6494
• 2021-11-21-distilbert_sequence_classifier_banking77_en by @jsl-models in 2021-11-21-distilbert_sequence_classifier_banking77_en #6495
• 2021-11-21-distilbert_multilingual_sequence_classifier_allocine_fr by @jsl-models in 2021-11-21-distilbert_multilingual_sequence_classifier_allocine_fr #6496
• 2021-11-21-distilbert_base_sequence_classifier_imdb_en by @jsl-models in 2021-11-21-distilbert_base_sequence_classifier_imdb_en #6497
• 2021-11-21-distilbert_base_sequence_classifier_ag_news_en by @jsl-models in 2021-11-21-distilbert_base_sequence_classifier_ag_news_en #6498
• 2021-11-21-distilbert_base_sequence_classifier_imdb_ur by @jsl-models in 2021-11-21-distilbert_base_sequence_classifier_imdb_ur #6499
• 2021-11-21-distilbert_base_sequence_classifier_amazon_polarity_en by @jsl-models in 2021-11-21-distilbert_base_sequence_classifier_amazon_polarity_en #6500
• Models hub by @maziyarpanahi in Models hub #6501
• Add a few changes by @agsfer in Add few changes #6503
• Release/333 release candidate by @maziyarpanahi in Release/333 release candidate #6480

New Contributors

• @xegulon made their first contribution in feature: script for setting up spark and spark nlp on a sagemaker kernel #6449

Full Changelog: 3.3.2...3.3.3