AssertionError while running text_emojize.py

[vidyap-xgboost]

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-11-6c7dc2606552> in <module>()
      3 
      4 for i in flatten_list:
----> 5   deepmojify(i, top_n = 5)

1 frames
/content/torchMoji/torchmoji/sentence_tokenizer.py in tokenize_sentences(self, sentences, reset_stats, max_sentences)
    117         # may filter the sentences etc.
    118         if not self.uses_custom_wordgen and not self.ignore_sentences_with_only_custom:
--> 119             assert len(sentences) == next_insert
    120         else:
    121             # adjust based on actual tokens received

AssertionError: 

---

Hi,

The above error keeps coming when I run text_emojize.py file.

I have given a list of around 4700+ sentences for the model to convert it into 5 emojis.

I made changes to this block of code >> st = SentenceTokenizer(vocabulary, 100)

What am I doing wrong? Is it because I gave too many sentences?

[vidyap-xgboost]

@thomwolf @hiepph Please help me understand this error!

[vidyap-xgboost]

@setu4993 any idea about this?

[setu4993]

@vidyap-xgboost : Hmm, can't reproduce on a single test sentence... I tried the setup in this Colab from #32. I tried locally, though.

[vidyap-xgboost]

I've checked that particular row where the running of text_emojize.py stops.
If the row contains \n or \t or something like this, the whole dataset collapses.

Please add an exception to ignore such kind of rows or values in the list.

Thank you.