Need help in understanding spacy debug data output

[vignesh-spericorn]

Hello,

In the following output for spacy debug data command there is a line "⚠ 20912 words in training data without vectors"
What does this line mean ? and How does it affects the training & model performance ?

Kindly seeking the community's help.
Thanks in Advance

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, ner
6795 training docs
500 evaluation docs
⚠ 105 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 83786 total word(s) in the data (12761 unique)
ℹ 514157 vectors (514157 unique keys, 300 dimensions)
⚠ 20912 words in training data without vectors (25%)

========================== Named Entity Recognition ==========================
ℹ 3 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries

================================== Summary ==================================
✔ 6 checks passed
⚠ 2 warnings

[rmitsch]

Hi @vignesh-spericorn, this means that 20912 words in your training data do not have vector representations in the set of vectors you are using. This can happen for specialized or newer terms that are not part of the model's vocabulary.

Have a look at the docs for more info on this.

[rmitsch]

Out-of-vocabulary tokens will be assigned a zero-vector, which will impact downstream accuracy negatively (as these tokens' embeddings won't contain meaningful information).

[vignesh-spericorn]

Thank you for the reply. Is it possible to identify and create vectors for these 20912 words ?

[rmitsch]

You can use Token.is_oov for that.

[vignesh-spericorn]

Is it possible to create to create vectors for these 20912 words ?

[rmitsch]

You would need to train a new set of word embeddings from scratch (see also this discussion) or realign the word embeddings trained on the 20912 OOV words with the old embedding, which might be tricky.