Fasttext Subword Embeddings in PyTorch

FastText is an incredible word embedding with a decent partial solution to handle OOV words and incorporate lexical similarity.

but what if we need to pass gradients through our fasttext embeddings?

Usage

Code snippet to demonstrate that it will replicate the original fasttext embeddings

# Implemented model gives the same emebddings
from src.model import Subword_Embedding
subword = SubwordEmbedding('resources/cc.en.300.bin')

# see something vec
something_vec = subword(['something', 'something is right'])
something_vec[:,:5]
tensor([[-0.0045,  0.0097,  0.0500,  0.0337, -0.0330],
        [ 0.0011,  0.0044,  0.0108,  0.0488, -0.0035]])

Converting original fasttext embeddings into ingestible formats

Download model.
python scripts/download_model.py "en"
Convert the weights of the downloaded model.
python scripts/convert_pretrained_weights.py

Modularity

To copy paste into your models, copy and paste 'src/model.py` simply as it is self-contained.

References

https://vimeo.com/234958672
https://www.aclweb.org/anthology/Q17-1010/
http://christopher5106.github.io/deep/learning/2020/04/02/fasttext_pretrained_embeddings_subword_word_representations.html