Transformer for text classification

It is well known that models such as Multilayer perceptrons (MLP) and convolutonal neural nets do not perform well when it comes to learning from long text snippets (They fails to capture the long term dependencies betwen words). LSTM tries to mitigate the problem but falls short for longer texts. In this notebook I train a custom transformer to aid the binary classification of long text. The dataset used is the IMDB movie reviews dataset. The dataset has an average of 323 words per review. The task is to classify each review as either holding a positive or negative sentiment.For more details have a look at the notebook