This project explores the application of pretrained
contextual embedding (PCE) models
in comparison to the bidirectional attentive
reader (baseline) implemented by the CS-388
staff. Particularly, I applied DistilBERT (Sanh
et al., 2019) as it is 60% faster to train
and retains 97% of the base BERT performance.
Model comparison was conducted on
the SQuAD 1.1 (Rajpurkar et al., 2016) and
SQuAD adverserial datasets (Jia and Liang,
2017). The HuggingFace implementation of
DistilBERT was utilized, hence I first had to
ensure the SQuAD datasets used match to
that provided by the class staff. Whereas the
baseline model achieved 60.63% and 46.76%
F1 scores on the SQuAD 1.1 and adverserial
SQuAD datasets, respectively, the fine-tuned
DistilBERT model achieved 85.4% and 69.4%,
respectively. Analysis shows that both models
struggle the most with ”why” questions, and
achieve lower F1 scores with increasing answer
lengths.
Explore the docs »
Report Bug
·
Request Feature
Distributed under the MIT License. See LICENSE for more information.
Name: @MJAljubran - [email protected]
Project Link: https://github.com/aljubrmj/CS388-Final-Project