Masked Word Prediction using BERT

Overview

This project involves creating an AI to predict a masked word in a text sequence using the BERT language model. The project is divided into two main parts:

1. Implementing a program to predict masked words using BERT.
2. Analyzing the attention diagrams generated by the model to understand what BERT's attention heads focus on.

Requirements

• Python Version: The latest version of Python you should use is Python 3.12.
• Dependencies: Install the required dependencies using pip3 install -r requirements.txt.

How to Run

To run the program, use the following command:

$ python mask.py

Usage

Text: We turned down a narrow lane and passed through a small [MASK].

We turned down a narrow lane and passed through a small field.
We turned down a narrow lane and passed through a small clearing.
We turned down a narrow lane and passed through a small park.