Transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction \n",
        "\n",
        "The Transformer is a state-of-the-art model architecture in the field of natural language processing (NLP) that has gained significant attention for its remarkable performance on various sequence modeling tasks. Unlike traditional recurrent models that process inputs sequentially, the Transformer takes a different approach by leveraging the power of self-attention mechanisms. The core idea behind the Transformer is to enable each word in a sentence to directly attend to all other words, capturing rich contextual information and dependencies in parallel.\n",
        "\n",
        "At the heart of the Transformer is the self-attention mechanism, which allows the model to weigh the importance of different words in a sentence when generating representations. By attending to relevant words, the model can effectively focus on the most informative context for each word, regardless of its position in the sequence. This ability to capture long-range dependencies and consider the entire context simultaneously is a key strength of the Transformer, enabling it to handle complex linguistic structures and capture subtle relationships between words.\n",
        "\n",
        "Another significant advantage of the Transformer is its parallelizable nature. Unlike recurrent models, which process inputs sequentially and suffer from sequential computation bottlenecks, the Transformer can process the entire input sequence in parallel. This parallel processing, made possible by the self-attention mechanism, accelerates training and inference, making the Transformer highly efficient for large-scale NLP tasks. Additionally, the Transformer can handle variable-length input sequences without the need for padding or truncation, allowing it to accommodate diverse sentence lengths commonly found in natural language data."
      ],
      "metadata": {
        "id": "1qQu5WsoEOvD"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nRm74SesB6Mm",
        "outputId": "2c72fa5d-737e-4706-860b-fc831aa5f1c4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Transformer:\n",
            "Input shape: torch.Size([10, 20, 512])\n",
            "Output shape: torch.Size([10, 20, 512])\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class Transformer(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, num_heads, num_layers):\n",
        "        super(Transformer, self).__init__()\n",
        "        \n",
        "        self.embedding = nn.Linear(input_dim, hidden_dim)\n",
        "        self.encoder_layers = nn.ModuleList([\n",
        "            EncoderLayer(hidden_dim, num_heads) for _ in range(num_layers)\n",
        "        ])\n",
        "        self.output_layer = nn.Linear(hidden_dim, input_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.embedding(x)\n",
        "        \n",
        "        for layer in self.encoder_layers:\n",
        "            x = layer(x)\n",
        "        \n",
        "        output = self.output_layer(x)\n",
        "        return output\n",
        "\n",
        "# Example input and output for Transformer class\n",
        "input_dim = 512\n",
        "hidden_dim = 256\n",
        "num_heads = 8\n",
        "num_layers = 4\n",
        "\n",
        "model = Transformer(input_dim, hidden_dim, num_heads, num_layers)\n",
        "input_data = torch.randn(10, 20, input_dim)  # Example input tensor of shape (batch_size, sequence_length, input_dim)\n",
        "output_data = model(input_data)  # Example output tensor\n",
        "\n",
        "print(\"Transformer:\")\n",
        "print(\"Input shape:\", input_data.shape)\n",
        "print(\"Output shape:\", output_data.shape)\n",
        "print()\n",
        "\n",
        "\n",
        "class EncoderLayer(nn.Module):\n",
        "    def __init__(self, hidden_dim, num_heads):\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        \n",
        "        self.self_attention = MultiheadAttention(hidden_dim, num_heads)\n",
        "        self.feed_forward = FeedForward(hidden_dim)\n",
        "        \n",
        "        self.layer_norm1 = nn.LayerNorm(hidden_dim)\n",
        "        self.layer_norm2 = nn.LayerNorm(hidden_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x_residual = x\n",
        "        \n",
        "        x = self.layer_norm1(x)\n",
        "        x = self.self_attention(x)\n",
        "        x = x_residual + x\n",
        "        \n",
        "        x_residual = x\n",
        "        \n",
        "        x = self.layer_norm2(x)\n",
        "        x = self.feed_forward(x)\n",
        "        x = x_residual + x\n",
        "        \n",
        "        return x\n",
        "\n",
        "class MultiheadAttention(nn.Module):\n",
        "    def __init__(self, hidden_dim, num_heads):\n",
        "        super(MultiheadAttention, self).__init__()\n",
        "        \n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.num_heads = num_heads\n",
        "        \n",
        "        self.query = nn.Linear(hidden_dim, hidden_dim)\n",
        "        self.key = nn.Linear(hidden_dim, hidden_dim)\n",
        "        self.value = nn.Linear(hidden_dim, hidden_dim)\n",
        "        self.output = nn.Linear(hidden_dim, hidden_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        batch_size, seq_len, hidden_dim = x.size()\n",
        "        \n",
        "        query = self.query(x).view(batch_size, seq_len, self.num_heads, hidden_dim // self.num_heads).transpose(1, 2)\n",
        "        key = self.key(x).view(batch_size, seq_len, self.num_heads, hidden_dim // self.num_heads).transpose(1, 2)\n",
        "        value = self.value(x).view(batch_size, seq_len, self.num_heads, hidden_dim // self.num_heads).transpose(1, 2)\n",
        "        \n",
        "        scores = torch.matmul(query, key.transpose(-2, -1)) / (hidden_dim // self.num_heads) ** 0.5\n",
        "        attention_weights = nn.functional.softmax(scores, dim=-1)\n",
        "        \n",
        "        x = torch.matmul(attention_weights, value).transpose(1, 2).contiguous().view(batch_size, seq_len, hidden_dim)\n",
        "        x = self.output(x)\n",
        "        \n",
        "        return x\n",
        "\n",
        "class FeedForward(nn.Module):\n",
        "    def __init__(self, hidden_dim):\n",
        "        super(FeedForward, self).__init__()\n",
        "        \n",
        "        self.hidden_dim = hidden_dim\n",
        "        \n",
        "        self.linear1 = nn.Linear(hidden_dim, hidden_dim * 4)\n",
        "        self.linear2 = nn.Linear(hidden_dim * 4, hidden_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.linear1(x)\n",
        "        x = nn.functional.relu(x)\n",
        "        x = self.linear2(x)\n",
        "        \n",
        "        return x\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Example usage\n",
        "model = Transformer(input_dim=512, hidden_dim=256, num_heads=8, num_layers=4)\n",
        "input_data = torch.randn(10, 20, 512)  # Input tensor of shape (batch_size, sequence_length, input_dimension)\n",
        "output = model(input_data)  # Forward pass through the model\n",
        "print(output.shape)  # Print the shape of the output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6mU6QmP3Erpl",
        "outputId": "798976f0-e936-4748-f626-07f048a64fe7"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([10, 20, 512])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Remark, what does the tensor torch.randn(10, 20, 512) mean? \n",
        "\n",
        " the input tensor torch.randn(10, 20, 512) corresponds to a batch of 10 sentences, where each sentence can have a maximum of 20 words. Each word is represented by a 512-dimensional vector (word embedding). This tensor can be fed into a Transformer model for further processing, such as self-attention and positional encoding, to capture the relationships and dependencies between the words in each sentence."
      ],
      "metadata": {
        "id": "ZPsZkOUQSYhd"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "HHNlE5gMFYFS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MultiheadAttention\n"
      ],
      "metadata": {
        "id": "YE6QHw5nFYh7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "# Example input for MultiheadAttention\n",
        "input_dim = 20\n",
        "hidden_dim = 20\n",
        "num_heads = 4\n",
        "\n",
        "attention = MultiheadAttention(hidden_dim, num_heads)\n",
        "\n",
        "# Create a random input tensor\n",
        "input_tensor = torch.randn(30, 5, input_dim)  # Example input tensor of shape (batch_size, sequence_length, input_dim)\n",
        "\n",
        "# Perform the forward pass through the attention module\n",
        "output = attention(input_tensor)\n",
        "\n",
        "# Print the input and output shapes\n",
        "print(\"Input shape:\", input_tensor.shape)\n",
        "print(\"Output shape:\", output.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LdSB1JwAB624",
        "outputId": "563a4d9c-9c6c-4aa7-c1f7-02d1dd5dfdc7"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([30, 5, 20])\n",
            "Output shape: torch.Size([30, 5, 20])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Lets run a transformer on an NLP example "
      ],
      "metadata": {
        "id": "Br0RKYzeYcQp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class Transformer(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, num_heads, num_layers):\n",
        "        super(Transformer, self).__init__()\n",
        "\n",
        "        self.embedding = nn.Embedding(input_dim, hidden_dim)\n",
        "        self.encoder_layer = nn.TransformerEncoderLayer(hidden_dim, num_heads)\n",
        "        self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers)\n",
        "\n",
        "    def forward(self, input_tensor):\n",
        "        embedded = self.embedding(input_tensor)\n",
        "        encoded = self.encoder(embedded)\n",
        "        return encoded\n",
        "\n",
        "# Example input sentences\n",
        "sentences = [\"I love using transformers!\", \"This is another example sentence.\", \"Transformers are powerful models.\"]\n",
        "\n",
        "# Tokenize input sentences using whitespace tokenizer\n",
        "tokenized_sentences = [sentence.split() for sentence in sentences]\n",
        "\n",
        "# Determine the maximum sequence length\n",
        "max_seq_length = max(len(tokens) for tokens in tokenized_sentences)\n",
        "\n",
        "# Pad the tokenized sentences\n",
        "padded_sentences = [tokens + ['<pad>'] * (max_seq_length - len(tokens)) for tokens in tokenized_sentences]\n",
        "\n",
        "# Build vocabulary from padded sentences\n",
        "vocab = {}\n",
        "for tokens in padded_sentences:\n",
        "    for token in tokens:\n",
        "        if token not in vocab:\n",
        "            vocab[token] = len(vocab)\n",
        "\n",
        "# Convert padded sentences to input tensors\n",
        "batch_size = len(padded_sentences)\n",
        "input_tensor = torch.tensor([[vocab[token] for token in tokens] for tokens in padded_sentences], dtype=torch.long)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# Define the Transformer model\n",
        "input_dim = len(vocab)\n",
        "hidden_dim = 32\n",
        "num_heads = 4\n",
        "num_layers = 2\n",
        "\n",
        "transformer = Transformer(input_dim, hidden_dim, num_heads, num_layers)\n",
        "\n",
        "\n",
        "# the input tensor input_tensor represents a batch of input sentences encoded as integer indices, with dimensions [batch_size, sequence_length]. \n",
        "# It is converted into dense word embeddings by the embedding layer and then processed by the Transformer encoder to capture contextual information.\n",
        "\n",
        "\n",
        "# Run input tensor through the Transformer model\n",
        "output = transformer(input_tensor)\n",
        "\n",
        "# Print the input and output shapes\n",
        "print(\"Input shape:\", input_tensor.shape)\n",
        "print(\"Output shape:\", output.shape)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XR8HMGrEFa7q",
        "outputId": "68e18845-0510-47ea-fb6d-f633f4156d6d"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([3, 5])\n",
            "Output shape: torch.Size([3, 5, 32])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The input tensor meaning \n",
        "\n",
        " the context of the Transformer model, the input tensor input_tensor represents a batch of input sentences encoded as integers. It has the shape **[batch_size, sequence_length]**, where:\n",
        "\n",
        "* batch_size refers to the number of input sentences in the batch.\n",
        "* sequence_length corresponds to the maximum length of the input sentences in terms of the number of tokens or words.\n",
        "For example, if we have a batch of three input sentences with varying lengths, the input_tensor might have a shape of [3, 10], indicating a batch size of 3 and a maximum sequence length of 10 tokens.\n",
        "\n",
        "The values within the input_tensor are integer indices that represent the tokens of the input sentences. Each integer value corresponds to a specific word in the vocabulary.\n",
        "\n",
        "The input_tensor tensor is created by converting the tokenized and padded sentences into a tensor using the vocabulary. Each element of the tensor represents the index of a word in the vocabulary.\n",
        "\n",
        "During the forward pass of the Transformer model, the input tensor is passed through the embedding layer (nn.Embedding), which maps each word index to its corresponding dense word embedding of size hidden_dim. This embedding layer converts the input tensor of indices into a tensor of dense word embeddings, allowing the model to work with continuous representations of words.\n",
        "\n",
        "The input tensor is then fed into the Transformer encoder, which applies self-attention and feed-forward operations to the embeddings to capture contextual information and relationships between the words in the input sentences. The Transformer encoder produces an output tensor of the same shape as the input tensor."
      ],
      "metadata": {
        "id": "i5HgmdxcZIWH"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "aaT-WTKtOmsl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Remark\n",
        "\n",
        "In the Transformer model, the output shape after passing the input tensor through the model will have the same number of hidden dimensions as the input tensor.\n",
        "\n",
        "The output shape is determined by the configuration of the Transformer model's encoder layer and encoder. In this minimal implementation, we use the nn.TransformerEncoderLayer and nn.TransformerEncoder classes provided by PyTorch.\n",
        "\n",
        "The nn.TransformerEncoderLayer applies self-attention and feed-forward operations to the input tensor. It operates on a sequence of word embeddings and transforms it by attending to different positions within the sequence. The resulting output from the self-attention mechanism is then passed through a feed-forward neural network. This process is repeated for multiple layers specified by the num_layers parameter.\n",
        "\n",
        "The nn.TransformerEncoder combines multiple nn.TransformerEncoderLayer instances and applies them sequentially to the input tensor. This helps capture dependencies between words at different positions in the input sequence.\n",
        "\n",
        "Since the input tensor is passed through the self-attention and feed-forward operations, the output tensor retains the same sequence length and hidden dimension as the input tensor. The purpose of the Transformer model is to capture and encode meaningful representations of the input sequence while preserving its structure.\n",
        "\n",
        "Therefore, the output shape of the Transformer model will be [batch_size, sequence_length, hidden_dim], where batch_size represents the number of input sentences, sequence_length is the maximum length of the input sentences, and hidden_dim represents the hidden dimension size.\n",
        "\n",
        "In summary, the Transformer model processes the input tensor through self-attention and feed-forward operations, resulting in an output tensor with the same sequence length and hidden dimension as the input tensor."
      ],
      "metadata": {
        "id": "DgZ4K3LxYS5p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Bert Models \n",
        "\n",
        "BERT (Bidirectional Encoder Representations from Transformers) is a popular pre-trained Transformer-based model developed by Google. It is designed to capture bidirectional contextual information from input text, enabling it to understand the meaning and relationships between words in a given sentence.\n",
        "\n",
        "BERT models are trained on large amounts of text data using unsupervised learning. The training objective involves predicting missing words in a sentence, which helps the model learn the contextual representations of words.\n",
        "\n",
        "The key features of BERT models include:\n",
        "\n",
        "Transformer Architecture: BERT models are based on the Transformer architecture, which consists of multiple layers of self-attention and feed-forward neural networks. This architecture allows the model to efficiently process sequential data and capture dependencies between words.\n",
        "\n",
        "Bidirectional Context: Unlike traditional language models that process text in one direction (either left-to-right or right-to-left), BERT leverages bidirectional context. It considers both the left and right contexts of each word during training, resulting in better contextual understanding.\n",
        "\n",
        "Pre-training and Fine-tuning: BERT models are pre-trained on large-scale text corpora using unsupervised learning. After pre-training, they can be fine-tuned on specific downstream tasks such as sentiment analysis, named entity recognition, question answering, and more. Fine-tuning involves training the model on task-specific labeled data to adapt it to the specific task.\n",
        "\n",
        "Large-Scale Training Data: BERT models are trained on massive amounts of publicly available text data, such as Wikipedia articles and books. This extensive training data helps the models learn rich language representations that can be generalized to various natural language processing tasks.\n",
        "\n",
        "BERT models have achieved state-of-the-art performance on various NLP benchmarks and tasks, showcasing their effectiveness in capturing contextual information and understanding natural language. Due to their versatility, BERT models have become widely adopted in both research and industry for a wide range of NLP applications.\n",
        "\n",
        "It's worth noting that BERT is just one example of a pre-trained Transformer-based model, and there are other variants and architectures available as well, such as GPT (Generative Pre-trained Transformer) and RoBERTa (Robustly Optimized BERT Approach).\n",
        "\n",
        "# Example"
      ],
      "metadata": {
        "id": "-ghyZN5icCXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import BertTokenizer, BertForSequenceClassification\n",
        "\n",
        "# Load the pre-trained Transformer model and tokenizer\n",
        "model_name = \"bert-base-uncased\"\n",
        "tokenizer = BertTokenizer.from_pretrained(model_name)\n",
        "model = BertForSequenceClassification.from_pretrained(model_name)\n",
        "\n",
        "# Example input sentences\n",
        "sentences = [\n",
        "    \"I love this product! It's amazing.\",\n",
        "    \"This movie was terrible. I wouldn't recommend it.\",\n",
        "    \"I think I like other movies better.\",\n",
        "]\n",
        "\n",
        "# Tokenize the input sentences\n",
        "encoded_inputs = tokenizer.batch_encode_plus(\n",
        "    sentences,\n",
        "    add_special_tokens=True,\n",
        "    padding=True,\n",
        "    truncation=True,\n",
        "    return_tensors=\"pt\"\n",
        ")\n",
        "\n",
        "# Obtain the input tensors\n",
        "input_ids = encoded_inputs[\"input_ids\"]\n",
        "attention_mask = encoded_inputs[\"attention_mask\"]\n",
        "\n",
        "# Perform sentiment analysis\n",
        "outputs = model(input_ids, attention_mask=attention_mask)\n",
        "logits = outputs.logits\n",
        "\n",
        "# Get the predicted labels\n",
        "predicted_labels = logits.argmax(dim=1)\n",
        "\n",
        "# Define the label mapping\n",
        "label_mapping = {0: \"Negative\", 1: \"Positive\"}\n",
        "\n",
        "# Print the predicted labels\n",
        "for i, label_idx in enumerate(predicted_labels):\n",
        "    label = label_mapping[label_idx.item()]\n",
        "    print(f\"Sentence: {sentences[i]}\")\n",
        "    print(f\"Predicted Sentiment: {label}\")\n",
        "    print()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9StZjQ-8YU8n",
        "outputId": "c725928b-b9dd-4a97-bd4a-cf3ab4bf6cf0"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence: I love this product! It's amazing.\n",
            "Predicted Sentiment: Negative\n",
            "\n",
            "Sentence: This movie was terrible. I wouldn't recommend it.\n",
            "Predicted Sentiment: Positive\n",
            "\n",
            "Sentence: I think I like other movies better.\n",
            "Predicted Sentiment: Negative\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# IN the example above\n",
        "* We import the necessary modules: BertTokenizer for tokenization and BertForSequenceClassification for the pre-trained sentiment analysis model.\n",
        "\n",
        "* We specify the pre-trained model to load. In this case, we're using the \"bert-base-uncased\" model, which is trained on lowercased text.\n",
        "\n",
        "* We create a tokenizer using the BertTokenizer.from_pretrained method and load the pre-trained model using BertForSequenceClassification.from_pretrained.\n",
        "\n",
        "* We define a list of example input sentences.\n",
        "\n",
        "* We use the tokenizer's batch_encode_plus method to tokenize and encode the input sentences. We add special tokens, pad the sequences, and truncate if necessary. The resulting encoded inputs include input_ids and attention_mask.\n",
        "\n",
        "* We pass the input_ids and attention_mask tensors to the pre-trained model, which performs sentiment analysis and produces logits.\n",
        "\n",
        "* We obtain the predicted labels by taking the argmax of the logits along the dimension representing the classes.\n",
        "\n",
        "* We define a label mapping to map the label indices to their corresponding sentiment labels.\n",
        "\n",
        "* Finally, we print the input sentences and their predicted sentiment labels."
      ],
      "metadata": {
        "id": "pMCLiYRmbf-Z"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "BIi0X8NBaOG4"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}