en_pa_MainNotebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "pa_bt_ilmulti sample run notebook-simple fine tuning",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ASBQtS_PpWMp"
      },
      "source": [
        "# Cloning and setting up\n",
        "\n",
        "Clone the repository, and change into directory. To ensure this notebook remains functional, this notebook will checkout a commit [`ec27a2c`](https://github.com/jerinphilip/ilmulti/commit/ec27a2c19ecf06991fea55a8a1d34617a07c1d87)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w6HTXcazjVXD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "5261511b-bb51-4538-ae0a-f706130184fd"
      },
      "source": [
        "!rm -rf ilmulti\n",
        "!git clone https://github.com/jerinphilip/ilmulti\n",
        "% cd ilmulti/\n",
        "!git checkout ec27a2c # future proofing with a known working commit;"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'ilmulti'...\n",
            "remote: Enumerating objects: 278, done.\u001b[K\n",
            "remote: Counting objects: 100% (278/278), done.\u001b[K\n",
            "remote: Compressing objects: 100% (203/203), done.\u001b[K\n",
            "remote: Total 950 (delta 135), reused 195 (delta 73), pack-reused 672\u001b[K\n",
            "Receiving objects: 100% (950/950), 5.49 MiB | 8.67 MiB/s, done.\n",
            "Resolving deltas: 100% (535/535), done.\n",
            "/content/ilmulti\n",
            "Note: checking out 'ec27a2c'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by performing another checkout.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -b with the checkout command again. Example:\n",
            "\n",
            "  git checkout -b <new-branch-name>\n",
            "\n",
            "HEAD is now at ec27a2c Reducing default model downloads\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXBs4h66pah9"
      },
      "source": [
        "# Changes to enable translation\n",
        "\n",
        "We will change the requirements.txt to uncomment the components that will enable translation. These are:\n",
        "\n",
        "* [fairseq-ilmt@lrec-2020](https://github.com/jerinphilip/fairseq-ilmt/tree/lrec-2020)\n",
        "* torch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GGZ0W2DYpBMh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "374ec3f7-0d19-4930-be55-a61265bc12c4"
      },
      "source": [
        "new_requirements = \"\"\"\n",
        "langid\n",
        "sentencepiece\n",
        "nltk\n",
        "\n",
        "# Optional, tokenizers, work without these as well.\n",
        "git+https://github.com/jerinphilip/fairseq-ilmt@lrec-2020\n",
        "torch==1.1.0\n",
        "\"\"\"\n",
        "\n",
        "with open(\"requirements.txt\", 'w+') as fp:\n",
        "  fp.write(new_requirements)\n",
        "\n",
        "!cat requirements.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "langid\n",
            "sentencepiece\n",
            "nltk\n",
            "\n",
            "# Optional, tokenizers, work without these as well.\n",
            "git+https://github.com/jerinphilip/fairseq-ilmt@lrec-2020\n",
            "torch==1.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Aae7WEzptaZ"
      },
      "source": [
        "# Install prerequisites and ilmulti\n",
        "\n",
        "With the modified requirements.txt, now run pip install to setup the environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P7EqL0wIpkJ0"
      },
      "source": [
        "%%capture\n",
        "!python3 -m pip install -r requirements.txt\n",
        "!python3 setup.py install"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwsiE2kAqR0G"
      },
      "source": [
        "# Download the pretrained models and setup these in $HOME.\n",
        "\n",
        "The following script is a convenience script to setup the models in `$HOME/.ilmulti` directory. The script copies the pretrained models and the respective fairseq-dictionaries to predefined locations here.\n",
        "\n",
        "This version downloads only M2M-1 and M2EN-3 from [Revisiting Low Resource Status of Indian Languages in Machine Translation](https://arxiv.org/abs/2008.04860)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGq7-eJQwpX1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 547
        },
        "outputId": "b8092daa-b585-4d58-f84d-4b2b55a4ae87"
      },
      "source": [
        "!bash scripts/download-and-setup-models.sh"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+ SEVEN_MODELS=()\n",
            "+ ELEVEN_MODELS=(\"mm-all-iter1\")\n",
            "+ MODELS=(\"${SEVEN_MODELS[@]}\" \"${ELEVEN_MODELS[@]}\")\n",
            "+ SAVE_DIR=/root/.ilmulti\n",
            "+ BASE_URL=http://preon.iiit.ac.in/~jerin/resources/models\n",
            "+ mkdir -p /root/.ilmulti/\n",
            "+ echo 'Downloading models'\n",
            "Downloading models\n",
            "+ for MODEL in ${MODELS[@]}\n",
            "+ MODEL_DIR=/root/.ilmulti/mm-all-iter1\n",
            "+ mkdir -p /root/.ilmulti/mm-all-iter1\n",
            "+ wget --continue http://preon.iiit.ac.in/~jerin/resources/models/mm-all-iter1 -O /root/.ilmulti/mm-all-iter1/checkpoint_last.pt\n",
            "--2020-10-22 08:36:34--  http://preon.iiit.ac.in/~jerin/resources/models/mm-all-iter1\n",
            "Resolving preon.iiit.ac.in (preon.iiit.ac.in)... 196.12.53.50\n",
            "Connecting to preon.iiit.ac.in (preon.iiit.ac.in)|196.12.53.50|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 781024711 (745M)\n",
            "Saving to: ‘/root/.ilmulti/mm-all-iter1/checkpoint_last.pt’\n",
            "\n",
            "/root/.ilmulti/mm-a 100%[===================>] 744.84M  1.76MB/s    in 6m 45s  \n",
            "\n",
            "2020-10-22 08:43:20 (1.84 MB/s) - ‘/root/.ilmulti/mm-all-iter1/checkpoint_last.pt’ saved [781024711/781024711]\n",
            "\n",
            "+ for MODEL in ${ELEVEN_MODELS[@]}\n",
            "+ MODEL_DIR=/root/.ilmulti/mm-all-iter1\n",
            "+ mkdir -p /root/.ilmulti/mm-all-iter1\n",
            "++ dirname scripts/download-and-setup-models.sh\n",
            "+ SRC_DIR=scripts/../\n",
            "+ cp scripts/..//resources/fairseq-dictionaries/ilmulti-v1.dict.txt /root/.ilmulti/mm-all-iter1/dict.src.txt\n",
            "+ cp scripts/..//resources/fairseq-dictionaries/ilmulti-v1.dict.txt /root/.ilmulti/mm-all-iter1/dict.tgt.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2LREkRbcSNy9"
      },
      "source": [
        "# Training begins\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9PO36IhXY9gE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "05e45ecd-6d61-423b-902d-ca9408d690f6"
      },
      "source": [
        "# from google.colab import drive\n",
        "# drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l4o0o_RR3KQu"
      },
      "source": [
        "!bash fairseq-ilmt/cmd.sh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mu_7HydHvu0R"
      },
      "source": [
        "# Sentence-wise BLEU after cyclic backtranslation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "okNtCou9Ee4B"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "backtranslated_sentences = open(\"\", encoding = 'utf-8')\n",
        "hyp_list = []\n",
        "for position, line in enumerate(backtranslated_sentences):\n",
        "  hyp_list.append(line)\n",
        "\n",
        "original_sentences = open(\"\", encoding = 'utf-8')\n",
        "ref_list = []\n",
        "# lines_to_read = [0, 2]\n",
        "for position, line in enumerate(b_file):\n",
        "  ref_list.append(line)\n",
        "\n",
        "bleu_list = []\n",
        "for i in range(0,len(hyp_list)):\n",
        "  hyp = hyp_list[i]\n",
        "  ref = ref_list[i]\n",
        "  hyp_tokens = word_tokenize(hyp)\n",
        "  ref_tokens = word_tokenize(ref)\n",
        "  BLEUscore = nltk.translate.bleu_score.sentence_bleu([ref_tokens], hyp_tokens)\n",
        "  bleu_list.append(BLEUscore)\n",
        "\n",
        "translated_sentences = open(\"\", encoding = 'utf-8')\n",
        "translated_list = []\n",
        "for position, line in enumerate(translated_sentences):\n",
        "  translated_list.append(line)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AoQWRd31vzhw"
      },
      "source": [
        "pairWise_BLEU = pd.DataFrame(list(zip(ref_list, hyp_list,bleu_list)),columns=['eng', 'punjabi', 'bleu'])\n",
        "pairWise_BLEU.drop_duplicates(subset='eng',keep='first',inplace=True)\n",
        "pairWise_BLEU.drop_duplicates(subset='hindi',keep='first',inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V63qe7ADeC1W"
      },
      "source": [
        "# Deciling the BLEU Scores\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zCaO_-lwEEV0"
      },
      "source": [
        "path = \"\"\n",
        "for i in np.linspace(0.5,1,5,endpoint=False):\n",
        "  print (i)\n",
        "  temp = en_hi_bt[en_hi_bt['bleu'].apply(lambda x:float(x))>i].copy()\n",
        "  print (temp.shape)\n",
        "  eng_temp = list(temp['eng'])\n",
        "  hindi_temp = list(temp['hindi'])\n",
        "  with open(path+\"train_\"+str(i)+\".en\", 'w') as f:\n",
        "    for item in eng_temp:\n",
        "        f.write(\"%s\" % item)\n",
        "  with open(path+\"train_\"+str(i)+\".hi\", 'w') as f:\n",
        "    for item in hindi_temp:\n",
        "        f.write(\"%s\" % item)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GoQvGR8mftFj"
      },
      "source": [
        "# ========================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WY0AE_Jqckha"
      },
      "source": [
        "## Demo Inference of the original Multilingual NMT model "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnPwSDO4cmSq"
      },
      "source": [
        "from ilmulti.translator import from_pretrained\n",
        "\n",
        "translator = from_pretrained(tag='mm-all-iter1')\n",
        "\n",
        "samples = [\n",
        "    (\"The quick brown fox jumps over the lazy dog.\", 'en', 'hi'),\n",
        "    (\"An apple a day keeps the doctor away. He is going.\", 'en', \"hi\"),\n",
        "    (\"This document is being produced at the behest of the perpetrator\", 'en', \"te\"),\n",
        "    (\"वह जा रहा है।\", \"hi\", \"ml\")\n",
        "\n",
        "]\n",
        "\n",
        "for idb, (sample, src_lang, tgt_lang) in enumerate(samples, 1):\n",
        "  translation = translator(sample, tgt_lang=tgt_lang, src_lang=src_lang)\n",
        "  print('---', idb)\n",
        "  for idx, segment in enumerate(translation, 1):\n",
        "    print(idx, '>', segment['src'])\n",
        "    print(idx, '<', segment['tgt'])\n",
        "  print('---')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}