DMD_Submission.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljqiCrjT5760"
      },
      "source": [
        "### Decoupled Multimodal Distilling for Emotion Recognition\n",
        "\n",
        "This model is based on https://github.com/mdswyz/DMD/ with MOSI dataset. The dataset is pretained with BERT and trained with 16 batch size and for 30 epochs until convergence. The dataset is labeled with sentimental scores, including\n",
        "highly negative, negative, weakly negative, neutral, weakly\n",
        "positive, positive, and highly positive. All of the tuning parameters are in config.json and the details explanation about the model can be found in the paper https://arxiv.org/pdf/2303.13802.pdf."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTIB1kuL0Jxc"
      },
      "source": [
        "### 1. Install Python 3.8\n",
        "Google Colab provide three versions of Python by default. The desired version can be chosen manually or typed as the command below. Pip also then needs to be installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "32uVxXBPEQ7E",
        "outputId": "947501a9-1d39-494d-8204-99009b0a4fd8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2 choices for the alternative python3 (providing /usr/bin/python3).\n",
            "\n",
            "  Selection    Path                 Priority   Status\n",
            "------------------------------------------------------------\n",
            "* 0            /usr/bin/python3.10   2         auto mode\n",
            "  1            /usr/bin/python3.10   2         manual mode\n",
            "  2            /usr/bin/python3.8    1         manual mode\n",
            "\n",
            "Press <enter> to keep the current choice[*], or type selection number: update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python3) in manual mode\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  python-pip-whl python3-setuptools python3-wheel\n",
            "Suggested packages:\n",
            "  python-setuptools-doc\n",
            "The following NEW packages will be installed:\n",
            "  python-pip-whl python3-pip python3-setuptools python3-wheel\n",
            "0 upgraded, 4 newly installed, 0 to remove and 15 not upgraded.\n",
            "Need to get 2,389 kB of archives.\n",
            "After this operation, 4,933 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1,805 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-wheel all 0.34.2-1ubuntu0.1 [23.9 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]\n",
            "Fetched 2,389 kB in 0s (16.4 MB/s)\n",
            "Selecting previously unselected package python-pip-whl.\n",
            "(Reading database ... 123105 files and directories currently installed.)\n",
            "Preparing to unpack .../python-pip-whl_20.0.2-5ubuntu1.9_all.deb ...\n",
            "Unpacking python-pip-whl (20.0.2-5ubuntu1.9) ...\n",
            "Selecting previously unselected package python3-setuptools.\n",
            "Preparing to unpack .../python3-setuptools_45.2.0-1ubuntu0.1_all.deb ...\n",
            "Unpacking python3-setuptools (45.2.0-1ubuntu0.1) ...\n",
            "Selecting previously unselected package python3-wheel.\n",
            "Preparing to unpack .../python3-wheel_0.34.2-1ubuntu0.1_all.deb ...\n",
            "Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...\n",
            "Selecting previously unselected package python3-pip.\n",
            "Preparing to unpack .../python3-pip_20.0.2-5ubuntu1.9_all.deb ...\n",
            "Unpacking python3-pip (20.0.2-5ubuntu1.9) ...\n",
            "Setting up python3-setuptools (45.2.0-1ubuntu0.1) ...\n",
            "Setting up python3-wheel (0.34.2-1ubuntu0.1) ...\n",
            "Setting up python-pip-whl (20.0.2-5ubuntu1.9) ...\n",
            "Setting up python3-pip (20.0.2-5ubuntu1.9) ...\n",
            "Processing triggers for man-db (2.9.1-1) ...\n",
            "Collecting pip\n",
            "  Downloading pip-23.2-py3-none-any.whl (2.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1 MB 8.9 MB/s \n",
            "\u001b[?25hInstalling collected packages: pip\n",
            "\u001b[33m  WARNING: The scripts pip, pip3, pip3.11 and pip3.8 are installed in '/root/.local/bin' which is not on PATH.\n",
            "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
            "Successfully installed pip-23.2\n"
          ]
        }
      ],
      "source": [
        "## List available installed python in Colab\n",
        "# !sudo update-alternatives --config python3\n",
        "!sudo update-alternatives --config python3 <<< '2'\n",
        "\n",
        "## Install pip\n",
        "!apt-get install python3-pip\n",
        "!python -m pip install --upgrade pip --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bM0DBMC11IOm"
      },
      "source": [
        "### 2. Create Virtual Environment\n",
        "Since different model will be implemented, creating virtual environment is very useful to install the requirements or the packages that are needed for each model. In addition to that, everything that is installed in Google Colab is only temporary. Saving the installed packages in the virtual environment into Google Drive will be more practical instead of installing everything everytime the runtime is restarted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NcO-41ICEkjk",
        "outputId": "5dc47a98-a337-4dc2-a55d-8498fb498267"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting virtualenv\n",
            "  Obtaining dependency information for virtualenv from https://files.pythonhosted.org/packages/c5/d5/f914b715f8b4c2ae8ca10112d389c04bed368ddd8888b70dafe740269bb5/virtualenv-20.24.0-py3-none-any.whl.metadata\n",
            "  Downloading virtualenv-20.24.0-py3-none-any.whl.metadata (4.5 kB)\n",
            "Collecting distlib<1,>=0.3.6 (from virtualenv)\n",
            "  Obtaining dependency information for distlib<1,>=0.3.6 from https://files.pythonhosted.org/packages/43/a0/9ba967fdbd55293bacfc1507f58e316f740a3b231fc00e3d86dc39bc185a/distlib-0.3.7-py2.py3-none-any.whl.metadata\n",
            "  Downloading distlib-0.3.7-py2.py3-none-any.whl.metadata (5.1 kB)\n",
            "Collecting filelock<4,>=3.12 (from virtualenv)\n",
            "  Obtaining dependency information for filelock<4,>=3.12 from https://files.pythonhosted.org/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl.metadata\n",
            "  Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)\n",
            "Collecting platformdirs<4,>=3.5.1 (from virtualenv)\n",
            "  Obtaining dependency information for platformdirs<4,>=3.5.1 from https://files.pythonhosted.org/packages/6d/a7/47b7088a28c8fe5775eb15281bf44d39facdbe4bc011a95ccb89390c2db9/platformdirs-3.9.1-py3-none-any.whl.metadata\n",
            "  Downloading platformdirs-3.9.1-py3-none-any.whl.metadata (11 kB)\n",
            "Downloading virtualenv-20.24.0-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m58.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.9/468.9 kB\u001b[0m \u001b[31m40.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading filelock-3.12.2-py3-none-any.whl (10 kB)\n",
            "Downloading platformdirs-3.9.1-py3-none-any.whl (16 kB)\n",
            "Installing collected packages: distlib, platformdirs, filelock, virtualenv\n",
            "Successfully installed distlib-0.3.7 filelock-3.12.2 platformdirs-3.9.1 virtualenv-20.24.0\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mMounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "## Install virtual environment using pip\n",
        "!pip install virtualenv\n",
        "\n",
        "## Mount Google Drive into Colab\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "## Save the venv into Google Drive\n",
        "#!virtualenv /content/drive/MyDrive/dmd_env"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9nWwyTK3czT"
      },
      "source": [
        "### 3. Install Requirements\n",
        "Always activate the venv before install the requirements or packages with pip. Installing the packages only needed to be done once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pIGYHRebEmm9"
      },
      "outputs": [],
      "source": [
        "## Install the requirements inside the venv\n",
        "\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install pynvml\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install -U scikit-learn\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install pandas\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install easydict\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install tqdm\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install transformers\n",
        "# !source /content/drive/MyDrive/dmd_env/bin/activate; pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6m1RmVt5Og2"
      },
      "source": [
        "###4. Run Python Codes\n",
        "To run each python code, activate the venv. Calling every python modules or files one by one is not needed as long as the files are in the correct folder."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "DMD model trains until the validation loss does not decrease for a specified number of epochs.\n"
      ],
      "metadata": {
        "id": "SoafvSDuyKO0"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8XTbeLv2EwT2"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "sys.path.append('/content/drive/MyDrive/DMD')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S6FxkGk2FDV7",
        "outputId": "57ec0d53-3397-4ead-b3af-19dfa9975a55"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/DMD\n"
          ]
        }
      ],
      "source": [
        "cd drive/MyDrive/DMD/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qicdL6WYE3Dk",
        "outputId": "9e4d7d7b-95e6-439c-ff52-8d136d60526c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MMSA - train samples: (1284,)\n",
            "MMSA - valid samples: (229,)\n",
            "MMSA - test samples: (686,)\n",
            "training for DMD\n",
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "100% 81/81 [00:31<00:00,  2.61it/s]\n",
            "MMSA - >> Epoch: 1 TRAIN-(dmd) [1/1/1] >> total_loss: 10.7034  Acc_2: 0.5435  F1_score: 0.4527  Acc_7: 0.1963  MAE: 1.3314 \n",
            "100% 15/15 [00:01<00:00,  8.49it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.5741  F1_score: 0.4187  Acc_7: 0.2140  MAE: 1.4206  Loss: 1.4343 \n",
            "100% 43/43 [00:04<00:00,  9.60it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.4223  F1_score: 0.2507  Acc_7: 0.1545  MAE: 1.4732  Loss: 1.4731 \n",
            "100% 81/81 [00:32<00:00,  2.49it/s]\n",
            "MMSA - >> Epoch: 2 TRAIN-(dmd) [1/2/1] >> total_loss: 9.2034  Acc_2: 0.5548  F1_score: 0.4037  Acc_7: 0.1947  MAE: 1.2839 \n",
            "100% 15/15 [00:01<00:00,  8.55it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.5833  F1_score: 0.4610  Acc_7: 0.2096  MAE: 1.3392  Loss: 1.3611 \n",
            "100% 43/43 [00:04<00:00,  9.56it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.4573  F1_score: 0.3232  Acc_7: 0.1545  MAE: 1.3681  Loss: 1.3682 \n",
            "100% 81/81 [00:32<00:00,  2.46it/s]\n",
            "MMSA - >> Epoch: 3 TRAIN-(dmd) [1/3/1] >> total_loss: 8.1434  Acc_2: 0.6515  F1_score: 0.5949  Acc_7: 0.2352  MAE: 1.1644 \n",
            "100% 15/15 [00:01<00:00,  8.26it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8426  F1_score: 0.8431  Acc_7: 0.2795  MAE: 1.1584  Loss: 1.1414 \n",
            "100% 43/43 [00:04<00:00,  9.36it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8598  F1_score: 0.8600  Acc_7: 0.2216  MAE: 1.1909  Loss: 1.1908 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 4 TRAIN-(dmd) [1/4/1] >> total_loss: 7.1846  Acc_2: 0.8562  F1_score: 0.8538  Acc_7: 0.2757  MAE: 0.9800 \n",
            "100% 15/15 [00:01<00:00,  8.34it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8333  F1_score: 0.8314  Acc_7: 0.2882  MAE: 1.0267  Loss: 1.0134 \n",
            "100% 43/43 [00:04<00:00,  9.79it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8186  F1_score: 0.8197  Acc_7: 0.2332  MAE: 1.0676  Loss: 1.0676 \n",
            "100% 81/81 [00:33<00:00,  2.45it/s]\n",
            "MMSA - >> Epoch: 5 TRAIN-(dmd) [1/5/1] >> total_loss: 6.4995  Acc_2: 0.9212  F1_score: 0.9211  Acc_7: 0.3341  MAE: 0.7619 \n",
            "100% 15/15 [00:01<00:00,  8.10it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8287  F1_score: 0.8283  Acc_7: 0.3362  MAE: 0.9115  Loss: 0.9160 \n",
            "100% 43/43 [00:04<00:00,  9.65it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8293  F1_score: 0.8299  Acc_7: 0.3892  MAE: 0.8756  Loss: 0.8755 \n",
            "100% 81/81 [00:33<00:00,  2.42it/s]\n",
            "MMSA - >> Epoch: 6 TRAIN-(dmd) [1/6/1] >> total_loss: 5.8567  Acc_2: 0.9480  F1_score: 0.9480  Acc_7: 0.5070  MAE: 0.5397 \n",
            "100% 15/15 [00:01<00:00,  7.76it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8380  F1_score: 0.8383  Acc_7: 0.3668  MAE: 0.8297  Loss: 0.8320 \n",
            "100% 43/43 [00:04<00:00,  9.46it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8384  F1_score: 0.8383  Acc_7: 0.4402  MAE: 0.7652  Loss: 0.7647 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 7 TRAIN-(dmd) [1/7/1] >> total_loss: 5.4341  Acc_2: 0.9464  F1_score: 0.9465  Acc_7: 0.5880  MAE: 0.4222 \n",
            "100% 15/15 [00:01<00:00,  7.63it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8522  Acc_7: 0.3843  MAE: 0.8272  Loss: 0.8122 \n",
            "100% 43/43 [00:04<00:00,  9.68it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8491  F1_score: 0.8487  Acc_7: 0.4402  MAE: 0.7498  Loss: 0.7493 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 8 TRAIN-(dmd) [1/8/1] >> total_loss: 5.1727  Acc_2: 0.9496  F1_score: 0.9497  Acc_7: 0.6706  MAE: 0.3708 \n",
            "100% 15/15 [00:01<00:00,  8.19it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8472  F1_score: 0.8475  Acc_7: 0.4279  MAE: 0.8102  Loss: 0.8018 \n",
            "100% 43/43 [00:04<00:00,  9.73it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8506  F1_score: 0.8505  Acc_7: 0.4402  MAE: 0.7522  Loss: 0.7524 \n",
            "100% 81/81 [00:33<00:00,  2.42it/s]\n",
            "MMSA - >> Epoch: 9 TRAIN-(dmd) [1/9/1] >> total_loss: 4.8936  Acc_2: 0.9643  F1_score: 0.9643  Acc_7: 0.7220  MAE: 0.3276 \n",
            "100% 15/15 [00:01<00:00,  7.77it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8524  Acc_7: 0.4061  MAE: 0.7947  Loss: 0.7963 \n",
            "100% 43/43 [00:04<00:00,  9.51it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8567  F1_score: 0.8551  Acc_7: 0.4490  MAE: 0.7296  Loss: 0.7293 \n",
            "100% 81/81 [00:33<00:00,  2.39it/s]\n",
            "MMSA - >> Epoch: 10 TRAIN-(dmd) [1/10/1] >> total_loss: 4.7329  Acc_2: 0.9724  F1_score: 0.9724  Acc_7: 0.7414  MAE: 0.2964 \n",
            "100% 15/15 [00:01<00:00,  7.75it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8657  F1_score: 0.8661  Acc_7: 0.4279  MAE: 0.7495  Loss: 0.7255 \n",
            "100% 43/43 [00:04<00:00,  9.53it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8567  F1_score: 0.8560  Acc_7: 0.4461  MAE: 0.7126  Loss: 0.7141 \n",
            "100% 81/81 [00:33<00:00,  2.42it/s]\n",
            "MMSA - >> Epoch: 11 TRAIN-(dmd) [1/11/1] >> total_loss: 4.5393  Acc_2: 0.9756  F1_score: 0.9756  Acc_7: 0.7305  MAE: 0.2689 \n",
            "100% 15/15 [00:01<00:00,  7.70it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8516  Acc_7: 0.4192  MAE: 0.7387  Loss: 0.7225 \n",
            "100% 43/43 [00:04<00:00,  9.39it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8521  F1_score: 0.8524  Acc_7: 0.4592  MAE: 0.7195  Loss: 0.7198 \n",
            "100% 81/81 [00:33<00:00,  2.40it/s]\n",
            "MMSA - >> Epoch: 12 TRAIN-(dmd) [1/12/1] >> total_loss: 4.4178  Acc_2: 0.9846  F1_score: 0.9846  Acc_7: 0.7368  MAE: 0.2654 \n",
            "100% 15/15 [00:01<00:00,  8.08it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8611  F1_score: 0.8613  Acc_7: 0.4454  MAE: 0.7324  Loss: 0.7564 \n",
            "100% 43/43 [00:04<00:00,  9.01it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8476  F1_score: 0.8477  Acc_7: 0.4431  MAE: 0.7226  Loss: 0.7222 \n",
            "100% 81/81 [00:32<00:00,  2.49it/s]\n",
            "MMSA - >> Epoch: 13 TRAIN-(dmd) [2/13/1] >> total_loss: 4.2583  Acc_2: 0.9821  F1_score: 0.9821  Acc_7: 0.7819  MAE: 0.2323 \n",
            "100% 15/15 [00:01<00:00,  8.05it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8522  Acc_7: 0.4148  MAE: 0.7339  Loss: 0.7161 \n",
            "100% 43/43 [00:04<00:00,  9.34it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8506  F1_score: 0.8503  Acc_7: 0.4417  MAE: 0.7071  Loss: 0.7066 \n",
            "100% 81/81 [00:33<00:00,  2.42it/s]\n",
            "MMSA - >> Epoch: 14 TRAIN-(dmd) [1/14/1] >> total_loss: 4.1758  Acc_2: 0.9846  F1_score: 0.9846  Acc_7: 0.7874  MAE: 0.2280 \n",
            "100% 15/15 [00:02<00:00,  7.38it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8566  Acc_7: 0.4279  MAE: 0.7199  Loss: 0.7186 \n",
            "100% 43/43 [00:04<00:00,  9.46it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8460  F1_score: 0.8459  Acc_7: 0.4650  MAE: 0.7222  Loss: 0.7230 \n",
            "100% 81/81 [00:33<00:00,  2.45it/s]\n",
            "MMSA - >> Epoch: 15 TRAIN-(dmd) [2/15/1] >> total_loss: 4.0664  Acc_2: 0.9870  F1_score: 0.9870  Acc_7: 0.7679  MAE: 0.2156 \n",
            "100% 15/15 [00:01<00:00,  7.80it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8569  Acc_7: 0.4236  MAE: 0.7234  Loss: 0.7183 \n",
            "100% 43/43 [00:04<00:00,  9.39it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8537  F1_score: 0.8532  Acc_7: 0.4534  MAE: 0.7032  Loss: 0.7027 \n",
            "100% 81/81 [00:32<00:00,  2.46it/s]\n",
            "MMSA - >> Epoch: 16 TRAIN-(dmd) [3/16/1] >> total_loss: 4.0009  Acc_2: 0.9821  F1_score: 0.9821  Acc_7: 0.7913  MAE: 0.1996 \n",
            "100% 15/15 [00:01<00:00,  7.98it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8472  F1_score: 0.8475  Acc_7: 0.4279  MAE: 0.7206  Loss: 0.7194 \n",
            "100% 43/43 [00:04<00:00,  9.00it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8537  F1_score: 0.8536  Acc_7: 0.4490  MAE: 0.7086  Loss: 0.7083 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 17 TRAIN-(dmd) [4/17/1] >> total_loss: 3.8896  Acc_2: 0.9878  F1_score: 0.9878  Acc_7: 0.7913  MAE: 0.1932 \n",
            "100% 15/15 [00:01<00:00,  7.91it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8516  Acc_7: 0.4541  MAE: 0.7008  Loss: 0.7044 \n",
            "100% 43/43 [00:04<00:00,  9.45it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8445  F1_score: 0.8449  Acc_7: 0.4563  MAE: 0.7238  Loss: 0.7229 \n",
            "100% 81/81 [00:33<00:00,  2.40it/s]\n",
            "MMSA - >> Epoch: 18 TRAIN-(dmd) [1/18/1] >> total_loss: 3.8332  Acc_2: 0.9821  F1_score: 0.9821  Acc_7: 0.8217  MAE: 0.1848 \n",
            "100% 15/15 [00:02<00:00,  7.37it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8472  F1_score: 0.8463  Acc_7: 0.4323  MAE: 0.7025  Loss: 0.6812 \n",
            "100% 43/43 [00:04<00:00,  9.38it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8430  F1_score: 0.8436  Acc_7: 0.4592  MAE: 0.7331  Loss: 0.7337 \n",
            "100% 81/81 [00:33<00:00,  2.40it/s]\n",
            "MMSA - >> Epoch: 19 TRAIN-(dmd) [1/19/1] >> total_loss: 3.8101  Acc_2: 0.9829  F1_score: 0.9829  Acc_7: 0.8170  MAE: 0.1880 \n",
            "100% 15/15 [00:01<00:00,  7.74it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8568  Acc_7: 0.4672  MAE: 0.7108  Loss: 0.7016 \n",
            "100% 43/43 [00:04<00:00,  9.11it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8476  F1_score: 0.8473  Acc_7: 0.4708  MAE: 0.7054  Loss: 0.7052 \n",
            "100% 81/81 [00:33<00:00,  2.45it/s]\n",
            "MMSA - >> Epoch: 20 TRAIN-(dmd) [2/20/1] >> total_loss: 3.7633  Acc_2: 0.9870  F1_score: 0.9870  Acc_7: 0.7967  MAE: 0.1802 \n",
            "100% 15/15 [00:01<00:00,  7.83it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8569  Acc_7: 0.4498  MAE: 0.7219  Loss: 0.7335 \n",
            "100% 43/43 [00:04<00:00,  9.44it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8537  F1_score: 0.8528  Acc_7: 0.4650  MAE: 0.6950  Loss: 0.6959 \n",
            "100% 81/81 [00:32<00:00,  2.46it/s]\n",
            "MMSA - >> Epoch: 21 TRAIN-(dmd) [3/21/1] >> total_loss: 3.7236  Acc_2: 0.9903  F1_score: 0.9903  Acc_7: 0.8115  MAE: 0.1761 \n",
            "100% 15/15 [00:02<00:00,  7.01it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8611  F1_score: 0.8615  Acc_7: 0.4236  MAE: 0.7221  Loss: 0.7158 \n",
            "100% 43/43 [00:04<00:00,  9.34it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8598  F1_score: 0.8588  Acc_7: 0.4446  MAE: 0.6966  Loss: 0.6963 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 22 TRAIN-(dmd) [4/22/1] >> total_loss: 3.729  Acc_2: 0.9813  F1_score: 0.9813  Acc_7: 0.8139  MAE: 0.1792 \n",
            "100% 15/15 [00:01<00:00,  7.86it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8520  Acc_7: 0.4454  MAE: 0.7269  Loss: 0.7242 \n",
            "100% 43/43 [00:04<00:00,  9.14it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8567  F1_score: 0.8563  Acc_7: 0.4679  MAE: 0.7070  Loss: 0.7073 \n",
            "100% 81/81 [00:33<00:00,  2.43it/s]\n",
            "MMSA - >> Epoch: 23 TRAIN-(dmd) [5/23/1] >> total_loss: 3.6342  Acc_2: 0.9862  F1_score: 0.9862  Acc_7: 0.8318  MAE: 0.1659 \n",
            "100% 15/15 [00:01<00:00,  7.76it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8611  F1_score: 0.8613  Acc_7: 0.4498  MAE: 0.7148  Loss: 0.7203 \n",
            "100% 43/43 [00:04<00:00,  9.23it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8506  F1_score: 0.8505  Acc_7: 0.4606  MAE: 0.7032  Loss: 0.7033 \n",
            "100% 81/81 [00:33<00:00,  2.43it/s]\n",
            "MMSA - >> Epoch: 24 TRAIN-(dmd) [6/24/1] >> total_loss: 3.545  Acc_2: 0.9959  F1_score: 0.9959  Acc_7: 0.8310  MAE: 0.1535 \n",
            "100% 15/15 [00:01<00:00,  7.50it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8516  Acc_7: 0.4629  MAE: 0.7106  Loss: 0.6953 \n",
            "100% 43/43 [00:04<00:00,  9.38it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8476  F1_score: 0.8478  Acc_7: 0.4679  MAE: 0.7084  Loss: 0.7087 \n",
            "Epoch    24: reducing learning rate of group 0 to 5.0000e-05.\n",
            "100% 81/81 [00:33<00:00,  2.45it/s]\n",
            "MMSA - >> Epoch: 25 TRAIN-(dmd) [7/25/1] >> total_loss: 3.4996  Acc_2: 0.9911  F1_score: 0.9911  Acc_7: 0.8505  MAE: 0.1538 \n",
            "100% 15/15 [00:01<00:00,  7.69it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8519  F1_score: 0.8519  Acc_7: 0.4279  MAE: 0.7117  Loss: 0.6933 \n",
            "100% 43/43 [00:04<00:00,  9.05it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8460  F1_score: 0.8460  Acc_7: 0.4621  MAE: 0.7015  Loss: 0.7007 \n",
            "100% 81/81 [00:33<00:00,  2.42it/s]\n",
            "MMSA - >> Epoch: 26 TRAIN-(dmd) [8/26/1] >> total_loss: 3.4419  Acc_2: 0.9919  F1_score: 0.9919  Acc_7: 0.8575  MAE: 0.1432 \n",
            "100% 15/15 [00:01<00:00,  7.70it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8568  Acc_7: 0.4454  MAE: 0.7132  Loss: 0.7180 \n",
            "100% 43/43 [00:04<00:00,  9.40it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8506  F1_score: 0.8504  Acc_7: 0.4650  MAE: 0.7113  Loss: 0.7106 \n",
            "100% 81/81 [00:33<00:00,  2.43it/s]\n",
            "MMSA - >> Epoch: 27 TRAIN-(dmd) [9/27/1] >> total_loss: 3.3679  Acc_2: 0.9943  F1_score: 0.9943  Acc_7: 0.8653  MAE: 0.1309 \n",
            "100% 15/15 [00:02<00:00,  7.34it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8561  Acc_7: 0.4410  MAE: 0.7047  Loss: 0.7054 \n",
            "100% 43/43 [00:04<00:00,  9.12it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8460  F1_score: 0.8461  Acc_7: 0.4679  MAE: 0.7125  Loss: 0.7118 \n",
            "100% 81/81 [00:33<00:00,  2.44it/s]\n",
            "MMSA - >> Epoch: 28 TRAIN-(dmd) [10/28/1] >> total_loss: 3.3296  Acc_2: 0.9927  F1_score: 0.9927  Acc_7: 0.8769  MAE: 0.1293 \n",
            "100% 15/15 [00:01<00:00,  7.64it/s]\n",
            "MMSA - VAL-(dmd) >>  Acc_2: 0.8565  F1_score: 0.8568  Acc_7: 0.4585  MAE: 0.7138  Loss: 0.7134 \n",
            "100% 43/43 [00:04<00:00,  9.14it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8506  F1_score: 0.8500  Acc_7: 0.4621  MAE: 0.6981  Loss: 0.6987 \n",
            "100% 43/43 [00:04<00:00,  9.62it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8445  F1_score: 0.8452  Acc_7: 0.4563  MAE: 0.7332  Loss: 0.7342 \n",
            "MMSA - Results saved to result/normal/mosi.csv.\n"
          ]
        }
      ],
      "source": [
        "# Train the model\n",
        "# Set the necessary parameters in the ./config/config.json\n",
        "\n",
        "!source /content/drive/MyDrive/dmd_env/bin/activate; python train.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NMMCJg_VE6q6",
        "outputId": "cf80d86e-e9f0-431a-d612-88c068880bea"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MMSA - train samples: (1284,)\n",
            "MMSA - valid samples: (229,)\n",
            "MMSA - test samples: (686,)\n",
            "testing phase for DMD\n",
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "100% 43/43 [00:04<00:00,  8.78it/s]\n",
            "MMSA - TEST-(dmd) >>  Acc_2: 0.8430  F1_score: 0.8437  Acc_7: 0.4548  MAE: 0.7334  Loss: 0.7334 \n",
            "[Press Any Key to start another run]\n"
          ]
        }
      ],
      "source": [
        "# Test the model\n",
        "# Set the path of trained model in run.py (line 174)\n",
        "\n",
        "!source /content/drive/MyDrive/dmd_env/bin/activate; python test.py"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}