MFR RouterTrain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **BUD Ecosystem Inc. | NLI Research | BUD *Muffler*-$\\alpha$ | Shreyan Chaubey**\n",
    "\n",
    "This notebook is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).\n",
    "See the LICENSE file in the GitHub repository for details. For the full terms, read the [AGPL license text](https://www.gnu.org/licenses/agpl-3.0.en.html).\n",
    "---\n",
    "\n",
    "### **BUD Muffler-$\\alpha$: A Binary Language Model Router | Author: Shreyan C**\n",
    "\n",
    "The notebook has been developed as part of a research initiative at BUD to survey the capabilities of the current SOTA techniques in the area of natural language inference optimization, more specficially, language model routing. In this notebook, we present a faithful reproduction of a SOTA methodology presented in the 2024 paper '**RouteLLM: Learning to Route LLMs with Preference Data**' ([Ong et. al.](https://arxiv.org/pdf/2406.18665)). This notebook covers everything from data preparation to router benchmarking, and provides an in-depth look at how the Muffler-$\\alpha$ router functions, including its model routing logic, training approach, and cost-adjustment mechanisms. It can be useful for researchers or developers seeking to optimize language model usage in cost-sensitive environments or for those interested in implementing cost efficient routing mechanisms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Recipe - Building the Dataset\n",
    "\n",
    "Before defining the architecture for the Language Model (LM) router, it is essential to take foundational steps to ensure effective model convergence. A critical part of this process involves creating a well-structured, supervised training dataset. Such a dataset might have the following structure:\n",
    "\n",
    "```\n",
    "id [INT] | model_a [STR] | model_b [STR] | prompt [STR] | winner_model_a [0/1] | winner_model_b [0/1] | tie_ab [0/1]\n",
    "```\n",
    "\n",
    "Each row represents a comparison between two models (`model_a` and `model_b`) based on their responses to a specific `prompt`. The outcomes (`winner_model_a`, `winner_model_b`, or `tie_ab`) indicate which model performed better or if there was a tie.\n",
    "\n",
    "#### Possible Augmentations\n",
    "\n",
    "##### 1. **Benchmark Augmented Datasets**\n",
    "   - These datasets are derived by evaluating models across well-established benchmarks. Benchmarks typically focus on specific domains of reasoning.\n",
    "   - By comparing the performance of a stronger model $M_{strong}$ and a weaker model $M_{weak}$ on these benchmarks, metrics such as *accuracy* or *exact match* can be collected. These metrics determine which model performs better for a given prompt. \n",
    "   - Such datasets are often referred to as *Golden Labelled Datasets* because they provide a clear indication of model aptitude within specific domains.\n",
    "\n",
    "##### 2. **Human Augmented Datasets**\n",
    "   - Publicly available datasets that capture human preferences for language model responses can also be used. Examples include:\n",
    "     - **`lmarena-ai/arena-human-preference-55k`**\n",
    "     - **`lmarena-ai/PPE-Human-Preference-V1`**\n",
    "   - These datasets reflect the nuanced preferences humans may have for certain types of responses. However, a potential downside is that they may introduce inherent biases into the training data. These biases can affect the router's performance in unpredictable ways. Moreover, obtaining a sufficient amount of datapoints in such augmentations is a costly & time consuming business.\n",
    "\n",
    "##### 3. **LLM Judge Augmented Datasets**\n",
    "   - To reduce reliance on human-annotated preferences and address potential human biases, a *Language Model Judge* (LLM Judge) can be employed. This judge compares responses from an LLM and an SLM (a smaller language model) and assigns a label to the better-performing model.\n",
    "   - The resulting datasets, known as *LLM Preference Datasets*, aim to offer a more balanced and scalable alternative to human-augmented datasets.\n",
    "\n",
    "**Each dataset type comes with strengths and limitations:**\n",
    "- **Benchmark Augmented Datasets** ensure strong domain coverage but may lack the richness of human preferences.\n",
    "- **Human Preference Datasets** capture real-world preferences but risk introducing subjective biases.\n",
    "- **LLM Preference Datasets** strike a balance by leveraging the capabilities of an LLM as a neutral arbiter, though the quality of the \"judge\" becomes critical.\n",
    "\n",
    "We thus begin by importing relevant python libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 64832,
     "status": "ok",
     "timestamp": 1731141145833,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "k2SYjsjrnMHJ",
    "outputId": "7eeebc7e-8349-4094-a079-84ea03f0b2ed"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import datasets\n",
    "from tqdm import tqdm, trange\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from torch import optim\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from datasets import load_dataset\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, roc_auc_score, roc_curve, auc\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sentence_transformers import SentenceTransformer\n",
    "import safetensors.torch as st\n",
    "\n",
    "# ----------------- #\n",
    "proj_dir = \".\" # Replace with '.' if running locally.\n",
    "custom_ds_dir = \".\"\n",
    "# ----------------- #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the various kinds of preference datasets that can be used:\n",
    "\n",
    "1. **LMSYS Arena Human Preference 55K (LMSYS)** [Human Preference]:\n",
    "    -  This dataset contains human preferences collected from the LMSYS Arena. It includes comparisons across 64 language models based on human judgments.\n",
    "    - This dataset is useful for training the router to understand *human preferences* and make routing decisions accordingly. However, this dataset is not suitable for model deployment because of *i) A large number of target models 2) Most target models are obsolete. 3) Human preferences are prone to a number of biases.*\n",
    "\n",
    "2. **GPT-4 Judge Battles (RouteLLM)** [LLM Preference]:\n",
    "    - This dataset includes comparisons where GPT-4 acts as a judge to determine the better response between itself and Mixtral 8x7B Mixture of Experts Language Model.\n",
    "    - Helps in training the router to align with GPT-4's judgment when obtaining human preference labels is a costly business.\n",
    "    - Prompts are synthetically generated.\n",
    "\n",
    "3. **Arena Human Preference + Judge Battles** [Human + LLM Preference]:\n",
    "    - A combination of the LMSYS Arena Human Preference 55K and GPT-4 Judge Battles datasets.\n",
    "\n",
    "4. **BUD Complexity Corpus (BUD Ecosystems Inc.)** [Benchmark Augmented/Golden Labelled]:\n",
    "    - This dataset focuses on complex queries and includes comparisons between models like Gemma-2-27b-it and Llama 3.2 1B.\n",
    "    - Specifically useful for training the router to handle complex queries and make decisions based on the complexity of the task.\n",
    "    \n",
    "5. **Custom Dataset**:\n",
    "    - A custom dataset created from various sources and processed into a parquet format.\n",
    "    - Allows for flexibility in training the router with specific preferences and scenarios tailored to the user's needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 536,
     "referenced_widgets": [
      "d5eb5aa17bd24b1fb71a952382280b38",
      "22784f5721524de9839ca5b6f197f56f",
      "5f13d45684054d2f9e76e7860a428ccb",
      "8317152662a24eed95910639362c15be",
      "84e7a06d41054098a5aa91669e982af0",
      "d74c0acdb9a542649c646b817fdec9f6",
      "944c71c7614d49709ece2c08f63a7bdb",
      "086df64dbc1e4c61af9c18504a9e946c",
      "03f298f8ab4e4453b3367f8fd3fa5025",
      "c2a4ecebcb6540de991fd60a24745750",
      "5813308d828347af8ef8c397165d2870",
      "ebc9c81e29f74b7f906c9d2d77339be3",
      "ba15b41674964896bd8b25d4b0b2f91d",
      "1b36afaa063840dda602e59e7566f0f0",
      "9ad16a62d8214b26a4f4754777bf2ea4",
      "8358c52188f8457b8151db6ac5c5482a",
      "98d06452c1c94e13b069510cdeef1752",
      "d9b4981f866147e6bb333392e0840cbf",
      "b5331fd4b4cb417fb2f6cff7f904aff6",
      "b0a595f63c9f49d59e23d325818993ba",
      "b18c9d61a8134f1783638c5c1fe901d0",
      "c0308f41ca194c9b9fead599d71bb245",
      "4dd260814cd14dabafc87da41f391ecf",
      "2a82d3857bab4113ba7ee32016c63791",
      "e3c9eb62e84c47ec8e57577afb9678ab",
      "0190da56d56c4bffb69d81f88cb15f62",
      "96d8650ef9a24a168306998473ea23a4",
      "5594ce7d34dc4955a9887be820f3bced",
      "b49d4825ba60437ca375b3bf869ee0c4",
      "4bdcd5c15eeb4520a5b85db5f3461106",
      "9f0c3e71290743a0af51d2ea2c1bcf07",
      "2b9fe30db46c4b4aa6eb104c686b6a6b",
      "27201685327c46839ba700de1d6d1363"
     ]
    },
    "executionInfo": {
     "elapsed": 23000,
     "status": "ok",
     "timestamp": 1731141168828,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "2Os9Op7TnMHL",
    "outputId": "6682e082-4a46-4763-dd6d-561661983d9d"
   },
   "outputs": [],
   "source": [
    "def decide_dataset(choice):\n",
    "    if choice == 0:\n",
    "        ds_name = \"lmsys/lmsys-arena-human-preference-55k\"\n",
    "        ds = load_dataset(\"lmsys/lmsys-arena-human-preference-55k\")[\"train\"].to_pandas()\n",
    "        ds = ds[ds['model_a'] == 'gpt-4-1106-preview'] and ds[ds['model_b'] == 'mixtral-8x7b-instruct-v0.1']\n",
    "    elif choice == 1:\n",
    "        ds_name = \"routellm/gpt4_judge_battles\"\n",
    "        ds = load_dataset(\"routellm/gpt4_judge_battles\")[\"train\"].to_pandas()\n",
    "    elif choice == 2:\n",
    "        ds_name = \"lmsys/lmsys-arena-human-preference-55k and routellm/gpt4_judge_battles\"\n",
    "        ds1 = load_dataset(\"lmsys/lmsys-arena-human-preference-55k\")[\"train\"].to_pandas()\n",
    "        ds2 = load_dataset(\"routellm/gpt4_judge_battles\")[\"train\"].to_pandas()\n",
    "        ds = pd.concat([ds1, ds2], ignore_index=True)\n",
    "        ds = ds[ds['model_a'] == 'gpt-4-1106-preview'] and ds[ds['model_b'] == 'mixtral-8x7b-instruct-v0.1']\n",
    "    elif choice == 3:\n",
    "        ds_name = \"own-dataset\"\n",
    "        ds = pd.read_csv(f'{custom_ds_dir}/combined_dataset.csv')\n",
    "    return ds\n",
    "\n",
    "ds_name = None\n",
    "choice = 4  # 0: lmsys-arena-human-preference-55k, 1: gpt4_judge_battles, 2: both (1+2), 3: custom dataset\n",
    "ds = decide_dataset(choice)\n",
    "print('Shape of the dataset after removing ties:', ds.shape)\n",
    "ds.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1CS0bvjpnMHL"
   },
   "source": [
    "We proceed with a 95% train 5% validation split ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 677
    },
    "executionInfo": {
     "elapsed": 13768,
     "status": "ok",
     "timestamp": 1731141202362,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "7nLDCSemnMHM",
    "outputId": "5a51938e-316a-4dee-ae7f-3b97f16e7435"
   },
   "outputs": [],
   "source": [
    "split_ratio = 0.95\n",
    "balanced_flag = False\n",
    "train_ds, test_ds = train_test_split(ds, test_size=1-split_ratio, shuffle=False, random_state=None, stratify=None)\n",
    "print(f\"Shape of the train dataset: {train_ds.shape}\\nShape of the test dataset: {test_ds.shape}\")\n",
    "\n",
    "# Dataset splits can be accessed in the '~/output/dataset/raw/' directory.\n",
    "os.makedirs(f\"{proj_dir}/output/dataset/raw/train/\", exist_ok=True)\n",
    "train_ds.to_parquet(f\"{proj_dir}/output/dataset/raw/train/train_ds.parquet\", index=True) # train_ds\n",
    "\n",
    "os.makedirs(f\"{proj_dir}/output/dataset/raw/test/\", exist_ok=True)\n",
    "test_ds.to_parquet(f\"{proj_dir}/output/dataset/raw/test/test_ds.parquet\", index=True) # test_ds\n",
    "\n",
    "print(f\"\\nTrain dataset saved at '~/output/dataset/raw/train/train_ds.parquet'\")\n",
    "display(train_ds.head(5))\n",
    "print(\"\\nTest Dataset saved at '~/output/dataset/raw/test/test_ds.parquet'\")\n",
    "display(test_ds.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QeWp_U5anMHN"
   },
   "source": [
    "Encodings happen as follows:\n",
    "- Model A - Integer ID (Stored in MODEL_ID)\n",
    "- Model B - Integer ID (Stored in MODEL_ID)\n",
    "- Prompt - Case Index\n",
    "- Winner - ('model_a'/'model_b')\n",
    "\n",
    "Tie positives (True, True) are encoded as 'tie_positive' whereas tie negatives (False, False) are encoded as 'tie_negative'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 449
    },
    "executionInfo": {
     "elapsed": 416,
     "status": "ok",
     "timestamp": 1731141252749,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "5SVF1f3JnMHN",
    "outputId": "ffac6b3d-c0ac-45a3-be57-4a4b7762153f"
   },
   "outputs": [],
   "source": [
    "model_encoder = LabelEncoder()\n",
    "model_encoder.fit(pd.concat([train_ds['model_a'], train_ds['model_b']]).unique())\n",
    "\n",
    "MODEL_IDS = {\n",
    "    model: model_encoder.transform([model])[0] for model in model_encoder.classes_\n",
    "}\n",
    "\n",
    "print(f\"Identified models : {MODEL_IDS}\")\n",
    "\n",
    "train_ds = train_ds.reset_index(drop=True)\n",
    "test_ds = test_ds.reset_index(drop=True)\n",
    "\n",
    "\n",
    "train_ds_enc = pd.DataFrame()\n",
    "\n",
    "train_ds_enc['model_a_idx'] = train_ds['model_a'].map(MODEL_IDS)\n",
    "train_ds_enc['model_b_idx'] = train_ds['model_b'].map(MODEL_IDS)\n",
    "train_ds_enc['prompt_idx'] = train_ds.index\n",
    "\n",
    "train_ds_enc['winner'] = np.where(\n",
    "    (train_ds['winner_model_a'] == 1) & (train_ds['winner_model_b'] == 0),\n",
    "    \"model_a\",\n",
    "    np.where(\n",
    "        (train_ds['winner_model_a'] == 0) & (train_ds['winner_model_b'] == 1),\n",
    "        \"model_b\",\n",
    "        np.where(\n",
    "            (train_ds['winner_model_a'] == 1) & (train_ds['winner_model_b'] == 1),  # Both win\n",
    "            #\"model_b\", \"model_a\" # Removing tie breaking mechanism for now.\n",
    "            \"tie_positive\", \"tie_negative\" # Due to the lack of tie-positive cases in GPT4-Judge-Battles ds, former label won't be observed)\n",
    "        )\n",
    "    )\n",
    ")\n",
    "\n",
    "display(train_ds_enc.head(10))\n",
    "\n",
    "test_ds_enc = pd.DataFrame()\n",
    "\n",
    "test_ds_enc['model_a_idx'] = test_ds['model_a'].map(MODEL_IDS)\n",
    "test_ds_enc['model_b_idx'] = test_ds['model_b'].map(MODEL_IDS)\n",
    "test_ds_enc['prompt_idx'] = test_ds.index\n",
    "\n",
    "test_ds_enc['winner'] = np.where(\n",
    "    (test_ds['winner_model_a'] == 1) & (test_ds['winner_model_b'] == 0),\n",
    "    \"model_a\",\n",
    "    np.where(\n",
    "        (test_ds['winner_model_a'] == 0) & (test_ds['winner_model_b'] == 1),\n",
    "        \"model_b\",\n",
    "        np.where(\n",
    "            (test_ds['winner_model_a'] == 1) & (test_ds['winner_model_b'] == 1),  # Both win\n",
    "            #\"model_b\", \"model_a\" # Removing tie breaking mechanism for now.\n",
    "            \"tie_positive\", \"tie_negative\" # Due to the lack of tie-positive cases in GPT4-Judge-Battles ds, former label won't be observed. This modification's for this specific ds only.\n",
    "        )\n",
    "    )\n",
    ")\n",
    "display(test_ds_enc.head(10))\n",
    "\n",
    "if choice == 3:\n",
    "    strong_model = 'gemma-2-27b-it'\n",
    "    weak_model = 'phi-3.5-mini-instruct'\n",
    "    print(f\"Strong model (a): {strong_model}\\nWeak model (b): {weak_model}\")\n",
    "\n",
    "elif choice == 4:\n",
    "    strong_model = 'Gemma-2-27b-it'\n",
    "    weak_model = 'Llama-3.2-1B-Instruct'\n",
    "    print(f\"Strong model (a): {strong_model}\\nWeak model (b): {weak_model}\")\n",
    "\n",
    "elif choice == 1:\n",
    "    strong_model = 'gpt-4-1106-preview'\n",
    "    weak_model = 'mixtral-8x7b-instruct-v0.1'\n",
    "    print(f\"Strong model (a): {strong_model}\\nWeak model (b): {weak_model}\")\n",
    "\n",
    "\n",
    "# Check if the strong and weak models are present in the MODEL_IDS dictionary\n",
    "if strong_model and weak_model not in MODEL_IDS:\n",
    "    raise ValueError(\"The strong and weak models must be present in the dataset.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EaoGggCnMHO"
   },
   "source": [
    "We now save encoded train & test splits locally, compute ratios in the training split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1387,
     "status": "ok",
     "timestamp": 1731141278306,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "iW1IdD10nMHO",
    "outputId": "65071277-5ff9-43c0-9739-df1b8cb08fc5"
   },
   "outputs": [],
   "source": [
    "os.makedirs(f\"{proj_dir}/output/dataset/encoded/train/\", exist_ok=True)\n",
    "train_ds_enc.to_parquet(f\"{proj_dir}/output/dataset/encoded/train/train_ds_encoded.parquet\")\n",
    "train_ds_enc = pd.read_parquet(f\"{proj_dir}/output/dataset/encoded/train/train_ds_encoded.parquet\")\n",
    "print('Shape of the encoded training dataset:', train_ds_enc.shape)\n",
    "print(\"Train dataset (encoded) saved at '~/output/dataset/encoded/train/train_ds_encoded.parquet'\")\n",
    "print(train_ds_enc.head(10))\n",
    "\n",
    "os.makedirs(f\"{proj_dir}/output/dataset/encoded/test/\", exist_ok=True)\n",
    "test_ds_enc.to_parquet(f\"{proj_dir}/output/dataset/encoded/test/test_ds_encoded.parquet\")\n",
    "test_ds_enc = pd.read_parquet(f\"{proj_dir}/output/dataset/encoded/test/test_ds_encoded.parquet\")\n",
    "print(\"\\nShape of the encoded test dataset:\", test_ds_enc.shape)\n",
    "print(\"Test dataset (encoded) saved at '~/output/dataset/encoded/test/test_ds_encoded.parquet'\")\n",
    "print(test_ds_enc.head(10))\n",
    "\n",
    "# Calculate ratios of wins by model a and model b\n",
    "\n",
    "# Assuming train_ds_enc contains the 'winner' column with \"model_a\" or \"model_b\" strings\n",
    "total_samples = len(train_ds_enc)\n",
    "\n",
    "# Count the wins for model_a and model_b\n",
    "model_a_wins = (train_ds_enc['winner'] == \"model_a\").sum()\n",
    "model_b_wins = (train_ds_enc['winner'] == \"model_b\").sum()\n",
    "\n",
    "# Calculate the ratios\n",
    "model_a_win_ratio = model_a_wins / total_samples\n",
    "model_b_win_ratio = model_b_wins / total_samples\n",
    "\n",
    "# Print the results\n",
    "print(f\"\\nModel A Win Ratio: {model_a_win_ratio:.2%}\")\n",
    "print(f\"Model B Win Ratio: {model_b_win_ratio:.2%}\")\n",
    "print(f\"Tie A (Positive) Ratio: {(train_ds_enc['winner'] == 'tie_positive').sum() / total_samples:.2%}\")\n",
    "print(f\"Tie B (Negative) Ratio: {(train_ds_enc['winner'] == 'tie_negative').sum() / total_samples:.2%}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ms37WSB8K8iu"
   },
   "source": [
    "# **Implementing Randomized Minority Oversampling Technique (RMOTE) for Mitigating Load Imbalance** (Skippable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SZYKLh1RUeQ4"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "\n",
    "def balance_classes(df, target_column, random_state=42):\n",
    "    class_counts = df[target_column].value_counts()\n",
    "    minority_class = class_counts.idxmin()\n",
    "    majority_class = class_counts.idxmax()\n",
    "    minority_count = class_counts[minority_class]\n",
    "    original_size = len(df)\n",
    "\n",
    "    majority_samples = df[df[target_column] == majority_class].sample(\n",
    "        n=minority_count, random_state=random_state\n",
    "    )\n",
    "    minority_samples = df[df[target_column] == minority_class]\n",
    "\n",
    "    balanced_df = pd.concat([minority_samples, majority_samples])\n",
    "    if len(balanced_df) < original_size:\n",
    "        additional_samples = balanced_df.sample(\n",
    "            n=original_size - len(balanced_df), random_state=random_state, replace=True\n",
    "        )\n",
    "        balanced_df = pd.concat([balanced_df, additional_samples])\n",
    "\n",
    "    balanced_df = balanced_df.sample(frac=1, random_state=random_state)\n",
    "    return balanced_df\n",
    "\n",
    "balanced_flag = True\n",
    "train_ds_enc = balance_classes(train_ds_enc, target_column='winner', random_state=42)\n",
    "test_ds_enc = balance_classes(test_ds_enc, target_column='winner', random_state=42)\n",
    "\n",
    "print('Shape of the balanced training dataset:', train_ds_enc.shape)\n",
    "print('Shape of the balanced test dataset:', test_ds_enc.shape)\n",
    "display(train_ds_enc.head(10))\n",
    "display(test_ds_enc.head(10))\n",
    "\n",
    "for split, dataset in [('Train', train_ds_enc), ('Test', test_ds_enc)]:\n",
    "    class_counts = dataset['winner'].value_counts(normalize=True) * 100\n",
    "    print(f\"\\n{split} dataset balanced.\")\n",
    "    print(f\"Model A Win Ratio: {class_counts.get('model_a', 0):.2f}%\")\n",
    "    print(f\"Model B Win Ratio: {class_counts.get('model_b', 0):.2f}%\")\n",
    "\n",
    "\"\"\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VleXgGqHnMHO"
   },
   "source": [
    "# Setting up Database & Dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 591,
     "status": "ok",
     "timestamp": 1731147957499,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "p_w2v6K6nMHO",
    "outputId": "e171516c-dcb7-4e21-a696-65e8fa96f840"
   },
   "outputs": [],
   "source": [
    "class BinaryClassificationDataset(Dataset):\n",
    "    def __init__(self, dataset):\n",
    "      self.model_a = torch.tensor(dataset['model_a_idx'].tolist())\n",
    "      self.model_b = torch.tensor(dataset['model_b_idx'].tolist())\n",
    "      self.prompt_id = dataset['prompt_idx'].tolist()\n",
    "      self.winner = dataset['winner'].tolist()\n",
    "\n",
    "      assert len(self.model_a) == len(self.model_b) == len(self.prompt_id) == len(self.winner), \"Data lengths do not match.\"\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.model_a)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "      if self.winner[idx] == \"model_a\":\n",
    "          return self.model_a[idx], self.model_b[idx], self.prompt_id[idx], 1\n",
    "      else:\n",
    "          return self.model_b[idx], self.model_a[idx], self.prompt_id[idx], 0\n",
    "\n",
    "    @staticmethod\n",
    "    def get_dataloaders(dataset, batch_size=64, shuffle=False):\n",
    "        return DataLoader(dataset, batch_size=batch_size, shuffle=shuffle)\n",
    "\n",
    "batch_size = 64\n",
    "print(\"Performing retrievals on 'train_ds_enc' for the first five indices:\")\n",
    "print(\"Winner, Loser, Prompt ID, Label\")\n",
    "for i in range(5):\n",
    "    print(BinaryClassificationDataset(train_ds_enc)[i])\n",
    "\n",
    "print(\"\\nPerforming retrievals on 'test_ds_enc' for the first five indices:\")\n",
    "print(\"Winner, Loser, Prompt ID, Label\")\n",
    "for i in range(5):\n",
    "    print(BinaryClassificationDataset(test_ds_enc)[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate static embeddings\n",
    "We now generate initializations of the $Q$ matrix in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424,
     "referenced_widgets": [
      "c16bb88c58c9449aa45deb43f4454db7",
      "d9f4f6509ea64f398adcec5c3edc4be1",
      "fc89b3edf24f47db8f06c58f2440a698",
      "cae9b189333841249f17537a61326ac1",
      "7ad3114669b04abfa71ee6edbb539df8",
      "cc6eaddd88e04f88b5ce68501f2e33ed",
      "136e258719e24276b2830cc35d7b3a6f",
      "74a88516d85640a6ab0f0799b8f84eaa",
      "736c17b38a0144cbbd586c24436b4058",
      "f96524a1483941808baa3189122855aa",
      "ab44d8e10a8441b7b59ccd780ed8889f",
      "69a0adcf50564fad84413c4cd7b2f886",
      "c570df5bb9194826ab0c3d65940c928e",
      "01d3d02837b54a41bc6968b72419ca5c",
      "a473ae7ad8214c75b0f34c6b59455a24",
      "d0156871b3c34f0c83aeb538445fa847",
      "2eb27638df5e4430be8f5abc791c9180",
      "9c08aed11ee14300b2cea357dcada841",
      "a8ec09d5a9844e4095df6b52acf0b3d9",
      "06af4b6c4c65401f8d2c0a7103d86a4d",
      "93f13fa677ad4b0db8f6b1827f60870e",
      "4091315db1924ed987ad5f14513fdeef",
      "6318f5c684584e3bb1a043dc1561c236",
      "ca494cdb14164e3294d61f99bf9b050f",
      "735d560d26df43edaa5ec37342e9f71a",
      "b6c82f17f0504e68b9abb2887c9a9a0b",
      "5ca1c5cd0b674f4fb7c8f82d8ab42a49",
      "84cc8ceebea74c8d91977b1754fde2ff",
      "8cd02cf221cb442091beb51ed9b7a09f",
      "3c65f4baee354488be5284aa7f4a3164",
      "6ef5fbf2f83d43bf8b90f29a7a16389c",
      "e67c6a592b4c4312994ef0945f713676",
      "4ac23602151c4bf28c8f417e0945bc0a",
      "1d9a6ee2a81c428ebb7bf594915571f7",
      "a80cb99b9fba471a833777125e89a24d",
      "91cff64ff62e435198564a9b67ccd8a2",
      "eec50b4d836c4dce9623dc7173220cfa",
      "674c3e28440c421b83be2f9196194ced",
      "dba66e67542442f18a79221582909f87",
      "876c999bfc17412ab5b66cd54ebc1d65",
      "c49a64a016354f9bbb2bb96e458a2d67",
      "350bac7528bc4265a215024a38491777",
      "325057d75312412b96f084601ee1b2df",
      "7f1a490e019e4da79d09865e18d8c204",
      "7d18d244a3ce416dbbb5ccf8f34b2c95",
      "93e0eed03a1948efbe87d40496d8c876",
      "10e250cc302446148c509483727c4979",
      "002177b896964f0aaa7ecc600b0513cb",
      "874565ca9f454f5091cb5105a46896ec",
      "c1a8140c53c74f92ae74cf10fc8fd36d",
      "35f95e7e3f9247618d2ee30fefbb8284",
      "b56a83eafc744ef2b6e5c2e502ae41b8",
      "29ebfce9e15341d382a0c4f24b585360",
      "7194332781924e1d8ebc58d8f7f16ef6",
      "46471099f8e4475abf1cfd9420329774",
      "f0c5f6d0056843ff82c4f2328f578959",
      "2b12b1093641443897b04be5e6b8195f",
      "0abd74af64fd435086157feb4ca3d8c4",
      "750ba152f40a4c2291c81da14ad4e364",
      "e94e389b54644b3ea8689525bfc8c399",
      "27b373cac40d43938c5f775b328416ed",
      "f10c9558c04f49c2b9430437056ac3c0",
      "578a592a3cbe433f9a0d902c017f098d",
      "2f06de91cfc64f44a6bc0d1c0bc3fe29",
      "f059007fccce41f4a4f1a487fd25d1d4",
      "109589570fe148c9b239a9be08ce3b94",
      "f1de0ba8b5c44238a6ff42b9ca244297",
      "d20b4ae836694dd8942909af1b00ddaa",
      "f29bd005bd644961b47a48ef3cd026dd",
      "1643c63b91f2466192584e7e336c3b06",
      "4c2ba62c4f8a4bfc94969e03aa646fd8",
      "92b3cb67c61c4fa5855a9de91ba0cbc6",
      "7ef4a7aa9b3c4f5094170f33a4e5e3ad",
      "9d212ae530ed47889b506bdcb49c5a19",
      "de76a6d133fb467382b609f91e6ad3f3",
      "062095703c844bf69ce028e18c25d28a",
      "9f170d452d284be08198702e83585129",
      "e15f7df8c2e6488c91acbd407faa18fe",
      "0f5bb5f6771d47e3be34ec1a6f1ffa01",
      "cfe7f87ad6504f9a972509e4db5ad536",
      "f3d2e3530097404aab643e496c2c7452",
      "9e672a44264045c98cfcb0ee84e9b728",
      "dc510ba5fab44c09a12a6688f4e036cc",
      "fda5c07ad06c48999d0728c2ee3695b7",
      "b0c95c854b93455696b44acba6736661",
      "f0f69756be25457080d4501069884578",
      "606da97e6c4e4dea85d63a9e96445652",
      "b63828704fb642dea1e2f9c85ac04724",
      "55ae3912f5674dfb88634511f19367ad",
      "5e25188e481044508e24f5d868ed17c0",
      "e36ee90305cd40a485f4a512ad666d96",
      "4d54d2e646964dada7c741d0731c7b43",
      "fc5005bfa20340c4a95b0495ec0d070e",
      "b6748055f7f44df39788108725b5d0c4",
      "fa7efd738b7a44a4887def399bded0b9",
      "42f0d4f204e946e6a0fd2e5575411859",
      "4723d473e946430f96456c6152b93462",
      "f9a78ce553954c8eaff8c54b1bdea197",
      "db4fd2b9f65a4b96bbdf2d8fc806281b",
      "af746483961b403988e0310ea2a9cd02",
      "161e0c7fd134456987f9aa7161d39492",
      "6c610f721471462191828c261f168125",
      "d4f1d59d1aa943c8994df940d54fe1b2",
      "4551d96223f04229bda5c29cc661f16f",
      "8da4332f1e8849a69da341a3099d0159",
      "e1a7384b05f346d3946d3451e5a2c419",
      "0fa9c417f6a94bf8a19a4c621a923783",
      "03895a74977445de90877ea1bc87b091",
      "230a5621b8d74eb39a5248a57a6f89d5",
      "714f809016ea460ea9b27867f2b2d7e7",
      "ba0495c43b0a4ccba393ca46e25df20c",
      "38e26bdc72b743e685458e34a16bca22",
      "fda9685a928843febe081fbce28aa9b5",
      "55d19c28064a481c9637d305735ccf4a",
      "1e4eea0a5d56428c8392403eb7e7dbd8",
      "c71f4935dd4a43c98f15c4f7d00bd26d",
      "fcd0257480894422832b1c90f29baf4e",
      "5f7fb54166c34fe9b6ed6368a4efe7dd",
      "b4aa5b3a9c184162ac280bf44c71713b",
      "fda0389270064beca44370713527b062",
      "e61e4a2733ad49a58f24556ea9c061ce"
     ]
    },
    "executionInfo": {
     "elapsed": 10930,
     "status": "ok",
     "timestamp": 1730808858929,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "YlGm8pw3XrB0",
    "outputId": "30d5ea3c-c584-41e7-ba81-519949c32905"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Try out different models from the Sentence Transformers library:\n",
    "- Need to implement support for infgrad/stella_en_400M_v5 (1024-dimensional embeddings) (Good performance, as claimed by RouteLLM authors)\n",
    "- all-mpnet-base-v2 (768-dimensional embeddings) # Okayish performance.\n",
    "- all-MiniLM-L6-v2 (384-dimensional embeddings) # Worse. Not recommended.\n",
    "\"\"\"\n",
    "\n",
    "#model = SentenceTransformer(\"sentence-transformers/all-mpnet-base-v2\", trust_remote_code=True, cache_folder=f\"{proj_dir}/output/embeddings/embedding-model/\", local_files_only=True)\n",
    "embedding_generator = \"sentence-transformers/all-mpnet-base-v2\"\n",
    "embedding_model = SentenceTransformer(embedding_generator, trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 402,
     "status": "ok",
     "timestamp": 1731148524928,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "hwUr047PIos4",
    "outputId": "73d5f21b-be30-4a57-d5cd-4e2a064518a4"
   },
   "outputs": [],
   "source": [
    "def calculate_win_percentages(dataset):\n",
    "    model_a_wins = 0\n",
    "    model_b_wins = 0\n",
    "\n",
    "    for winner in dataset.winner:\n",
    "        if winner == \"model_a\":\n",
    "            model_a_wins += 1\n",
    "        elif winner == \"model_b\":\n",
    "            model_b_wins += 1\n",
    "\n",
    "    total_matches = model_a_wins + model_b_wins\n",
    "\n",
    "    model_a_win_percentage = (model_a_wins / total_matches) * 100 if total_matches > 0 else 0\n",
    "    model_b_win_percentage = (model_b_wins / total_matches) * 100 if total_matches > 0 else 0\n",
    "\n",
    "    return model_a_win_percentage, model_b_win_percentage\n",
    "\n",
    "print(\"\\nTraining dataset:\")\n",
    "model_a_win_percentage, model_b_win_percentage = calculate_win_percentages(train_ds_enc)\n",
    "print(f\"Model A Win Percentage: {model_a_win_percentage:.2f}%\")\n",
    "print(f\"Model B Win Percentage: {model_b_win_percentage:.2f}%\")\n",
    "\n",
    "print(\"\\nTest dataset:\")\n",
    "model_a_win_percentage, model_b_win_percentage = calculate_win_percentages(test_ds_enc)\n",
    "print(f\"Model A Win Percentage: {model_a_win_percentage:.2f}%\")\n",
    "print(f\"Model B Win Percentage: {model_b_win_percentage:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6482,
     "status": "ok",
     "timestamp": 1731145378925,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "8PF9ZEjWnMHP",
    "outputId": "8ef11e6c-971d-4cb5-cf31-6476b9f10beb"
   },
   "outputs": [],
   "source": [
    "def load_npy_file(npy_location):\n",
    "    embeddings = np.load(npy_location)\n",
    "    return embeddings, embeddings.shape[0]\n",
    "\n",
    "def match_shape(embeddings, split_size):\n",
    "    return embeddings.shape[0] == split_size\n",
    "\n",
    "def generate_embeddings(split, model):\n",
    "    prompts = train_ds['prompt'].tolist() if split == 'train' else test_ds['prompt'].tolist()\n",
    "    embeddings = embedding_model.encode(\n",
    "        prompts, normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=True, batch_size=1\n",
    "    )\n",
    "    return embeddings\n",
    "\n",
    "embeddings_path = f\"{proj_dir}/output/embeddings/\"\n",
    "\n",
    "if not os.path.exists(embeddings_path):\n",
    "    os.makedirs(embeddings_path, exist_ok=True)\n",
    "    print(f\"'embeddings' directory created at {embeddings_path}.\")\n",
    "\n",
    "# Process embeddings for each split\n",
    "for split in [\"train\", \"test\"]:\n",
    "    split_size = len(train_ds_enc) if split == 'train' else len(test_ds_enc)\n",
    "    file_path = f\"{embeddings_path}/query_embeddings_{split}.npy\"\n",
    "    \n",
    "    if os.path.exists(file_path):\n",
    "        query_embeddings, embed_dim = load_npy_file(file_path)\n",
    "        print(f\"Precomputed embeddings for {split} split loaded successfully.\")\n",
    "        assert match_shape(query_embeddings, split_size), f\"Embedding shape mismatch for {split} split.\"\n",
    "        \n",
    "\n",
    "    else:\n",
    "        print(f\"No precomputed embeddings found for {split} split. Generating embeddings.\")\n",
    "        query_embeddings = generate_embeddings(split, embedding_model)\n",
    "        np.save(file_path, query_embeddings)\n",
    "        print(f\"Saved embeddings for {split} split at {file_path}.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "miS5zE7rnMHP"
   },
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The logits for the model selection are computed using the following formula:\n",
    "  $$z = \\mathbf{W}_c \\cdot \\left( (\\mathbf{P}(m_{\\text{win}}) - \\mathbf{P}(m_{\\text{loss}})) \\odot \\mathbf{Q}(q) \\right)$$\n",
    "  Where:\n",
    "  - $\\mathbf{P}(m_{\\text{win}})$ and $\\mathbf{P}(m_{\\text{loss}})$ represent learnable embeddings of the winning and losing models, respectively.\n",
    "  - $\\mathbf{Q}(q)$ is the frozen query embedding generated by a sentence transformer, used for comparison with model embeddings.\n",
    "  - $\\mathbf{W}_c$ is a linear classifier that maps the interaction between the model-query pair to a logit.\n",
    "  - $\\odot$ denotes the element-wise (Hadamard) product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 540,
     "status": "ok",
     "timestamp": 1731146518264,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "O4HBv8ownMHP"
   },
   "outputs": [],
   "source": [
    "class MFGate(nn.Module):\n",
    "      def __init__(\n",
    "          self,\n",
    "          num_models=2,\n",
    "          model_dim=128,\n",
    "          num_queries=len(train_ds['prompt']),\n",
    "          text_dim=768,\n",
    "          num_output_nodes=1,\n",
    "          down_projection=True,\n",
    "          embedding_path=None,\n",
    "          ):\n",
    "        \n",
    "        super(MFGate, self).__init__()\n",
    "        \n",
    "        # P (Model embeddings)\n",
    "        self.num_models = num_models\n",
    "        self.P_dim = model_dim\n",
    "        self.P = nn.Embedding(self.num_models, self.P_dim).requires_grad_(True)\n",
    "        #self.P.weight.data.normal_(0, 1) # Initialize model embeddings with normal distribution (experimental)\n",
    "\n",
    "        # Q (Prompt/Query embeddings)\n",
    "        self.num_queries = num_queries\n",
    "        self.Q_dim = text_dim\n",
    "        self.Q = nn.Embedding(self.num_queries, self.Q_dim).requires_grad_(False) # Query embeddings are frozen. You dont't want to update them.\n",
    "\n",
    "        if embedding_path is not None:\n",
    "          self.checkpointfile = embedding_path\n",
    "          try:\n",
    "            embeddings = np.load(self.checkpointfile)\n",
    "            self.Q.weight.data.copy_(torch.tensor(embeddings))\n",
    "          except FileNotFoundError:\n",
    "            print(\"Could not load prompt embeddings from file. Please check the path.\")\n",
    "\n",
    "        # W_proj (Q -> P) - Down-projection layer\n",
    "        self.down_proj = down_projection\n",
    "        if self.down_proj:\n",
    "          self.downproj_QtoP = nn.Linear(self.Q_dim, self.P_dim, bias=False).requires_grad_(True)\n",
    "          #self.downproj_QtoP.weight.data.normal_(0, 1) # Initialize down-projection layer with normal distribution (experimental)\n",
    "          \n",
    "        else:\n",
    "          assert self.Q_dim == self.P_dim, \"If not using projection: make sure Q_dim and P_dim match.\"\n",
    "          self.downproj_QtoP = None\n",
    "\n",
    "        # Linear classifier\n",
    "        self.num_ways = num_output_nodes\n",
    "        self.Classifier = nn.Linear(self.P_dim, self.num_ways, bias=False).requires_grad_(True)\n",
    "        #self.Classifier.weight.data.normal_(0, 1) # Initialize classifier layer with normal distribution (experimental)\n",
    "\n",
    "      def get_device(self):\n",
    "        return next(self.parameters()).device\n",
    "\n",
    "      def forward(self, model_win, model_loss, prompt, pe_noise=0):\n",
    "        \n",
    "        model_win = model_win.to(self.get_device())\n",
    "        model_loss = model_loss.to(self.get_device())\n",
    "        prompt = prompt.to(self.get_device())\n",
    "\n",
    "        model_strong_embedding = self.P(model_win)\n",
    "        model_strong_embedding = F.normalize(model_strong_embedding, p=2, dim=1)\n",
    "        model_weak_embedding = self.P(model_loss)\n",
    "        model_weak_embedding = F.normalize(model_weak_embedding, p=2, dim=1)\n",
    "        prompt_embedding = self.Q(prompt)\n",
    "\n",
    "        #if prompt_embedding.norm(p=2, dim=1).mean() > 1.001: # Uncomment if not already normalized.\n",
    "          #prompt_embedding = F.normalize(prompt_embedding, p=2, dim=1) # Normalize the prompt embeddings if they are not already normalized.\n",
    "\n",
    "        if pe_noise > 0:\n",
    "          prompt_embedding += pe_noise * torch.randn_like(prompt_embedding)\n",
    "        if self.down_proj:\n",
    "          prompt_embedding = self.downproj_QtoP(prompt_embedding)\n",
    "        return self.Classifier((model_strong_embedding - model_weak_embedding) * prompt_embedding).squeeze()\n",
    "\n",
    "      @torch.no_grad()\n",
    "      def predict(self, model_win, model_loss, prompt, pe_noise=0.05):\n",
    "        logits = self.forward(model_win, model_loss, prompt, pe_noise)\n",
    "        return logits > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eoRN0YKunMHP"
   },
   "source": [
    "# Training Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is trained using a **Binary Cross-Entropy with Logits Loss** function:\n",
    "$$L = - \\frac{1}{N} \\sum_{i=1}^{N} \\left[ y_i \\cdot \\log(\\sigma(z_i)) + (1 - y_i) \\cdot \\log(1 - \\sigma(z_i)) \\right]$$\n",
    "Which applies a sigmoid activation of the form $P(\\text{M}_{strong} | q) = \\sigma(z) = \\frac{1}{1 + e^{-z}}$\n",
    "to the logits before computing the error term. Where $y_i$ is the binary target label for each query (1 for the stronger model, 0 for the weaker model). The loss function essentially becomes:\n",
    "$$L = - \\frac{1}{N} \\sum_{i=1}^{N} \\left[ y_i \\cdot \\log(\\sigma(z_i))\\right]$$\n",
    "\n",
    "The loss measures the error between the predicted logits and the target label of ones, which essentially converts the problem into likelihood maximization task, where the model is encouraged to assign a higher score to the stronger model (LLM) relative to the weaker model (SLM)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gkpHqBgCnMHP"
   },
   "outputs": [],
   "source": [
    "class MFTrainer:\n",
    "    def __init__(\n",
    "            self,\n",
    "            device,\n",
    "            model,\n",
    "            train_loader,\n",
    "            val_loader,\n",
    "            noise=0.0,\n",
    "            optimizer=None,\n",
    "            scheduler=None,\n",
    "            early_stopping_patience=3\n",
    "    ):\n",
    "        self.device = torch.device(device if torch.cuda.is_available() else 'cpu')\n",
    "        self.model = model.to(self.device)\n",
    "        self.train_loader = train_loader\n",
    "        self.val_loader = val_loader\n",
    "        self.train_criterion = nn.BCEWithLogitsLoss(reduction=\"mean\")\n",
    "        self.val_criterion = nn.BCEWithLogitsLoss(reduction=\"sum\")\n",
    "        self.noise = noise\n",
    "        self.optimizer = optimizer\n",
    "        self.scheduler = scheduler\n",
    "        self.best_val_loss = float('inf')\n",
    "        self.early_stopping_patience = early_stopping_patience\n",
    "        self.early_stopping_counter = 0\n",
    "        self.train_losses = []\n",
    "        self.val_losses = []\n",
    "        self.weight_norms = []\n",
    "\n",
    "    def _compute_weight_norm(self):\n",
    "        weight_norm = 0.0\n",
    "        for param in self.model.parameters():\n",
    "            if param.requires_grad:\n",
    "                weight_norm += param.data.norm().item() ** 2\n",
    "        return weight_norm ** 0.5\n",
    "\n",
    "    def train_epoch(self):\n",
    "        \"\"\"\n",
    "        Run a single training pass.\n",
    "        \"\"\"\n",
    "        self.model.train()\n",
    "        running_loss = 0.0\n",
    "\n",
    "        for models_a, models_b, prompts, labels in tqdm(self.train_loader, desc=\"Training...\", leave=False):\n",
    "            models_a, models_b, prompts, labels = self._move_to_device(models_a, models_b, prompts, labels)\n",
    "            self.optimizer.zero_grad()         \n",
    "\n",
    "            # Forward pass\n",
    "            outputs = self.model(model_win=models_a, model_loss=models_b, prompt=prompts, pe_noise=0.0)\n",
    "            # Compute loss\n",
    "            loss = self.train_criterion(outputs, torch.ones_like(outputs)) # BCEWithLogitsLoss computes loss after passing applying a sigmoid transformation for normalization.\n",
    "            # Backpropagate\n",
    "            loss.backward()\n",
    "            # Monitor gradient norms before clipping\n",
    "            #torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)\n",
    "\n",
    "            self.optimizer.step()\n",
    "\n",
    "            running_loss += loss.item() * models_a.size(0)\n",
    "        epoch_loss = running_loss / len(self.train_loader.dataset)\n",
    "        weight_norm = self._compute_weight_norm()\n",
    "        self.weight_norms.append(weight_norm)\n",
    "\n",
    "        return epoch_loss\n",
    "\n",
    "    def validate_epoch(self):\n",
    "        \"\"\"\n",
    "        Run a single validation pass.\n",
    "        \"\"\"\n",
    "        self.model.eval()\n",
    "        running_loss = 0.0\n",
    "        correct_predictions = 0\n",
    "        total_samples = 0\n",
    "\n",
    "        with torch.no_grad():\n",
    "            for models_a, models_b, prompts, labels in tqdm(self.val_loader, desc=\"Validating...\", leave=False):\n",
    "                models_a, models_b, prompts, labels = self._move_to_device(models_a, models_b, prompts, labels)\n",
    "\n",
    "                # Forward pass\n",
    "                outputs = self.model(model_win=models_a, model_loss=models_b, prompt=prompts, pe_noise=0.05)\n",
    "                # Compute loss\n",
    "                loss = self.val_criterion(outputs, torch.ones_like(outputs))\n",
    "                running_loss += loss.item()\n",
    "                # Compute predictions and accuracy\n",
    "                predictions = self.model.predict(models_a, models_b, prompts, pe_noise=0.1)\n",
    "                correct_predictions += (predictions == torch.ones_like(predictions)).sum().item()\n",
    "                total_samples += torch.ones_like(predictions).size(0)\n",
    "\n",
    "        epoch_loss = running_loss / len(self.val_loader.dataset)\n",
    "        accuracy = correct_predictions / total_samples\n",
    "\n",
    "        return epoch_loss, accuracy\n",
    "\n",
    "    def train(self, num_epochs, save_best=1):\n",
    "        \"\"\"\n",
    "        Main training loop that runs over multiple epochs.\n",
    "        \"\"\"\n",
    "        for epoch in range(num_epochs):\n",
    "            epoch_str = f\"Epoch {epoch + 1}/{num_epochs}\"\n",
    "            \n",
    "            train_loss = self.train_epoch()\n",
    "            val_loss, val_acc = self.validate_epoch()\n",
    "            \n",
    "            if save_best:\n",
    "                is_best = val_loss < self.best_val_loss\n",
    "            else:\n",
    "                is_best = True\n",
    "            \n",
    "            self._save_best_model(val_loss, train_loss, epoch, num_epochs, best=is_best)\n",
    "\n",
    "            if self.scheduler:\n",
    "                self.scheduler.step(val_loss)\n",
    "\n",
    "            self.train_losses.append(train_loss)\n",
    "            self.val_losses.append(val_loss)\n",
    "\n",
    "            if not self._early_stopping(val_loss):\n",
    "                break\n",
    "\n",
    "        self.plot_training_curve()\n",
    "        self.plot_weight_norms()\n",
    "\n",
    "    def plot_weight_norms(self):\n",
    "        \"\"\"\n",
    "        Plot the weight norm over epochs to track vanishing/exploding gradients.\n",
    "        \"\"\"\n",
    "        epochs = len(self.weight_norms)\n",
    "        plt.figure(figsize=(8, 5))\n",
    "        plt.plot(range(epochs), self.weight_norms, label=\"Weight Norm\")\n",
    "        plt.title(\"Weight Norm over Epochs\")\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Weight Norm (L2 Norm)\")\n",
    "        plt.legend()\n",
    "        plt.grid(True)\n",
    "        plt.tight_layout()\n",
    "        plt.show()\n",
    "\n",
    "\n",
    "    def _move_to_device(self, models_a, models_b, prompts, labels):\n",
    "        models_a = models_a.to(self.device)\n",
    "        models_b = models_b.to(self.device)\n",
    "        prompts = prompts.to(self.device)\n",
    "        labels = labels.to(self.device)\n",
    "        return models_a, models_b, prompts, labels\n",
    "\n",
    "    def _save_best_model(self, val_loss, train_loss, epoch, num_epochs, best=False):\n",
    "        if best:\n",
    "            if val_loss < self.best_val_loss:\n",
    "                self.best_val_loss = val_loss\n",
    "                checkpoint = {\n",
    "                    'model': self.model.state_dict(),\n",
    "                    'optimizer': self.optimizer.state_dict(),\n",
    "                    'scheduler': self.scheduler.state_dict() if self.scheduler else None,\n",
    "                    'epoch': epoch,\n",
    "                    'train_loss': train_loss,\n",
    "                    'val_loss': val_loss\n",
    "                }\n",
    "                torch.save(checkpoint, f\"{proj_dir}/output/best_train.pth\")\n",
    "                print(f\"[Best model saved at epoch {epoch + 1} with Validation Loss: {val_loss:.4f}]\")\n",
    "                self.early_stopping_counter = 0\n",
    "            else:\n",
    "                self.early_stopping_counter += 1\n",
    "        else:\n",
    "            # Save model unconditionally\n",
    "            checkpoint = {\n",
    "                'model': self.model.state_dict(),\n",
    "                'optimizer': self.optimizer.state_dict(),\n",
    "                'scheduler': self.scheduler.state_dict() if self.scheduler else None,\n",
    "                'epoch': epoch,\n",
    "                'train_loss': train_loss,\n",
    "                'val_loss': val_loss\n",
    "            }\n",
    "            torch.save(checkpoint, f\"{proj_dir}/output/best_train.pth\")\n",
    "            print(f\"[Model saved at epoch {epoch + 1}] | Training Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}\")\n",
    "\n",
    "    def _early_stopping(self, val_loss):\n",
    "        \"\"\"\n",
    "        Helper function to handle early stopping logic.\n",
    "        \"\"\"\n",
    "        if self.early_stopping_patience is None:\n",
    "            return True  # Continue training\n",
    "\n",
    "        if self.early_stopping_counter >= self.early_stopping_patience:\n",
    "            print(f\"\\nNo improvement observed at patience level {self.early_stopping_patience}. Stopping early.\")\n",
    "            return False\n",
    "        return True  # Continue training\n",
    "\n",
    "    def plot_training_curve(self):\n",
    "        \"\"\"\n",
    "        Plot the training and validation loss curves after training is complete.\n",
    "        \"\"\"\n",
    "        epochs = len(self.train_losses)\n",
    "        plt.figure(figsize=(8, 5))\n",
    "        plt.plot(range(epochs), self.train_losses, label=\"Training Loss\")\n",
    "        plt.plot(range(epochs), self.val_losses, label=\"Validation Loss\")\n",
    "        plt.title(\"Loss over Epochs\")\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Loss\")\n",
    "        plt.legend()\n",
    "        plt.grid(True)\n",
    "        plt.tight_layout()\n",
    "        plt.show()\n",
    "\n",
    "# --------------------------------------------------------------------------------------\n",
    "\n",
    "model_dim = 128\n",
    "embedding_path = f\"{proj_dir}/output/embeddings/query_embeddings_train.npy\"\n",
    "query_embeddings = np.load(embedding_path)\n",
    "num_queries = len(train_ds['prompt'])\n",
    "assert query_embeddings.shape[0] == num_queries, f\"Number of queries ({num_queries}) does not match number of embeddings ({query_embeddings.shape[0]})\"\n",
    "\n",
    "model = MFGate(\n",
    "    num_models=len(MODEL_IDS),\n",
    "    model_dim=model_dim,\n",
    "    num_queries=num_queries,\n",
    "    text_dim=768,\n",
    "    num_output_nodes=1,\n",
    "    down_projection=True,\n",
    "    embedding_path=embedding_path\n",
    ").cuda()\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_dataset = BinaryClassificationDataset(train_ds_enc)\n",
    "test_dataset = BinaryClassificationDataset(test_ds_enc)\n",
    "\n",
    "batch_size = 64\n",
    "train_spl_loader = BinaryClassificationDataset.get_dataloaders(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "val_spl_loader = BinaryClassificationDataset.get_dataloaders(test_dataset, batch_size=1024, shuffle=True)\n",
    "\n",
    "lr = 3e-4\n",
    "weight_decay = 1e-5\n",
    "noise = 0.05 # Setting to 1 destroys query embedding information.\n",
    "optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)\n",
    "scheduler = None\n",
    "#scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min')\n",
    "patience = None  # Set early stopping patience to avoid overfitting, set to None to disable\n",
    "\n",
    "print(f\"{model}\\n\\nNumber of parameters in the model: {sum(p.numel() for p in model.parameters())}\")\n",
    "\n",
    "trainer = MFTrainer(\n",
    "    device=device,\n",
    "    model=model,\n",
    "    train_loader=train_spl_loader,\n",
    "    val_loader=val_spl_loader,\n",
    "    noise=noise,\n",
    "    optimizer=optimizer,\n",
    "    scheduler=scheduler,\n",
    "    early_stopping_patience=patience\n",
    ")\n",
    "\n",
    "trainer.train(num_epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = f\"{proj_dir}/output/best_train.pth\"\n",
    "checkpoint = torch.load(model_path)\n",
    "\n",
    "model = MFGate(\n",
    "    num_models=len(MODEL_IDS),\n",
    "    model_dim=128,\n",
    "    num_queries=len(train_ds['prompt']),\n",
    "    text_dim=768,\n",
    "    num_output_nodes=1,\n",
    "    down_projection=True,\n",
    "    embedding_path=f\"{proj_dir}/output/embeddings/query_embeddings_train.npy\"\n",
    ").cuda()\n",
    "\n",
    "model.load_state_dict(checkpoint['model'])\n",
    "model.eval()\n",
    "print(\"Model loaded successfully from best_train.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bEshTiqjnMHQ"
   },
   "source": [
    "# Decision Boundary Estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, the proportion of calls can be adjusted via specifying a parameter $\\text{PCT}_{strong}$, based on which a cost threshold $\\alpha$ is computed that controls the proportion of calls made to the LLM, which is calculated as the $\\alpha^{th}$ percentile over the distribution of router predictions (win-rates) as $$\\alpha = Q(1-\\text{PCT}_{strong})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "executionInfo": {
     "elapsed": 3424,
     "status": "ok",
     "timestamp": 1730798696971,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "DPZUS3pHKmFT",
    "outputId": "e924182a-9ab4-4a85-fc53-465ff48c88cf"
   },
   "outputs": [],
   "source": [
    "def calculate_and_plot_threshold(win_rates, strong_model_pct):\n",
    "    threshold = win_rates.quantile(1 - strong_model_pct)\n",
    "    return threshold\n",
    "\n",
    "test_dataset = BinaryClassificationDataset(test_ds_enc)\n",
    "test_loader = BinaryClassificationDataset.get_dataloaders(test_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "win_rates = []\n",
    "count = 0\n",
    "with torch.no_grad():\n",
    "    for models_a, models_b, prompts, labels in tqdm(test_loader, desc=\"Calculating Win Rates...\", leave=False):\n",
    "        models_a, models_b, prompts, labels = models_a.cuda(), models_b.cuda(), prompts.cuda(), labels.cuda()\n",
    "\n",
    "        outputs = model(models_a, models_b, prompts, pe_noise=0)\n",
    "        outputs = torch.sigmoid(outputs).cuda()\n",
    "        count += 1\n",
    "        win_rates.extend(outputs.cpu().numpy())\n",
    "\n",
    "tdf = pd.DataFrame(win_rates, columns=['mf_wr'])\n",
    "\n",
    "strong_model_pcts = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]\n",
    "router_model = 'mf_wr'\n",
    "\n",
    "threshold_results = pd.DataFrame({\n",
    "    'strong_model_pct': strong_model_pcts,\n",
    "    'threshold': [calculate_and_plot_threshold(tdf[router_model], pct) for pct in strong_model_pcts]\n",
    "})\n",
    "\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.hist(tdf[router_model], bins=50, color='blue', alpha=0.7)\n",
    "plt.title(\"Distribution of Win Rates\")\n",
    "plt.xlabel(\"Win Rate\")\n",
    "plt.ylabel(\"Frequency\")\n",
    "plt.grid(True)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "display(threshold_results)\n",
    "print(count*batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yE1wDMl1nMHS"
   },
   "source": [
    "# Save model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 480,
     "status": "ok",
     "timestamp": 1730798702403,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "HPcS5OhCnMHS",
    "outputId": "66c4f5ab-c034-4929-b7ac-6ca2b95a350c"
   },
   "outputs": [],
   "source": [
    "inference_model = {\n",
    "    'P.weight': model.P.weight,\n",
    "    'downproj_QtoP.weight': model.downproj_QtoP.weight,\n",
    "    'Classifier.weight': model.Classifier.weight\n",
    "}\n",
    "\n",
    "torch.save(inference_model, f\"{proj_dir}/output/BUD-MFR-Binary-1.0.pth\")\n",
    "print(\"Inference model saved at ~/output/BUD-MFR-Binary-1.0.pth'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q1KheVOT2Vk2"
   },
   "source": [
    "# Model (Inference pass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JO58vi4szC6Z"
   },
   "outputs": [],
   "source": [
    "class MFRB(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            num_models=len(MODEL_IDS),\n",
    "            model_dim=128,\n",
    "            text_dim=768,\n",
    "            num_output_nodes=1,\n",
    "            down_projection=True\n",
    "        ):\n",
    "        super(MFRB, self).__init__()\n",
    "\n",
    "        # P (Model embeddings)\n",
    "        self.num_models = num_models\n",
    "        self.P_dim = model_dim\n",
    "        self.P = nn.Embedding(self.num_models, self.P_dim)\n",
    "\n",
    "        # W_proj (Q -> P)\n",
    "        self.down_proj = down_projection\n",
    "        if self.down_proj:\n",
    "            self.downproj_QtoP = nn.Linear(text_dim, model_dim, bias=False)\n",
    "        else:\n",
    "            assert text_dim == model_dim, \"Q_dim must be equal to P_dim.\"\n",
    "            self.downproj_QtoP = None\n",
    "\n",
    "        # Linear classifier\n",
    "        self.num_ways = num_output_nodes\n",
    "        self.Classifier = nn.Linear(model_dim, self.num_ways, bias=False)\n",
    "\n",
    "    def get_device(self):\n",
    "        return next(self.parameters()).device\n",
    "\n",
    "    def forward(self, model_ids, prompt_embed, noise):\n",
    "        model_ids = torch.tensor(model_ids, dtype=torch.long).to(self.get_device())\n",
    "        model_embed = self.P(model_ids)\n",
    "        model_embed = F.normalize(model_embed, p=2, dim=1)\n",
    "\n",
    "        if self.down_proj:\n",
    "            prompt_embed = self.downproj_QtoP(prompt_embed)\n",
    "\n",
    "        prompt_embed += torch.randn_like(prompt_embed) * noise\n",
    "        #prompt_embed = F.normalize(prompt_embed, p=2, dim=1) - No need. Prompt embeddings are already normalized across rows.\n",
    "\n",
    "        return self.Classifier(model_embed * prompt_embed).squeeze()\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def predict_win_rate(self, model_win_idx, model_loss_idx, prompt_text_embed, noise):\n",
    "        prompt = prompt_text_embed\n",
    "        #prompt = embedding_model.encode(prompt_text, convert_to_tensor=True, normalize_embeddings=True).to(self.get_device())\n",
    "        logits = self.forward(model_ids=[model_win_idx, model_loss_idx], prompt_embed=prompt, noise=noise)\n",
    "        winrate = torch.sigmoid(logits[0] - logits[1]).item()\n",
    "        return winrate # Returns confidence score in between [0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 552,
     "status": "ok",
     "timestamp": 1730798706767,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "Kh7TbXgNzYKU",
    "outputId": "66610b9e-298e-4b9d-8ef9-06b70853587b"
   },
   "outputs": [],
   "source": [
    "model_dir = \"./output/\"\n",
    "\n",
    "checkpoint_path=f\"{model_dir}/BUD-MFR-Binary-1.0.pth\"\n",
    "inf_model = MFRB().to(\"cuda\")\n",
    "\n",
    "try:\n",
    "    inf_model.load_state_dict(torch.load(checkpoint_path, weights_only=True))\n",
    "    print(f\"✅ BUD MFR Binary checkpoints successfully loaded from {checkpoint_path}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"❌ Error loading BUD MFR Binary checkpoints from {checkpoint_path}: {str(e)}\")\n",
    "    raise RuntimeError(f\"Failed to load the model from {checkpoint_path}.\") from e\n",
    "\n",
    "finally:\n",
    "    inf_model.eval()\n",
    "    print(inf_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TvSpqUwlnMHR"
   },
   "source": [
    "# Compute metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 842
    },
    "executionInfo": {
     "elapsed": 19185,
     "status": "ok",
     "timestamp": 1730798914898,
     "user": {
      "displayName": "Shreyan Chaubey",
      "userId": "05168610280685752469"
     },
     "user_tz": -330
    },
    "id": "SPDu0UE-nMHS",
    "outputId": "cf46e4a4-8402-40c4-ea7f-d18e28e88ffa"
   },
   "outputs": [],
   "source": [
    "def route(model_a, model_b, prompt, threshold):\n",
    "    win_rate = inf_model.predict_win_rate(model_a, model_b, prompt, noise=0.05) # Jitter for improved generalization, set at 0.05\n",
    "    if win_rate >= threshold:\n",
    "        return \"model_a\"\n",
    "    else:\n",
    "        return \"model_b\"\n",
    "\n",
    "def batch_process(dataset, threshold):\n",
    "    true_positives = 0\n",
    "    true_negatives = 0\n",
    "    false_positives = 0\n",
    "    false_negatives = 0\n",
    "    model_a_calls = 0\n",
    "    model_b_calls = 0\n",
    "    win_rates = []\n",
    "    true_labels = []\n",
    "\n",
    "    for idx in tqdm(range(len(dataset)), desc=\"Processing Batches...\"):\n",
    "        model_a, model_b, prompt, label = dataset[idx]\n",
    "        prompt = test_query_embeddings[prompt].cuda()\n",
    "        win_rate = inf_model.predict_win_rate(model_a, model_b, prompt, noise=0.05)\n",
    "        win_rates.append(win_rate)\n",
    "        true_labels.append(label)\n",
    "        \n",
    "        model_prediction = \"model_a\" if win_rate >= threshold else \"model_b\"\n",
    "\n",
    "        if model_prediction == \"model_a\":\n",
    "            model_a_calls += 1\n",
    "            if label == 1:\n",
    "                true_positives += 1\n",
    "            else:\n",
    "                false_positives += 1\n",
    "        else:\n",
    "            model_b_calls += 1\n",
    "            if label == 0:\n",
    "                true_negatives += 1\n",
    "            else:\n",
    "                false_negatives += 1\n",
    "\n",
    "    total_samples = true_positives + true_negatives + false_positives + false_negatives\n",
    "    return (true_positives, true_negatives, false_positives, false_negatives, \n",
    "            total_samples, model_a_calls, model_b_calls, \n",
    "            np.array(win_rates), np.array(true_labels))\n",
    "\n",
    "test_query_embeddings = torch.tensor(np.load(f\"./output/embeddings/query_embeddings_test.npy\"))\n",
    "dataset = test_dataset\n",
    "\n",
    "results_df = pd.DataFrame()\n",
    "for spct, threshold in zip(threshold_results['strong_model_pct'], threshold_results['threshold']):\n",
    "    metrics = batch_process(dataset, threshold)\n",
    "    (true_positives, true_negatives, false_positives, false_negatives, \n",
    "     total_samples, model_a_calls, model_b_calls, win_rates, true_labels) = metrics\n",
    "    \n",
    "    # Accuracy\n",
    "    accuracy = (true_positives + true_negatives) / total_samples\n",
    "    \n",
    "    # Positive class metrics\n",
    "    precision_positive = np.nan if (true_positives + false_positives) == 0 else \\\n",
    "                        true_positives / (true_positives + false_positives)\n",
    "    recall_positive = np.nan if (true_positives + false_negatives) == 0 else \\\n",
    "                     true_positives / (true_positives + false_negatives)\n",
    "    \n",
    "    # Negative class metrics\n",
    "    precision_negative = np.nan if (true_negatives + false_positives) == 0 else \\\n",
    "                        true_negatives / (true_negatives + false_positives)\n",
    "    recall_negative = np.nan if (true_negatives + false_negatives) == 0 else \\\n",
    "                     true_negatives / (true_negatives + false_negatives)\n",
    "    \n",
    "    # F1 scores\n",
    "    f1_score_positive = np.nan if (precision_positive + recall_positive) == 0 else \\\n",
    "                       2 * (precision_positive * recall_positive) / (precision_positive + recall_positive)\n",
    "    f1_score_negative = np.nan if (precision_negative + recall_negative) == 0 else \\\n",
    "                       2 * (precision_negative * recall_negative) / (precision_negative + recall_negative)\n",
    "    \n",
    "    ratio_calls_a = model_a_calls / (model_a_calls + model_b_calls)\n",
    "    \n",
    "    confidence_mean = np.mean(win_rates)\n",
    "    confidence_std = np.std(win_rates)\n",
    "\n",
    "    result = pd.DataFrame([{\n",
    "        'Strong Model Pct': spct,\n",
    "        'Threshold': threshold,\n",
    "        'Calls Model A (%)': ratio_calls_a,\n",
    "        'Accuracy': accuracy,\n",
    "        'Precision Positive': precision_positive,\n",
    "        'Recall Positive': recall_positive,\n",
    "        'F1 Score Positive': f1_score_positive,\n",
    "        'Precision Negative': precision_negative,\n",
    "        'Recall Negative': recall_negative,\n",
    "        'F1 Score Negative': f1_score_negative,\n",
    "        'Confidence Mean': confidence_mean,\n",
    "        'Confidence Std': confidence_std,\n",
    "    }])\n",
    "\n",
    "    results_df = pd.concat([results_df, result], ignore_index=True)\n",
    "\n",
    "display(results_df)\n",
    "results_df.to_csv(f\"{proj_dir}/output/threshold_results.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = pd.read_csv(f\"{proj_dir}/output/threshold_results.csv\")\n",
    "\n",
    "plt.figure(figsize=(12, 8))\n",
    "\n",
    "# Plot F1 scores\n",
    "plt.subplot(2, 2, 1)\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['F1 Score Positive'], label='F1 Score Positive', marker='o')\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['F1 Score Negative'], label='F1 Score Negative', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('F1 Score')\n",
    "plt.title('F1 Score vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "\n",
    "# Plot Precision\n",
    "plt.subplot(2, 2, 2)\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['Precision Positive'], label='Precision Positive', marker='o')\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['Precision Negative'], label='Precision Negative', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('Precision')\n",
    "plt.title('Precision vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "\n",
    "# Plot Recall\n",
    "plt.subplot(2, 2, 3)\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['Recall Positive'], label='Recall Positive', marker='o')\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['Recall Negative'], label='Recall Negative', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('Recall')\n",
    "plt.title('Recall vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "\n",
    "# Plot Calls to Model A\n",
    "plt.subplot(2, 2, 4)\n",
    "plt.plot(results_df['Strong Model Pct'], results_df['Calls Model A (%)'], label='Calls Model A (%)', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('Calls Model A (%)')\n",
    "plt.title('Calls to Model A vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Benchmark**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_and_concatenate_csvs(directory, prefix):\n",
    "    csv_files = [f for f in os.listdir(directory) if f.startswith(prefix) and f.endswith('.csv')]\n",
    "    dfs = [pd.read_csv(os.path.join(directory, f)) for f in csv_files]\n",
    "    concatenated_df = pd.concat(dfs, ignore_index=True)\n",
    "    concatenated_df.rename(columns={\"mistralai/Mixtral-8x7B-Instruct-v0.1\": \"mixtral-8x7b-instruct-v0.1\"}, inplace=True)\n",
    "    return concatenated_df\n",
    "\n",
    "# Define benchmark datasets here. Format must adhere to that of the below csvs.\n",
    "mmlu_df = load_and_concatenate_csvs('./evals/gemma2-llama3.2/', 'samples_mmlu')\n",
    "gsm8k_df = load_and_concatenate_csvs('./evals/gemma2-llama3.2/', 'samples_gsm8k')\n",
    "\n",
    "class EvalDataset(Dataset):\n",
    "    def __init__(self, df):\n",
    "        self.prompts = df['prompt'].tolist()\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.prompts)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.prompts[idx]\n",
    "\n",
    "def get_dataloader(df, batch_size=64):\n",
    "    dataset = EvalDataset(df)\n",
    "    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n",
    "    return dataloader\n",
    "\n",
    "mmlu_loader = get_dataloader(mmlu_df)\n",
    "gsm8k_loader = get_dataloader(gsm8k_df)\n",
    "\n",
    "strong_model_idx = MODEL_IDS[strong_model]\n",
    "weak_model_idx = MODEL_IDS[weak_model]\n",
    "\n",
    "def generate_win_rates(dataloader, cache_path):\n",
    "    if os.path.exists(cache_path):\n",
    "        print(f\"Prediction cache found at {cache_path}. Load existing cache? (y/n)\")\n",
    "        choice = input(\"Use precomputed cache? (y/n) : \").lower()\n",
    "        if choice == 'y':\n",
    "            print(f\"Loading caches from {cache_path}...\")\n",
    "            return np.load(cache_path).tolist()\n",
    "    \n",
    "    win_rates = []\n",
    "    with torch.no_grad():\n",
    "        for prompts in tqdm(dataloader, desc=\"Generating Win Rates...\"):\n",
    "            prompt_embeddings = embedding_model.encode(prompts, convert_to_tensor=True, normalize_embeddings=True).cuda()\n",
    "            for prompt_embedding in prompt_embeddings:\n",
    "                win_rate = inf_model.predict_win_rate(strong_model_idx, weak_model_idx, prompt_embedding, noise=0.05)\n",
    "                win_rates.append(win_rate)\n",
    "    \n",
    "    print(f\"Caching win rates to {cache_path}...\")\n",
    "    np.save(cache_path, win_rates)\n",
    "    return win_rates\n",
    "\n",
    "mmlu_cache_path = f\"{proj_dir}/output/mmlu_win_rates.npy\"\n",
    "gsm8k_cache_path = f\"{proj_dir}/output/gsm8k_win_rates.npy\"\n",
    "\n",
    "mmlu_win_rates = generate_win_rates(mmlu_loader, mmlu_cache_path)\n",
    "gsm8k_win_rates = generate_win_rates(gsm8k_loader, gsm8k_cache_path)\n",
    "np.save(mmlu_cache_path, mmlu_win_rates)\n",
    "np.save(gsm8k_cache_path, gsm8k_win_rates)\n",
    "\n",
    "def calculate_thresholds(win_rates, strong_model_pcts):\n",
    "    thresholds = [calculate_and_plot_threshold(pd.Series(win_rates), pct) for pct in strong_model_pcts]\n",
    "    return thresholds\n",
    "\n",
    "strong_model_pcts = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]\n",
    "mmlu_thresholds = calculate_thresholds(mmlu_win_rates, strong_model_pcts)\n",
    "gsm8k_thresholds = calculate_thresholds(gsm8k_win_rates, strong_model_pcts)\n",
    "\n",
    "# Make routing decisions based on the calculated win rates and the selected threshold\n",
    "def make_routing_decisions(win_rates, thresholds, strong_model_pct):\n",
    "    threshold = thresholds[strong_model_pcts.index(strong_model_pct)]\n",
    "    routing_decisions = [strong_model if win_rate >= threshold else weak_model for win_rate in win_rates]\n",
    "    return routing_decisions\n",
    "\n",
    "# Compare the routing decisions against the ground truth labels\n",
    "def compare_routing_decisions(df, routing_decisions):\n",
    "    correct_decisions = 0\n",
    "    for i, decision in enumerate(routing_decisions):\n",
    "        if decision == strong_model:\n",
    "            correct_decisions += df.iloc[i][strong_model]\n",
    "        else:\n",
    "            correct_decisions += df.iloc[i][weak_model]\n",
    "    return correct_decisions / len(routing_decisions)\n",
    "\n",
    "results = []\n",
    "for spct, threshold in zip(strong_model_pcts, mmlu_thresholds):\n",
    "    mmlu_routing_decisions = make_routing_decisions(mmlu_win_rates, mmlu_thresholds, spct)\n",
    "    gsm8k_routing_decisions = make_routing_decisions(gsm8k_win_rates, gsm8k_thresholds, spct)\n",
    "\n",
    "    mmlu_strong_calls_pct = mmlu_routing_decisions.count(strong_model) / len(mmlu_routing_decisions) * 100\n",
    "    gsm8k_strong_calls_pct = gsm8k_routing_decisions.count(strong_model) / len(gsm8k_routing_decisions) * 100\n",
    "    \n",
    "    mmlu_correct_decisions = compare_routing_decisions(mmlu_df, mmlu_routing_decisions) * 100\n",
    "    gsm8k_correct_decisions = compare_routing_decisions(gsm8k_df, gsm8k_routing_decisions) * 100\n",
    "\n",
    "    results.append({\n",
    "        'Strong Model Pct': spct,\n",
    "        'MMLU': mmlu_correct_decisions,\n",
    "        'MMLU Strong Calls (%)': mmlu_strong_calls_pct,\n",
    "        'GSM8K': gsm8k_correct_decisions,\n",
    "        'GSM8K Strong Calls (%)': gsm8k_strong_calls_pct,\n",
    "    })\n",
    "\n",
    "results_df = pd.DataFrame(results)\n",
    "display(results_df)\n",
    "results_df.to_csv(f\"{proj_dir}/output/benchmark_results.csv\", index=False)\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(results_df['MMLU Strong Calls (%)']/100, results_df['MMLU'], label='Scores-MMLU', marker='o')\n",
    "plt.plot(results_df['GSM8K Strong Calls (%)']/100, results_df['GSM8K'], label='Scores-GSM8K', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('Scores')\n",
    "plt.title('Benchmark Scores vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Benchmark scores at Strong Model PCT 0 correspond to the scores of the Weaker Model\n",
    "- Benchmark scores at Strong Model PCT 1 correspond to the scores of the Stronger Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cost-Performance Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def auc_metric(results_df, benchmark_name): # AUC calculation\n",
    "    auc = np.trapezoid(results_df[f\"{benchmark_name}\"]/100, results_df[f\"{benchmark_name} Strong Calls (%)\"]/100)\n",
    "    return auc\n",
    "\n",
    "def pgr_metric(score, weak_score, strong_score): # PGR calculation\n",
    "    return (score - weak_score) / (strong_score - weak_score)\n",
    "\n",
    "def apgr_metric(pgr_df, results_df, benchmark_name): # APGR calculation\n",
    "    apgr = np.trapezoid(pgr_df[f\"{benchmark_name} PGR\"], results_df[f\"{benchmark_name} Strong Calls (%)\"]/100)\n",
    "    return apgr\n",
    "\n",
    "def pct_metric(pgr_df, benchmark_name, pgr): # CPT calculation @ different PGR values\n",
    "    weak_model_accuracy = results_df.loc[results_df['Strong Model Pct'] == 0.0, benchmark_name].values[0]\n",
    "    strong_model_accuracy = results_df.loc[results_df['Strong Model Pct'] == 1.0, benchmark_name].values[0]\n",
    "    target_performance = pgr * (strong_model_accuracy - weak_model_accuracy) + weak_model_accuracy\n",
    "    cpt = np.interp(target_performance, results_df[benchmark_name], results_df[f\"{benchmark_name} Strong Calls (%)\"])\n",
    "    return cpt\n",
    "\n",
    "# PGR Calculation\n",
    "mmlu_weak_score = results_df.loc[results_df['Strong Model Pct'] == 0.0, 'MMLU'].values[0]\n",
    "mmlu_strong_score = results_df.loc[results_df['Strong Model Pct'] == 1.0, 'MMLU'].values[0]\n",
    "\n",
    "gsm8k_weak_score = results_df.loc[results_df['Strong Model Pct'] == 0.0, 'GSM8K'].values[0]\n",
    "gsm8k_strong_score = results_df.loc[results_df['Strong Model Pct'] == 1.0, 'GSM8K'].values[0]\n",
    "\n",
    "results_df['MMLU PGR'] = results_df['MMLU'].apply(pgr_metric, args=(mmlu_weak_score, mmlu_strong_score))\n",
    "results_df['GSM8K PGR'] = results_df['GSM8K'].apply(pgr_metric, args=(gsm8k_weak_score, gsm8k_strong_score))\n",
    "\n",
    "pgr_df = results_df[['Strong Model Pct', 'MMLU PGR', 'GSM8K PGR']]\n",
    "display(pgr_df)\n",
    "pgr_df.to_csv(f\"{proj_dir}/output/pgr_results.csv\", index=False)\n",
    "\n",
    "metrics_df = pd.DataFrame({\n",
    "    'Benchmark': ['MMLU', 'GSM8K'],\n",
    "    'AUC': [auc_metric(results_df, 'MMLU'), auc_metric(results_df, 'GSM8K')],\n",
    "    'APGR': [apgr_metric(pgr_df, results_df, 'MMLU'), apgr_metric(pgr_df, results_df, 'GSM8K')],\n",
    "    'CPT @ 20% PGain': [pct_metric(pgr_df, 'MMLU', 0.2), pct_metric(pgr_df, 'GSM8K', 0.2)],\n",
    "    'CPT @ 50% PGain': [pct_metric(pgr_df, 'MMLU', 0.5), pct_metric(pgr_df, 'GSM8K', 0.5)],\n",
    "    'CPT @ 80% PGain': [pct_metric(pgr_df, 'MMLU', 0.8), pct_metric(pgr_df, 'GSM8K', 0.8)],\n",
    "})\n",
    "\n",
    "display(metrics_df)\n",
    "metrics_df.to_csv(f\"{proj_dir}/output/metrics_results.csv\", index=False)\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(pgr_df['Strong Model Pct'], pgr_df['MMLU PGR'], label='PGR-MMLU', marker='o')\n",
    "plt.plot(pgr_df['Strong Model Pct'], pgr_df['GSM8K PGR'], label='PGR-GSM8K', marker='o')\n",
    "plt.xlabel('Strong Model PCT')\n",
    "plt.ylabel('PGR')\n",
    "plt.title('PGR vs Strong Model PCT')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### APGR\n",
    "\n",
    "APGR (Average Performance Gap Recovered) is a metric used to assess the effectiveness of LLM routers. It quantifies how well a router can bridge the performance difference between a weaker, more cost-effective LLM and a stronger, more expensive one. The performance gap is the difference in quality between the outputs of a strong LLM (like GPT-4) and a weak LLM/SLM (like Mixtral-8x7B). This gap is usually measured using benchmark datasets or human evaluations. Routers aim to \"recover\" this performance gap by intelligently deciding which LLM to use for a given query. Ideally, they'd route simple queries to the weak model and complex ones to the strong model, maximizing quality while minimizing cost. APGR measures the router's ability to recover this performance gap across a range of cost constraints. It's calculated as the area under the router's performance curve (plotting quality against the percentage of calls to the strong model). A higher APGR value indicates that the router is more effective at selecting the appropriate LLM for different types of queries. This means it can achieve a good balance between cost-efficiency and response quality.\n",
    "\n",
    "### AUC\n",
    "\n",
    "AUC is calculated as the Area Under the Call-Performance Curve.\n",
    "\n",
    "- AUC at 1: Implies maximum routing performance.\n",
    "- AUC close to 1:  Suggests the router effectively routes most queries to the appropriate model (strong or weak), achieving near-optimal performance.\n",
    "- AUC close to 0.5:  Indicates the router performs similarly to a random routing strategy, showing limited effectiveness in differentiating between query types.\n",
    "- AUC below 0.5: Implies the router might be making poor routing decisions, potentially performing worse than random chance.\n",
    "- AUC at 0: Lowest tier performance across all samples."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "budmfr",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "002177b896964f0aaa7ecc600b0513cb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7194332781924e1d8ebc58d8f7f16ef6",
      "placeholder": "​",
      "style": "IPY_MODEL_46471099f8e4475abf1cfd9420329774",
      "value": " 571/571 [00:00&lt;00:00, 35.0kB/s]"
     }
    },
    "0190da56d56c4bffb69d81f88cb15f62": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2b9fe30db46c4b4aa6eb104c686b6a6b",
      "placeholder": "​",
      "style": "IPY_MODEL_27201685327c46839ba700de1d6d1363",
      "value": " 109101/109101 [00:12&lt;00:00, 17066.87 examples/s]"
     }
    },
    "01d3d02837b54a41bc6968b72419ca5c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a8ec09d5a9844e4095df6b52acf0b3d9",
      "max": 116,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_06af4b6c4c65401f8d2c0a7103d86a4d",
      "value": 116
     }
    },
    "03895a74977445de90877ea1bc87b091": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "03f298f8ab4e4453b3367f8fd3fa5025": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "062095703c844bf69ce028e18c25d28a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "06af4b6c4c65401f8d2c0a7103d86a4d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "086df64dbc1e4c61af9c18504a9e946c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0abd74af64fd435086157feb4ca3d8c4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_578a592a3cbe433f9a0d902c017f098d",
      "max": 437971872,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2f06de91cfc64f44a6bc0d1c0bc3fe29",
      "value": 437971872
     }
    },
    "0f5bb5f6771d47e3be34ec1a6f1ffa01": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dc510ba5fab44c09a12a6688f4e036cc",
      "placeholder": "​",
      "style": "IPY_MODEL_fda5c07ad06c48999d0728c2ee3695b7",
      "value": "vocab.txt: 100%"
     }
    },
    "0fa9c417f6a94bf8a19a4c621a923783": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "109589570fe148c9b239a9be08ce3b94": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "10e250cc302446148c509483727c4979": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b56a83eafc744ef2b6e5c2e502ae41b8",
      "max": 571,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_29ebfce9e15341d382a0c4f24b585360",
      "value": 571
     }
    },
    "136e258719e24276b2830cc35d7b3a6f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "161e0c7fd134456987f9aa7161d39492": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8da4332f1e8849a69da341a3099d0159",
      "placeholder": "​",
      "style": "IPY_MODEL_e1a7384b05f346d3946d3451e5a2c419",
      "value": "special_tokens_map.json: 100%"
     }
    },
    "1643c63b91f2466192584e7e336c3b06": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_062095703c844bf69ce028e18c25d28a",
      "placeholder": "​",
      "style": "IPY_MODEL_9f170d452d284be08198702e83585129",
      "value": " 363/363 [00:00&lt;00:00, 12.0kB/s]"
     }
    },
    "1b36afaa063840dda602e59e7566f0f0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b5331fd4b4cb417fb2f6cff7f904aff6",
      "max": 167184802,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b0a595f63c9f49d59e23d325818993ba",
      "value": 167184802
     }
    },
    "1d9a6ee2a81c428ebb7bf594915571f7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a80cb99b9fba471a833777125e89a24d",
       "IPY_MODEL_91cff64ff62e435198564a9b67ccd8a2",
       "IPY_MODEL_eec50b4d836c4dce9623dc7173220cfa"
      ],
      "layout": "IPY_MODEL_674c3e28440c421b83be2f9196194ced"
     }
    },
    "1e4eea0a5d56428c8392403eb7e7dbd8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "22784f5721524de9839ca5b6f197f56f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d74c0acdb9a542649c646b817fdec9f6",
      "placeholder": "​",
      "style": "IPY_MODEL_944c71c7614d49709ece2c08f63a7bdb",
      "value": "README.md: 100%"
     }
    },
    "230a5621b8d74eb39a5248a57a6f89d5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "27201685327c46839ba700de1d6d1363": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "27b373cac40d43938c5f775b328416ed": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "29ebfce9e15341d382a0c4f24b585360": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2a82d3857bab4113ba7ee32016c63791": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5594ce7d34dc4955a9887be820f3bced",
      "placeholder": "​",
      "style": "IPY_MODEL_b49d4825ba60437ca375b3bf869ee0c4",
      "value": "Generating train split: 100%"
     }
    },
    "2b12b1093641443897b04be5e6b8195f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_27b373cac40d43938c5f775b328416ed",
      "placeholder": "​",
      "style": "IPY_MODEL_f10c9558c04f49c2b9430437056ac3c0",
      "value": "model.safetensors: 100%"
     }
    },
    "2b9fe30db46c4b4aa6eb104c686b6a6b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2eb27638df5e4430be8f5abc791c9180": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2f06de91cfc64f44a6bc0d1c0bc3fe29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "325057d75312412b96f084601ee1b2df": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "350bac7528bc4265a215024a38491777": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "35f95e7e3f9247618d2ee30fefbb8284": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "38e26bdc72b743e685458e34a16bca22": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c71f4935dd4a43c98f15c4f7d00bd26d",
      "placeholder": "​",
      "style": "IPY_MODEL_fcd0257480894422832b1c90f29baf4e",
      "value": "1_Pooling/config.json: 100%"
     }
    },
    "3c65f4baee354488be5284aa7f4a3164": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4091315db1924ed987ad5f14513fdeef": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "42f0d4f204e946e6a0fd2e5575411859": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4551d96223f04229bda5c29cc661f16f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "46471099f8e4475abf1cfd9420329774": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4723d473e946430f96456c6152b93462": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4ac23602151c4bf28c8f417e0945bc0a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4bdcd5c15eeb4520a5b85db5f3461106": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4c2ba62c4f8a4bfc94969e03aa646fd8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d54d2e646964dada7c741d0731c7b43": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f9a78ce553954c8eaff8c54b1bdea197",
      "placeholder": "​",
      "style": "IPY_MODEL_db4fd2b9f65a4b96bbdf2d8fc806281b",
      "value": " 466k/466k [00:00&lt;00:00, 899kB/s]"
     }
    },
    "4dd260814cd14dabafc87da41f391ecf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2a82d3857bab4113ba7ee32016c63791",
       "IPY_MODEL_e3c9eb62e84c47ec8e57577afb9678ab",
       "IPY_MODEL_0190da56d56c4bffb69d81f88cb15f62"
      ],
      "layout": "IPY_MODEL_96d8650ef9a24a168306998473ea23a4"
     }
    },
    "5594ce7d34dc4955a9887be820f3bced": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "55ae3912f5674dfb88634511f19367ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5e25188e481044508e24f5d868ed17c0",
       "IPY_MODEL_e36ee90305cd40a485f4a512ad666d96",
       "IPY_MODEL_4d54d2e646964dada7c741d0731c7b43"
      ],
      "layout": "IPY_MODEL_fc5005bfa20340c4a95b0495ec0d070e"
     }
    },
    "55d19c28064a481c9637d305735ccf4a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fda0389270064beca44370713527b062",
      "placeholder": "​",
      "style": "IPY_MODEL_e61e4a2733ad49a58f24556ea9c061ce",
      "value": " 190/190 [00:00&lt;00:00, 9.52kB/s]"
     }
    },
    "578a592a3cbe433f9a0d902c017f098d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5813308d828347af8ef8c397165d2870": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5ca1c5cd0b674f4fb7c8f82d8ab42a49": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5e25188e481044508e24f5d868ed17c0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b6748055f7f44df39788108725b5d0c4",
      "placeholder": "​",
      "style": "IPY_MODEL_fa7efd738b7a44a4887def399bded0b9",
      "value": "tokenizer.json: 100%"
     }
    },
    "5f13d45684054d2f9e76e7860a428ccb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_086df64dbc1e4c61af9c18504a9e946c",
      "max": 586,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_03f298f8ab4e4453b3367f8fd3fa5025",
      "value": 586
     }
    },
    "5f7fb54166c34fe9b6ed6368a4efe7dd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "606da97e6c4e4dea85d63a9e96445652": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6318f5c684584e3bb1a043dc1561c236": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ca494cdb14164e3294d61f99bf9b050f",
       "IPY_MODEL_735d560d26df43edaa5ec37342e9f71a",
       "IPY_MODEL_b6c82f17f0504e68b9abb2887c9a9a0b"
      ],
      "layout": "IPY_MODEL_5ca1c5cd0b674f4fb7c8f82d8ab42a49"
     }
    },
    "674c3e28440c421b83be2f9196194ced": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "69a0adcf50564fad84413c4cd7b2f886": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c570df5bb9194826ab0c3d65940c928e",
       "IPY_MODEL_01d3d02837b54a41bc6968b72419ca5c",
       "IPY_MODEL_a473ae7ad8214c75b0f34c6b59455a24"
      ],
      "layout": "IPY_MODEL_d0156871b3c34f0c83aeb538445fa847"
     }
    },
    "6c610f721471462191828c261f168125": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0fa9c417f6a94bf8a19a4c621a923783",
      "max": 239,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_03895a74977445de90877ea1bc87b091",
      "value": 239
     }
    },
    "6ef5fbf2f83d43bf8b90f29a7a16389c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "714f809016ea460ea9b27867f2b2d7e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7194332781924e1d8ebc58d8f7f16ef6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "735d560d26df43edaa5ec37342e9f71a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3c65f4baee354488be5284aa7f4a3164",
      "max": 10621,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6ef5fbf2f83d43bf8b90f29a7a16389c",
      "value": 10621
     }
    },
    "736c17b38a0144cbbd586c24436b4058": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "74a88516d85640a6ab0f0799b8f84eaa": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "750ba152f40a4c2291c81da14ad4e364": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f059007fccce41f4a4f1a487fd25d1d4",
      "placeholder": "​",
      "style": "IPY_MODEL_109589570fe148c9b239a9be08ce3b94",
      "value": " 438M/438M [00:02&lt;00:00, 181MB/s]"
     }
    },
    "7ad3114669b04abfa71ee6edbb539df8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7d18d244a3ce416dbbb5ccf8f34b2c95": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_93e0eed03a1948efbe87d40496d8c876",
       "IPY_MODEL_10e250cc302446148c509483727c4979",
       "IPY_MODEL_002177b896964f0aaa7ecc600b0513cb"
      ],
      "layout": "IPY_MODEL_874565ca9f454f5091cb5105a46896ec"
     }
    },
    "7ef4a7aa9b3c4f5094170f33a4e5e3ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7f1a490e019e4da79d09865e18d8c204": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8317152662a24eed95910639362c15be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c2a4ecebcb6540de991fd60a24745750",
      "placeholder": "​",
      "style": "IPY_MODEL_5813308d828347af8ef8c397165d2870",
      "value": " 586/586 [00:00&lt;00:00, 31.5kB/s]"
     }
    },
    "8358c52188f8457b8151db6ac5c5482a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "84cc8ceebea74c8d91977b1754fde2ff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "84e7a06d41054098a5aa91669e982af0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "874565ca9f454f5091cb5105a46896ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "876c999bfc17412ab5b66cd54ebc1d65": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8cd02cf221cb442091beb51ed9b7a09f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8da4332f1e8849a69da341a3099d0159": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "91cff64ff62e435198564a9b67ccd8a2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c49a64a016354f9bbb2bb96e458a2d67",
      "max": 53,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_350bac7528bc4265a215024a38491777",
      "value": 53
     }
    },
    "92b3cb67c61c4fa5855a9de91ba0cbc6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "93e0eed03a1948efbe87d40496d8c876": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c1a8140c53c74f92ae74cf10fc8fd36d",
      "placeholder": "​",
      "style": "IPY_MODEL_35f95e7e3f9247618d2ee30fefbb8284",
      "value": "config.json: 100%"
     }
    },
    "93f13fa677ad4b0db8f6b1827f60870e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "944c71c7614d49709ece2c08f63a7bdb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "96d8650ef9a24a168306998473ea23a4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "98d06452c1c94e13b069510cdeef1752": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9ad16a62d8214b26a4f4754777bf2ea4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b18c9d61a8134f1783638c5c1fe901d0",
      "placeholder": "​",
      "style": "IPY_MODEL_c0308f41ca194c9b9fead599d71bb245",
      "value": " 167M/167M [00:04&lt;00:00, 42.6MB/s]"
     }
    },
    "9c08aed11ee14300b2cea357dcada841": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9d212ae530ed47889b506bdcb49c5a19": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9e672a44264045c98cfcb0ee84e9b728": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9f0c3e71290743a0af51d2ea2c1bcf07": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9f170d452d284be08198702e83585129": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a473ae7ad8214c75b0f34c6b59455a24": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_93f13fa677ad4b0db8f6b1827f60870e",
      "placeholder": "​",
      "style": "IPY_MODEL_4091315db1924ed987ad5f14513fdeef",
      "value": " 116/116 [00:00&lt;00:00, 6.53kB/s]"
     }
    },
    "a80cb99b9fba471a833777125e89a24d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dba66e67542442f18a79221582909f87",
      "placeholder": "​",
      "style": "IPY_MODEL_876c999bfc17412ab5b66cd54ebc1d65",
      "value": "sentence_bert_config.json: 100%"
     }
    },
    "a8ec09d5a9844e4095df6b52acf0b3d9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ab44d8e10a8441b7b59ccd780ed8889f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "af746483961b403988e0310ea2a9cd02": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_161e0c7fd134456987f9aa7161d39492",
       "IPY_MODEL_6c610f721471462191828c261f168125",
       "IPY_MODEL_d4f1d59d1aa943c8994df940d54fe1b2"
      ],
      "layout": "IPY_MODEL_4551d96223f04229bda5c29cc661f16f"
     }
    },
    "b0a595f63c9f49d59e23d325818993ba": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b0c95c854b93455696b44acba6736661": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b18c9d61a8134f1783638c5c1fe901d0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b49d4825ba60437ca375b3bf869ee0c4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b4aa5b3a9c184162ac280bf44c71713b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b5331fd4b4cb417fb2f6cff7f904aff6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b56a83eafc744ef2b6e5c2e502ae41b8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b63828704fb642dea1e2f9c85ac04724": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b6748055f7f44df39788108725b5d0c4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b6c82f17f0504e68b9abb2887c9a9a0b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e67c6a592b4c4312994ef0945f713676",
      "placeholder": "​",
      "style": "IPY_MODEL_4ac23602151c4bf28c8f417e0945bc0a",
      "value": " 10.6k/10.6k [00:00&lt;00:00, 687kB/s]"
     }
    },
    "ba0495c43b0a4ccba393ca46e25df20c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_38e26bdc72b743e685458e34a16bca22",
       "IPY_MODEL_fda9685a928843febe081fbce28aa9b5",
       "IPY_MODEL_55d19c28064a481c9637d305735ccf4a"
      ],
      "layout": "IPY_MODEL_1e4eea0a5d56428c8392403eb7e7dbd8"
     }
    },
    "ba15b41674964896bd8b25d4b0b2f91d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_98d06452c1c94e13b069510cdeef1752",
      "placeholder": "​",
      "style": "IPY_MODEL_d9b4981f866147e6bb333392e0840cbf",
      "value": "train-00000-of-00001.parquet: 100%"
     }
    },
    "c0308f41ca194c9b9fead599d71bb245": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c16bb88c58c9449aa45deb43f4454db7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d9f4f6509ea64f398adcec5c3edc4be1",
       "IPY_MODEL_fc89b3edf24f47db8f06c58f2440a698",
       "IPY_MODEL_cae9b189333841249f17537a61326ac1"
      ],
      "layout": "IPY_MODEL_7ad3114669b04abfa71ee6edbb539df8"
     }
    },
    "c1a8140c53c74f92ae74cf10fc8fd36d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c2a4ecebcb6540de991fd60a24745750": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c49a64a016354f9bbb2bb96e458a2d67": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c570df5bb9194826ab0c3d65940c928e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2eb27638df5e4430be8f5abc791c9180",
      "placeholder": "​",
      "style": "IPY_MODEL_9c08aed11ee14300b2cea357dcada841",
      "value": "config_sentence_transformers.json: 100%"
     }
    },
    "c71f4935dd4a43c98f15c4f7d00bd26d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ca494cdb14164e3294d61f99bf9b050f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_84cc8ceebea74c8d91977b1754fde2ff",
      "placeholder": "​",
      "style": "IPY_MODEL_8cd02cf221cb442091beb51ed9b7a09f",
      "value": "README.md: 100%"
     }
    },
    "cae9b189333841249f17537a61326ac1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f96524a1483941808baa3189122855aa",
      "placeholder": "​",
      "style": "IPY_MODEL_ab44d8e10a8441b7b59ccd780ed8889f",
      "value": " 349/349 [00:00&lt;00:00, 14.0kB/s]"
     }
    },
    "cc6eaddd88e04f88b5ce68501f2e33ed": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cfe7f87ad6504f9a972509e4db5ad536": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b0c95c854b93455696b44acba6736661",
      "max": 231536,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f0f69756be25457080d4501069884578",
      "value": 231536
     }
    },
    "d0156871b3c34f0c83aeb538445fa847": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d20b4ae836694dd8942909af1b00ddaa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_92b3cb67c61c4fa5855a9de91ba0cbc6",
      "placeholder": "​",
      "style": "IPY_MODEL_7ef4a7aa9b3c4f5094170f33a4e5e3ad",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "d4f1d59d1aa943c8994df940d54fe1b2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_230a5621b8d74eb39a5248a57a6f89d5",
      "placeholder": "​",
      "style": "IPY_MODEL_714f809016ea460ea9b27867f2b2d7e7",
      "value": " 239/239 [00:00&lt;00:00, 11.0kB/s]"
     }
    },
    "d5eb5aa17bd24b1fb71a952382280b38": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_22784f5721524de9839ca5b6f197f56f",
       "IPY_MODEL_5f13d45684054d2f9e76e7860a428ccb",
       "IPY_MODEL_8317152662a24eed95910639362c15be"
      ],
      "layout": "IPY_MODEL_84e7a06d41054098a5aa91669e982af0"
     }
    },
    "d74c0acdb9a542649c646b817fdec9f6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d9b4981f866147e6bb333392e0840cbf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d9f4f6509ea64f398adcec5c3edc4be1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cc6eaddd88e04f88b5ce68501f2e33ed",
      "placeholder": "​",
      "style": "IPY_MODEL_136e258719e24276b2830cc35d7b3a6f",
      "value": "modules.json: 100%"
     }
    },
    "db4fd2b9f65a4b96bbdf2d8fc806281b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dba66e67542442f18a79221582909f87": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dc510ba5fab44c09a12a6688f4e036cc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "de76a6d133fb467382b609f91e6ad3f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "e15f7df8c2e6488c91acbd407faa18fe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0f5bb5f6771d47e3be34ec1a6f1ffa01",
       "IPY_MODEL_cfe7f87ad6504f9a972509e4db5ad536",
       "IPY_MODEL_f3d2e3530097404aab643e496c2c7452"
      ],
      "layout": "IPY_MODEL_9e672a44264045c98cfcb0ee84e9b728"
     }
    },
    "e1a7384b05f346d3946d3451e5a2c419": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e36ee90305cd40a485f4a512ad666d96": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_42f0d4f204e946e6a0fd2e5575411859",
      "max": 466021,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4723d473e946430f96456c6152b93462",
      "value": 466021
     }
    },
    "e3c9eb62e84c47ec8e57577afb9678ab": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4bdcd5c15eeb4520a5b85db5f3461106",
      "max": 109101,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9f0c3e71290743a0af51d2ea2c1bcf07",
      "value": 109101
     }
    },
    "e61e4a2733ad49a58f24556ea9c061ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e67c6a592b4c4312994ef0945f713676": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e94e389b54644b3ea8689525bfc8c399": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ebc9c81e29f74b7f906c9d2d77339be3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ba15b41674964896bd8b25d4b0b2f91d",
       "IPY_MODEL_1b36afaa063840dda602e59e7566f0f0",
       "IPY_MODEL_9ad16a62d8214b26a4f4754777bf2ea4"
      ],
      "layout": "IPY_MODEL_8358c52188f8457b8151db6ac5c5482a"
     }
    },
    "eec50b4d836c4dce9623dc7173220cfa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_325057d75312412b96f084601ee1b2df",
      "placeholder": "​",
      "style": "IPY_MODEL_7f1a490e019e4da79d09865e18d8c204",
      "value": " 53.0/53.0 [00:00&lt;00:00, 2.84kB/s]"
     }
    },
    "f059007fccce41f4a4f1a487fd25d1d4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f0c5f6d0056843ff82c4f2328f578959": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2b12b1093641443897b04be5e6b8195f",
       "IPY_MODEL_0abd74af64fd435086157feb4ca3d8c4",
       "IPY_MODEL_750ba152f40a4c2291c81da14ad4e364"
      ],
      "layout": "IPY_MODEL_e94e389b54644b3ea8689525bfc8c399"
     }
    },
    "f0f69756be25457080d4501069884578": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f10c9558c04f49c2b9430437056ac3c0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f1de0ba8b5c44238a6ff42b9ca244297": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d20b4ae836694dd8942909af1b00ddaa",
       "IPY_MODEL_f29bd005bd644961b47a48ef3cd026dd",
       "IPY_MODEL_1643c63b91f2466192584e7e336c3b06"
      ],
      "layout": "IPY_MODEL_4c2ba62c4f8a4bfc94969e03aa646fd8"
     }
    },
    "f29bd005bd644961b47a48ef3cd026dd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9d212ae530ed47889b506bdcb49c5a19",
      "max": 363,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_de76a6d133fb467382b609f91e6ad3f3",
      "value": 363
     }
    },
    "f3d2e3530097404aab643e496c2c7452": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_606da97e6c4e4dea85d63a9e96445652",
      "placeholder": "​",
      "style": "IPY_MODEL_b63828704fb642dea1e2f9c85ac04724",
      "value": " 232k/232k [00:00&lt;00:00, 673kB/s]"
     }
    },
    "f96524a1483941808baa3189122855aa": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f9a78ce553954c8eaff8c54b1bdea197": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fa7efd738b7a44a4887def399bded0b9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fc5005bfa20340c4a95b0495ec0d070e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fc89b3edf24f47db8f06c58f2440a698": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_74a88516d85640a6ab0f0799b8f84eaa",
      "max": 349,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_736c17b38a0144cbbd586c24436b4058",
      "value": 349
     }
    },
    "fcd0257480894422832b1c90f29baf4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fda0389270064beca44370713527b062": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fda5c07ad06c48999d0728c2ee3695b7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fda9685a928843febe081fbce28aa9b5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5f7fb54166c34fe9b6ed6368a4efe7dd",
      "max": 190,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b4aa5b3a9c184162ac280bf44c71713b",
      "value": 190
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}