imdb_cleaning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOTM5QhTjSIKCUzGr834Q+V",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Karem1986/LetsPython/blob/main/imdb_cleaning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data cleaning-Duplicates verwijderen\n",
        "\n",
        "In deze exercise ga je werken met een filmdataset van IMDb. Soms komt het voor in een dataset dat een datapunt door een fout twee keer of zelfs meerdere keren voorkomt. Dit geeft een verkeerd beeld van de werkelijkheid en daarom wil je deze duplicates verwijderen."
      ],
      "metadata": {
        "id": "ZydPx2ZcYj3R"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## File upload\n",
        "door het importeren van google colab files, is het mogelijk om een file in de notebook te importeren."
      ],
      "metadata": {
        "id": "XSlndGLyZL2z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/gdrive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dm4hmzgTZM4F",
        "outputId": "0bbc4da7-e034-4ae7-cc93-f97ea4d0d57e"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### CSV bestaand lezen naar een dataframe\n",
        "\n",
        "Laad het bestand in een Pandas dataframe. Zet het argument index_col op 'index'."
      ],
      "metadata": {
        "id": "lFHwAK4_ZTU9"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "JXzC1tVeYdZD"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Load the dataset into a pandas dataframe with index column set to 'index'\n",
        "df = pd.read_csv('gdrive/My Drive/Colab Notebooks/IMDB_films.csv', index_col='index')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Het aantal rijen\n",
        "\n",
        "Print het aantal rijen. Hoeveel datapunten heb je?"
      ],
      "metadata": {
        "id": "mdSMBxm7coIm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "original_rows = len(df)\n",
        "print(f\"Oorspronkelijke datapunten:: {original_rows}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T_cMe7fFcrF2",
        "outputId": "54189786-4a6b-440a-aebe-7cd5e7b6a970"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Oorspronkelijke datapunten:: 21782\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Verwijder dubbele rijen.\n",
        "\n",
        "Met de pandas functie drop_duplicates."
      ],
      "metadata": {
        "id": "HfQNCln-cwG1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Remove duplicate rows\n",
        "df = df.drop_duplicates()\n",
        "\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "N6YSjmj2c7AA",
        "outputId": "fef5bf76-97ab-4523-dfca-9453ad4de02d"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                            title  year date_published    genre  duration  \\\n",
              "index                                                                       \n",
              "0                      Miss Jerry  1894     1894-10-09  Romance        45   \n",
              "2                  Den sorte drøm  1911     1911-08-19    Drama        53   \n",
              "9                     Richard III  1912     1912-10-15    Drama        55   \n",
              "10                       Atlantis  1913     1913-12-26    Drama       121   \n",
              "12       Il calvario di una madre  1913     1915-10-18    Drama        96   \n",
              "...                           ...   ...            ...      ...       ...   \n",
              "85847                       Ottam  2019     2019-03-08    Drama       120   \n",
              "85848                   Pengalila  2019     2019-03-08    Drama       111   \n",
              "85850                     Le lion  2020     2020-01-29   Comedy        95   \n",
              "85852   Padmavyuhathile Abhimanyu  2019     2019-03-08    Drama       130   \n",
              "85854  La vida sense la Sara Amat  2019     2020-02-05    Drama        74   \n",
              "\n",
              "        language                                        description  \\\n",
              "index                                                                 \n",
              "0           None  The adventures of a female reporter in the 1890s.   \n",
              "2            NaN  Two men of high rank are both wooing the beaut...   \n",
              "9        English  Richard of Gloucester uses manipulation and mu...   \n",
              "10        Danish  After Dr. Friedrich's wife becomes mentally un...   \n",
              "12           NaN  Single mother is separated from her children d...   \n",
              "...          ...                                                ...   \n",
              "85847  Malayalam  Set in Trivandrum, the story of Ottam unfolds ...   \n",
              "85848  Malayalam  An unusual bond between a sixty year old Dalit...   \n",
              "85850     French  A psychiatric hospital patient pretends to be ...   \n",
              "85852  Malayalam                                                NaN   \n",
              "85854    Catalan  Pep, a 13-year-old boy, is in love with a girl...   \n",
              "\n",
              "       reviews_from_users  reviews_from_critics  \n",
              "index                                            \n",
              "0                     1.0                   2.0  \n",
              "2                     5.0                   2.0  \n",
              "9                     8.0                   1.0  \n",
              "10                    9.0                   9.0  \n",
              "12                   16.0                   7.0  \n",
              "...                   ...                   ...  \n",
              "85847                 1.0                   NaN  \n",
              "85848                 NaN                   NaN  \n",
              "85850                 NaN                   4.0  \n",
              "85852                 NaN                   NaN  \n",
              "85854                 NaN                   2.0  \n",
              "\n",
              "[21770 rows x 9 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-93122f14-8026-4ab3-9b77-f630597388cf\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>year</th>\n",
              "      <th>date_published</th>\n",
              "      <th>genre</th>\n",
              "      <th>duration</th>\n",
              "      <th>language</th>\n",
              "      <th>description</th>\n",
              "      <th>reviews_from_users</th>\n",
              "      <th>reviews_from_critics</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>index</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Miss Jerry</td>\n",
              "      <td>1894</td>\n",
              "      <td>1894-10-09</td>\n",
              "      <td>Romance</td>\n",
              "      <td>45</td>\n",
              "      <td>None</td>\n",
              "      <td>The adventures of a female reporter in the 1890s.</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Den sorte drøm</td>\n",
              "      <td>1911</td>\n",
              "      <td>1911-08-19</td>\n",
              "      <td>Drama</td>\n",
              "      <td>53</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Two men of high rank are both wooing the beaut...</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Richard III</td>\n",
              "      <td>1912</td>\n",
              "      <td>1912-10-15</td>\n",
              "      <td>Drama</td>\n",
              "      <td>55</td>\n",
              "      <td>English</td>\n",
              "      <td>Richard of Gloucester uses manipulation and mu...</td>\n",
              "      <td>8.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Atlantis</td>\n",
              "      <td>1913</td>\n",
              "      <td>1913-12-26</td>\n",
              "      <td>Drama</td>\n",
              "      <td>121</td>\n",
              "      <td>Danish</td>\n",
              "      <td>After Dr. Friedrich's wife becomes mentally un...</td>\n",
              "      <td>9.0</td>\n",
              "      <td>9.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Il calvario di una madre</td>\n",
              "      <td>1913</td>\n",
              "      <td>1915-10-18</td>\n",
              "      <td>Drama</td>\n",
              "      <td>96</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Single mother is separated from her children d...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>7.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85847</th>\n",
              "      <td>Ottam</td>\n",
              "      <td>2019</td>\n",
              "      <td>2019-03-08</td>\n",
              "      <td>Drama</td>\n",
              "      <td>120</td>\n",
              "      <td>Malayalam</td>\n",
              "      <td>Set in Trivandrum, the story of Ottam unfolds ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85848</th>\n",
              "      <td>Pengalila</td>\n",
              "      <td>2019</td>\n",
              "      <td>2019-03-08</td>\n",
              "      <td>Drama</td>\n",
              "      <td>111</td>\n",
              "      <td>Malayalam</td>\n",
              "      <td>An unusual bond between a sixty year old Dalit...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85850</th>\n",
              "      <td>Le lion</td>\n",
              "      <td>2020</td>\n",
              "      <td>2020-01-29</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>95</td>\n",
              "      <td>French</td>\n",
              "      <td>A psychiatric hospital patient pretends to be ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85852</th>\n",
              "      <td>Padmavyuhathile Abhimanyu</td>\n",
              "      <td>2019</td>\n",
              "      <td>2019-03-08</td>\n",
              "      <td>Drama</td>\n",
              "      <td>130</td>\n",
              "      <td>Malayalam</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85854</th>\n",
              "      <td>La vida sense la Sara Amat</td>\n",
              "      <td>2019</td>\n",
              "      <td>2020-02-05</td>\n",
              "      <td>Drama</td>\n",
              "      <td>74</td>\n",
              "      <td>Catalan</td>\n",
              "      <td>Pep, a 13-year-old boy, is in love with a girl...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>21770 rows × 9 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-93122f14-8026-4ab3-9b77-f630597388cf')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-93122f14-8026-4ab3-9b77-f630597388cf button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-93122f14-8026-4ab3-9b77-f630597388cf');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-446978e0-c17c-4f6a-83ab-f1223d756217\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-446978e0-c17c-4f6a-83ab-f1223d756217')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-446978e0-c17c-4f6a-83ab-f1223d756217 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Het aantal overgebleven datapunten\n",
        "\n",
        "Print het aantal overgebleven datapunten."
      ],
      "metadata": {
        "id": "1y0_uMPBdLug"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Print the number of remaining datapoints\n",
        "remaining_rows = len(df)\n",
        "print(f\"Remaining datapoints after removing duplicates: {remaining_rows}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "27MAkKrKdV2p",
        "outputId": "89fd18d3-bd59-4294-cd85-cb98c6579b33"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Remaining datapoints after removing duplicates: 21770\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hoeveel duplicates\n",
        "\n",
        "Hoeveel duplicates zaten er in de data?"
      ],
      "metadata": {
        "id": "6yVqXn5fdbac"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate the number of duplicates 21782 21770\n",
        "duplicates_removed = original_rows - remaining_rows\n",
        "print(f\"Duplicates: {duplicates_removed}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pT2HmL59d27F",
        "outputId": "103c4b35-aa24-402d-d7ae-64f778cde28f"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Duplicates: 12\n"
          ]
        }
      ]
    }
  ]
}