1. 자연어_처리(Natural_Language_Processing).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLv9iQK0HOlX"
      },
      "source": [
        "# 자연어 처리(Natural Language Processing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inehXda7HZhF"
      },
      "source": [
        "* 자연어는 일상 생활에서 사용하는 언어\n",
        "* 자연어 처리는 자연어의 의미를 분석 처리하는 일\n",
        "* 텍스트 분류, 감성 분석, 문서 요약, 번역, 질의 응답, 음성 인식, 챗봇과 같은 응용"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W53tu5dXuZR_"
      },
      "source": [
        "## 텍스트 처리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7gXKJ2H15wF_"
      },
      "source": [
        "s = 'No pain no gain'"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eDpKqMwU6G6M",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8465db8a-26dd-4e84-c8d1-146db3073b26"
      },
      "source": [
        "'pain' in s"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kXoIXAD6a7J",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "56a2d1dc-5b6f-484f-f0de-fd91baa44e0b"
      },
      "source": [
        "s.split()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['No', 'pain', 'no', 'gain']"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71btJnM_6Jp5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "246b611c-b59d-4633-bb37-c21e0e4a445c"
      },
      "source": [
        "s.split().index('gain')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRcGQS4h6nnH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e81700eb-403e-45b5-bb7a-a607e6dd7487"
      },
      "source": [
        "s[-4:] # 인덱싱 접근"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'gain'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "igK5VIBI6PMo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "422ddffc-fc5b-4d7e-bf4e-3aa3b10012e6"
      },
      "source": [
        "s.split()[1]"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'pain'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bjIMwR776SIv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "8f9f6de2-64e1-4f28-ef32-a362d72a9b7a"
      },
      "source": [
        "s.split()[2][::-1] #끝에서 reverse 접근"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'on'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lqk6iozeks2f"
      },
      "source": [
        "s = '한글도 처리 가능'"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KLusoOBWkvhI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "653b632e-4cdb-4870-8f7d-0b96ce9f9cf9"
      },
      "source": [
        "'처리' in s"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dy75iFEMkycR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca2fa492-9aae-49d8-d1a1-e3bb88d2c40d"
      },
      "source": [
        "s.split()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['한글도', '처리', '가능']"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrAyEDZdk1Jf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "283c8275-5663-4d5b-df00-b69e25da9416"
      },
      "source": [
        "s.split()[0]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'한글도'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--ANCtg2uPwW"
      },
      "source": [
        "## 영어 처리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxPYRMKXuiAM"
      },
      "source": [
        "#### 대소문자 통합"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkwMuEpRvFI3"
      },
      "source": [
        "* 대소문자를 통합하지 않는다면 컴퓨터는 같은 단어를 다르게 받아들임\n",
        "* 파이썬의 내장 함수 `lower()`, `upper()`를 통해 간단하게 통합 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9oloj0ivF34",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f79adc2d-5963-415a-8d0a-1b4142f8d6a4"
      },
      "source": [
        "s = 'AbCdEFgh'\n",
        "str_lower = s.lower()\n",
        "str_upper = s.upper()\n",
        "\n",
        "print(str_lower, str_upper)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "abcdefgh ABCDEFGH\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwDDC4HjPaWn"
      },
      "source": [
        "### 정규화(Normalization)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yRo-_ohoPdAz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e55b46e-a874-4a70-d23d-c38cb811a143"
      },
      "source": [
        "s = 'I visited UK from Us on 22-09-20'\n",
        "print(s)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I visited UK from Us on 22-09-20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4nP3cRP_Pp09",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ed12c8a-32a8-4a9c-913c-5ad4067f53ae"
      },
      "source": [
        "new_s = s.replace(\"UK\", \"United Kingdom\").replace(\"US\", \"United States\").replace(\"-20\",\"-2020\")\n",
        "\n",
        "print(new_s)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I visited United Kingdom from Us on 22-09-2020\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwfz36YS1FCU"
      },
      "source": [
        "### 정규표현식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myBZJRSL1JLE"
      },
      "source": [
        "* 정규 표현식은 특정 문자들을 편리하게 지정하고 추가, 삭제 가능\n",
        "* 데이터 전처리에서 정규 표현식을 많이 사용\n",
        "* 파이썬에서는 정규 표현식을 지원하는 `re` 패키지 제공"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lLT-dbGQ1NSe"
      },
      "source": [
        "* 정규 표현식 문법\n",
        "  \n",
        "| 특수문자 | 설명 |\n",
        "| - | - |\n",
        "| `.` | 앞의 문자 1개를 표현 |\n",
        "| `?` | 문자 한개를 표현하나 존재할 수도, 존재하지 않을 수도 있음(0개 또는 1개) |\n",
        "| `*` | 앞의 문자가 0개 이상 |\n",
        "| `+` | 앞의 문자가 최소 1개 이상 |\n",
        "| `^` | 뒤의 문자로 문자열이 시작 |\n",
        "| `\\$` | 앞의 문자로 문자열이 끝남 |\n",
        "| `\\{n\\}` | `n`번만큼 반복 |\n",
        "| `\\{n1, n2\\}` | `n1` 이상, `n2` 이하만큼 반복, n2를 지정하지 않으면 `n1` 이상만 반복 |\n",
        "| `\\[ abc \\]` | 안에 문자들 중 한 개의 문자와 매치, a-z처럼 범위도 지정 가능 |\n",
        "| `\\[ ^a \\]` | 해당 문자를 제외하고 매치 |\n",
        "| `a\\|b` | `a` 또는 `b`를 나타냄 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGv0DLZu1PqE"
      },
      "source": [
        "* 정규 표현식에 자주 사용하는 역슬래시(\\\\)를 이용한 문자 규칙\n",
        "\n",
        "| 문자 | 설명 |\n",
        "| - | - |\n",
        "| `\\\\` | 역슬래시 자체를 의미 |\n",
        "| `\\d` | 모든 숫자를 의미, [0-9]와 동일 |\n",
        "| `\\D` | 숫자를 제외한 모든 문자를 의미, [^0-9]와 동일 |\n",
        "| `\\s` | 공백을 의미, [ \\t\\n\\r\\f\\v]와 동일|\n",
        "| `\\S` | 공백을 제외한 모든 문자를 의미, [^ \\t\\n\\r\\f\\v]와 동일 |\n",
        "| `\\w` | 문자와 숫자를 의미, [a-zA-Z0-9]와 동일 |\n",
        "| `\\W` | 문자와 숫자를 제외한 다른 문자를 의미, [^a-zA-Z0-9]와 동일 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEIH8rWo1bJf"
      },
      "source": [
        "#### match\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 문자열이 정규 표현식과 맞는지 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8UD0NCw8UVm"
      },
      "source": [
        "import re #regularexpresion"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lxzhmTi11wS1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dff18ef8-1547-44aa-9166-f34ead0cab34"
      },
      "source": [
        "check = 'ab.'\n",
        "\n",
        "print(re.match(check, 'abc'))\n",
        "print(re.match(check, 'c'))\n",
        "print(re.match(check, 'ab'))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 3), match='abc'>\n",
            "None\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lICVJ-q_1c_M"
      },
      "source": [
        "#### compile\n",
        "\n",
        "* compile을 사용하면 여러 번 사용할 경우 일반 사용보다 더 빠른 속도를 보임\n",
        "* compile을 통해 정규 표현식을 사용할 경우 `re`가 아닌 컴파일한 객체 이름을 통해 사용해야 함\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDTSl5S31wms",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d0ed1f4e-d5e0-4b48-fbbb-bad25db6b617"
      },
      "source": [
        "import time\n",
        "\n",
        "normal_s_time = time.time()\n",
        "\n",
        "r= 'ab.'\n",
        "\n",
        "for i in range(1000):\n",
        "  re.match(check, 'abc')\n",
        "\n",
        "print('일반 사용시 소요 시간: ', time.time() - normal_s_time)\n",
        "\n",
        "compile_s_time = time.time()\n",
        "r = re.compile('ab.')\n",
        "for i in range(1000):\n",
        "  r.match(check)\n",
        "\n",
        "print('컴파일 사용시 소요 시간: ', time.time() - compile_s_time)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "일반 사용시 소요 시간:  0.004087686538696289\n",
            "컴파일 사용시 소요 시간:  0.0007593631744384766\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6OcEzN81fNV"
      },
      "source": [
        "#### search\n",
        "\n",
        "* match와 다르게, search는 문자열의 전체를 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mg1iU0Kg1w70",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "273f3e2e-11f5-475b-83a1-90502944f588"
      },
      "source": [
        "check = 'ab?'\n",
        "\n",
        "print(re.search('a', check))\n",
        "print(re.match('kkkab', check))\n",
        "print(re.search('kkkab', check))\n",
        "print(re.match('ab', check))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 1), match='a'>\n",
            "None\n",
            "None\n",
            "<re.Match object; span=(0, 2), match='ab'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nmkvc1741g5P"
      },
      "source": [
        "#### split\n",
        "\n",
        "* 정규표현식에 해당하는 문자열을 기준으로 문자열을 나눔"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6bLf5Hgs1xZk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e402fb0-3b7e-41c9-88d4-cd6610430ddc"
      },
      "source": [
        "r = re.compile(' ')\n",
        "print(r.split('abc abbc abcbab'))\n",
        "\n",
        "r = re.compile('c')\n",
        "print(r.split('abc abbc abcbab'))\n",
        "\n",
        "r = re.compile('[1-9]')\n",
        "print(r.split('s1abc 2v3s 4sss 5a'))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['abc', 'abbc', 'abcbab']\n",
            "['ab', ' abb', ' ab', 'bab']\n",
            "['s', 'abc ', 'v', 's ', 'sss ', 'a']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fTor-yg_1h49"
      },
      "source": [
        "#### sub\n",
        "\n",
        "* 정규 표현식과 일치하는 부분을 다른 문자열로 교체\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZlqOk9DY1xtF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f54783d-3a5d-42f4-83b3-f4a69bad4ed5"
      },
      "source": [
        "print(re.sub('[a-z]', 'abcdefg','1'))\n",
        "\n",
        "print(re.sub('[^a-z]', 'abc defg', '1'))"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "abc defg\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BUzY6Vz31i8U"
      },
      "source": [
        "#### findall\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 정규 표현식과 맞는 모든 문자(열)을 리스트로 반환\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Osi3UYYv1yHP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "25e9a075-1e69-4e90-956a-1da222f2717c"
      },
      "source": [
        "print(re.findall('[\\d]', '1ab 2cd 3ef 4g'))\n",
        "\n",
        "print(re.findall('[\\W]', '!abcd@@#'))\n",
        "\n",
        "print(re.findall('[\\w]', '!abcd@@#'))\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['1', '2', '3', '4']\n",
            "['!', '@', '@', '#']\n",
            "['a', 'b', 'c', 'd']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxLLmwJu1kTt"
      },
      "source": [
        "#### finditer\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 정규 표현식과 맞는 모든 문자(열)을 `iterator` 객체로 반환\n",
        "* `iterator` 객체를 이용하면 생성된 객체를 하나씩 자동으로 가져올 수 있어 처리가 간편함"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6NOFG3Hy1ycd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7fa2290b-5b9c-4f7c-e809-cba11759cdd7"
      },
      "source": [
        "iter1 = re.finditer('[\\d]', '1ab 2cd 3ef 4g')\n",
        "\n",
        "print(iter1)\n",
        "\n",
        "for i in iter1:\n",
        "  print(i)\n",
        "\n",
        "iter2 = re.finditer('[\\W]', '!abcd@@#')\n",
        "\n",
        "print(iter2)\n",
        "\n",
        "for i in iter2:\n",
        "  print(i)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<callable_iterator object at 0x7a5572a89ba0>\n",
            "<re.Match object; span=(0, 1), match='1'>\n",
            "<re.Match object; span=(4, 5), match='2'>\n",
            "<re.Match object; span=(8, 9), match='3'>\n",
            "<re.Match object; span=(12, 13), match='4'>\n",
            "<callable_iterator object at 0x7a5572a8be20>\n",
            "<re.Match object; span=(0, 1), match='!'>\n",
            "<re.Match object; span=(5, 6), match='@'>\n",
            "<re.Match object; span=(6, 7), match='@'>\n",
            "<re.Match object; span=(7, 8), match='#'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sn4C67oivxxN"
      },
      "source": [
        "### 토큰화(Tokenization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqPZHQZPzFvc"
      },
      "source": [
        "* 특수문자에 대한 처리\n",
        "\n",
        "  + 단어에 일반적으로 사용되는 알파벳, 숫자와는 다르게 특수문자는 별도의 처리가 필요            \n",
        "  + 일괄적으로 단어의 특수문자를 제거하는 방법도 있지만 특수문자가 단어에 특별한 의미를 가질 때 이를 학습에 반영시키지 못할 수도 있음\n",
        "  + 특수문자에 대한 일괄적인 제거보다는 데이터의 특성을 파악하고, 처리를 하는 것이 중요\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FrZTwnlTyzYV"
      },
      "source": [
        "* 특정 단어에 대한 토큰 분리 방법\n",
        "\n",
        "  + 한 단어지만 토큰으로 분리할 때 판단되는 문자들로 이루어진 we're, United Kingdom 등의 단어는 어떻게 분리해야 할지 선택이 필요   \n",
        "  + we're은 한 단어이나 분리해도 단어의 의미에 별 영향을 끼치진 않지만 United Kingdom은 두 단어가 모여 특정 의미를 가리켜 분리해선 안됨\n",
        "  + 사용자가 단어의 특성을 고려해 토큰을 분리하는 것이 학습에 유리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U9_k9rA6v0KM"
      },
      "source": [
        "#### 단어 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t02uSgcSv6IQ"
      },
      "source": [
        "* 파이썬 내장 함수인 `split`을 활용해 단어 토큰화\n",
        "* 공백을 기준으로 단어를 분리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bV6CNv1Sv62F",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a7b2e52c-ea0c-47c1-f114-0187967b9a34"
      },
      "source": [
        "sentence = 'Time is gold'\n",
        "tokens = [x for x in sentence.split(' ')]\n",
        "tokens"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Time', 'is', 'gold']"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_PDCM7YwEi9"
      },
      "source": [
        "* 토큰화는 `nltk` 패키지의 `tokenize` 모듈을 사용해 손쉽게 구현 가능\n",
        "* 단어 토큰화는 `word_tokenize()` 함수를 사용해 구현 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "isgXAhV7wEo7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f709981-5bd8-4bb5-93cd-0c7613562494"
      },
      "source": [
        "#nature language talken\n",
        "\n",
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vXkXYwoywHAl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "26da8d90-429b-4f12-8303-5ee00958e0bc"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "tokens = word_tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Time', 'is', 'gold']"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VcmIjFyRwJpW"
      },
      "source": [
        "#### 문장 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Jkvs6Jov7QF"
      },
      "source": [
        "* 문장 토큰화는 줄바꿈 문자('\\n')를 기준으로 문장을 분리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MkJ60tlTv7Ux",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7909064a-8c5c-4633-dab1-0324bc2d5ca2"
      },
      "source": [
        "sentences = 'The world is a beautiful book.\\nBut of little use to him who cannot read it'\n",
        "print(sentences)\n",
        "\n",
        "tokens = [x for x in sentences.split('\\n')]\n",
        "tokens"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The world is a beautiful book.\n",
            "But of little use to him who cannot read it\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The world is a beautiful book.',\n",
              " 'But of little use to him who cannot read it']"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDSxHUHTwVzG"
      },
      "source": [
        "* 문장 토큰화는 `sent_tokenize()` 함수를 사용해 구현 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oR8iHWiwV6f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7dd99370-5f54-4d2b-b169-a3d3cf4b4358"
      },
      "source": [
        "from nltk.tokenize import sent_tokenize\n",
        "tokens = sent_tokenize(sentences)\n",
        "tokens"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The world is a beautiful book.',\n",
              " 'But of little use to him who cannot read it']"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ca3lOErwbGN"
      },
      "source": [
        "* 문장 토큰화에서는 온점(.)의 처리를 위해 이진 분류기를 사용할 수도 있음\n",
        "* 온점은 문장과 문장을 구분해줄 수도, 문장에 포함된 단어를 구성할 수도 있기 때문에 이를 이진 분류기로 분류해 더욱 좋은 토큰화를 구현할 수도 있음"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVPmqCnV2Ydt"
      },
      "source": [
        "#### 정규 표현식을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hkxXlZb2j0O"
      },
      "source": [
        "* 토큰화 기능을 직접 구현할 수도 있지만 정규 표현식을 이용해 간단하게 구현할 수도 있음\n",
        "* `nltk` 패키지는 정규 표현식을 사용하는 토큰화 도구인 `RegexpTokenizer`를 제공"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9KOEXR7R2cyV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3d169d1-6c5f-4890-8555-26fe6737bf2d"
      },
      "source": [
        "from nltk.tokenize import RegexpTokenizer\n",
        "\n",
        "sentence = 'Where ther\\'s a will, there\\'s a way'\n",
        "\n",
        "tokenizer = RegexpTokenizer(\"[\\w]+\")\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Where', 'ther', 's', 'a', 'will', 'there', 's', 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lhdy85nr2fTt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c022b6aa-d16c-4bf2-ec2d-523f8bb2b32f"
      },
      "source": [
        "tokenizer = RegexpTokenizer(\"[\\s]+\", gaps=True)\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Where', \"ther's\", 'a', 'will,', \"there's\", 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMmCGvTFirsi"
      },
      "source": [
        "#### 케라스를 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pghLPLfSirOJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24bacc52-974b-4ebc-f181-5fda982a3dcb"
      },
      "source": [
        "from tensorflow.keras.preprocessing.text import text_to_word_sequence\n",
        "\n",
        "sentence = 'Where there\\'s a will, there\\'s a way'\n",
        "\n",
        "text_to_word_sequence(sentence)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['where', \"there's\", 'a', 'will', \"there's\", 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOZj2hkzi9Ki"
      },
      "source": [
        "#### TextBlob을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oIDyF_gLi84n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0552c9be-4554-4e6b-bda8-4221bca8755d"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "sentence = 'Where there\\'s a will, there\\'s a way'\n",
        "\n",
        "blob = TextBlob(sentence)\n",
        "blob.words"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "WordList(['Where', 'there', \"'s\", 'a', 'will', 'there', \"'s\", 'a', 'way'])"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5Vvqfynlf9Z"
      },
      "source": [
        "#### 기타 토크나이저\n",
        "\n",
        "* WhiteSpaceTokenizer: 공백을 기준으로 토큰화\n",
        "* WordPunktTokenizer: 텍스트를 알파벳 문자, 숫자, ㅇ라파벳 이외의 문자 리스트로 토큰화\n",
        "* MWETokenizer: MWE는 Multi-Word Expression의 약자로 'republic of korea'와 같이 여러 단어로 이뤄진 특정 그룹을 한 개체로 취급\n",
        "* TweetTokenizer: 트위터에서 사용되는 문장의 토큰화를 위해서 만들어졌으며, 문장 속 감성의 표현과 감정을 다룸"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQh3deRog1It"
      },
      "source": [
        "### n-gram 추출\n",
        "\n",
        "* n-gram은 n개의 어절이나 음절을 연쇄적으로 분류해 그 빈도를 분석\n",
        "* n=1일 때는 unigram, n=2일 때는 bigram, n=3일 때는 trigram으로 불림"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Jj0RiGng7bV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f168943f-1970-4316-bf04-373287ae0f40"
      },
      "source": [
        "from nltk import ngrams\n",
        "\n",
        "sentence = 'There is no royal road to learning'\n",
        "\n",
        "bigram = list(ngrams(sentence.split(),2))\n",
        "\n",
        "print(bigram)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('There', 'is'), ('is', 'no'), ('no', 'royal'), ('royal', 'road'), ('road', 'to'), ('to', 'learning')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-sEZ6IfGh2q2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32e9ad34-e6df-4f3c-d593-216f6ed30b57"
      },
      "source": [
        "trigram = list(ngrams(sentence.split(),3))\n",
        "\n",
        "print(trigram)"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('There', 'is', 'no'), ('is', 'no', 'royal'), ('no', 'royal', 'road'), ('royal', 'road', 'to'), ('road', 'to', 'learning')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0g1yxT62iMvA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d9095b83-9466-4a1f-a0c7-152cec71d18d"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "blob = TextBlob(sentence)\n",
        "blob.ngrams(n=2)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[WordList(['There', 'is']),\n",
              " WordList(['is', 'no']),\n",
              " WordList(['no', 'royal']),\n",
              " WordList(['royal', 'road']),\n",
              " WordList(['road', 'to']),\n",
              " WordList(['to', 'learning'])]"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2hkjduP0idli",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "302c21ee-7d70-4f32-bdd9-6c7e769370e8"
      },
      "source": [
        "blob.ngrams(n=3)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[WordList(['There', 'is', 'no']),\n",
              " WordList(['is', 'no', 'royal']),\n",
              " WordList(['no', 'royal', 'road']),\n",
              " WordList(['royal', 'road', 'to']),\n",
              " WordList(['road', 'to', 'learning'])]"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1iRYvPrOZbGQ"
      },
      "source": [
        "### PoS(Parts of Speech) 태깅"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2xRajzx4_5ng"
      },
      "source": [
        "* PoS는 품사를 의미하며, PoS 태깅은 문장 내에서 단어에 해당하는 각 품사를 태깅"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vo4n1kbt_4Pt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a17fef7c-9d9c-4d3a-b698-d1290d0b85f9"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "from nltk import word_tokenize"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OcVS13f8ATMT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2c3f2f81-c029-40c1-932f-84229f4ec353"
      },
      "source": [
        "word = word_tokenize('Think like man of action and act like man of thought.')\n",
        "word"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Think',\n",
              " 'like',\n",
              " 'man',\n",
              " 'of',\n",
              " 'action',\n",
              " 'and',\n",
              " 'act',\n",
              " 'like',\n",
              " 'man',\n",
              " 'of',\n",
              " 'thought',\n",
              " '.']"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YvoGzfP2AjqJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e4baa89-f390-41d8-a521-82ec262433a9"
      },
      "source": [
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.pos_tag(word)"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Think', 'VBP'),\n",
              " ('like', 'IN'),\n",
              " ('man', 'NN'),\n",
              " ('of', 'IN'),\n",
              " ('action', 'NN'),\n",
              " ('and', 'CC'),\n",
              " ('act', 'NN'),\n",
              " ('like', 'IN'),\n",
              " ('man', 'NN'),\n",
              " ('of', 'IN'),\n",
              " ('thought', 'NN'),\n",
              " ('.', '.')]"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5AaI3nFwB_qL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "629800fa-71ee-4d3b-c0f4-27e1abb879c0"
      },
      "source": [
        "nltk.pos_tag(word_tokenize('A rolling stone gathers no moss'))"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('A', 'DT'),\n",
              " ('rolling', 'VBG'),\n",
              " ('stone', 'NN'),\n",
              " ('gathers', 'NNS'),\n",
              " ('no', 'DT'),\n",
              " ('moss', 'NN')]"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWUanibELmmi"
      },
      "source": [
        "* PoS 태그 리스트"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8RGD-0sVAwaV"
      },
      "source": [
        "| Number | Tag | Description | 설명 |\n",
        "| -- | -- | -- | -- |\n",
        "| 1 | `CC` | Coordinating conjunction |\n",
        "| 2 | `CD` | Cardinal number |\n",
        "| 3 | `DT` | Determiner | 한정사\n",
        "| 4 | `EX` | Existential there |\n",
        "| 5 | `FW` | Foreign word | 외래어 |\n",
        "| 6 | `IN` | Preposition or subordinating conjunction | 전치사 또는 종속 접속사 |\n",
        "| 7 | `JJ` | Adjective | 형용사 |\n",
        "| 8 | `JJR` | Adjective, comparative | 헝용사, 비교급 |\n",
        "| 9 | `JJS` | Adjective, superlative | 형용사, 최상급 |\n",
        "| 10 | `LS` | List item marker |\n",
        "| 11 | `MD` | Modal |\n",
        "| 12 | `NN` | Noun, singular or mass | 명사, 단수형 |\n",
        "| 13 | `NNS` | Noun, plural | 명사, 복수형 |\n",
        "| 14 | `NNP` | Proper noun, singular | 고유명사, 단수형 |\n",
        "| 15 | `NNPS` | Proper noun, plural | 고유명사, 복수형 |\n",
        "| 16 | `PDT` | Predeterminer | 전치한정사 |\n",
        "| 17 | `POS` | Possessive ending | 소유형용사 |\n",
        "| 18 | `PRP` | Personal pronoun | 인칭 대명사 |\n",
        "| 19 | `PRP$` | Possessive pronoun | 소유 대명사 |\n",
        "| 20 | `RB` | Adverb | 부사 |\n",
        "| 21 | `RBR` | Adverb, comparative | 부사, 비교급 |\n",
        "| 22 | `RBS` | Adverb, superlative | 부사, 최상급 |\n",
        "| 23 | `RP` | Particle |\n",
        "| 24 | `SYM` | Symbol | 기호\n",
        "| 25 | `TO` | to |\n",
        "| 26 | `UH` | Interjection | 감탄사 |\n",
        "| 27 | `VB` | Verb, base form | 동사, 원형 |\n",
        "| 28 | `VBD` | Verb, past tense | 동사, 과거형 |\n",
        "| 29 | `VBG` | Verb, gerund or present participle | 동사, 현재분사 |\n",
        "| 30 | `VBN` | Verb, past participle | 동사, 과거분사 |\n",
        "| 31 | `VBP` | Verb, non-3rd person singular present | 동사, 비3인칭 단수 |\n",
        "| 32 | `VBZ` | Verb, 3rd person singular present | 동사, 3인칭 단수 |\n",
        "| 33 | `WDT` | Wh-determiner |\n",
        "| 34 | `WP` | Wh-pronoun |\n",
        "| 35 | `WP$` | Possessive wh-pronoun |\n",
        "| 36 | `WRB` | Wh-adverb |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmDuyhjxugFN"
      },
      "source": [
        "### 불용어 제거"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fj2rvWzwuy7o"
      },
      "source": [
        "* 영어의 전치사(on, in), 한국어의 조사(을, 를) 등은 분석에 필요하지 않은 경우가 많음\n",
        "* 길이가 짧은 단어, 등장 빈도 수가 적은 단어들도 분석에 큰 영향을 주지 않음\n",
        "* 일반적으로 사용되는 도구들은 해당 단어들을 제거해주지만 완벽하게 제거되지는 않음\n",
        "* 사용자가 불용어 사전을 만들어 해당 단어들을 제거하는 것이 좋음\n",
        "* 도구들이 걸러주지 않는 전치사, 조사 등을 불용어 사전을 만들어 불필요한 단어들을 제거\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hjFp9u9quy_t",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "66dff50f-d7e2-4baf-9645-6561cb8bf258"
      },
      "source": [
        "stop_words = 'on in the'\n",
        "stop_words = stop_words.split(' ')\n",
        "stop_words"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['on', 'in', 'the']"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9nKwwtR6uzCr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fe9063dd-2bb7-44cb-d82a-fcf831c9e391"
      },
      "source": [
        "sentence = 'singer on the stage'\n",
        "sentence = sentence.split(' ')\n",
        "nouns = []\n",
        "for noun in sentence:\n",
        "  if noun not in stop_words:\n",
        "    nouns.append(noun)\n",
        "\n",
        "nouns"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['singer', 'stage']"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykTS2yeMMEXF"
      },
      "source": [
        "* `nltk` 패키지에 불용어 리스트 사용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XlDpO3e-MHwa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "43d50789-342e-412e-ae94-5d9ab4ead58a"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import stopwords"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1jC408N2MRR0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "046bf969-f5f3-4648-eca0-913f3a2f8069"
      },
      "source": [
        "stop_words = stopwords.words('english')\n",
        "print(stop_words)"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zShCTozjMYWG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb106dae-83ae-42bf-ae0a-17592edf1c6e"
      },
      "source": [
        "s = 'If you do not walk today, you will have to run tomorrow'\n",
        "words = word_tokenize(s)\n",
        "print(words)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['If', 'you', 'do', 'not', 'walk', 'today', ',', 'you', 'will', 'have', 'to', 'run', 'tomorrow']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qd_gC78EOPdv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f5f37e0-1011-4a1c-fcf5-4ce3c39235ce"
      },
      "source": [
        "no_stopwords = []\n",
        "for w in words:\n",
        "  if w not in stop_words:\n",
        "    no_stopwords.append(w)\n",
        "\n",
        "print(no_stopwords)"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['If', 'walk', 'today', ',', 'run', 'tomorrow']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXO6E3E--gQy"
      },
      "source": [
        "### 철자 교정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AM6lTU2w-q7A"
      },
      "source": [
        "* 텍스트에 오탈자가 존재하는 경우가 있음\n",
        "* 예를 들어, 단어 'apple'을 'aplpe'과 같이 철자 순서가 바뀌거나 spple 같이 철자가 틀릴 수 있음\n",
        "* 사람이 적절한 추정을 통해 이해하는데는 문제가 없지만, 컴퓨터는 이러한 단어를 그대로 받아들여 처리가 필요\n",
        "* 철자 교정 알고리즘은 이미 개발되어 워드 프로세서나 다양한 서비스에서 많이 적용됨\n",
        "됨"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3hUcYaYRwVP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fdb9325e-cd04-4f34-c69e-0a36a3350f64"
      },
      "source": [
        "!pip install autocorrect"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting autocorrect\n",
            "  Downloading autocorrect-2.6.1.tar.gz (622 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/622.8 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m622.8/622.8 kB\u001b[0m \u001b[31m25.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: autocorrect\n",
            "  Building wheel for autocorrect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for autocorrect: filename=autocorrect-2.6.1-py3-none-any.whl size=622364 sha256=80e765d27c3a5d321a6ab675df223aa2d8ab4a44383fa89c3c5abcf7b8953967\n",
            "  Stored in directory: /root/.cache/pip/wheels/b5/7b/6d/b76b29ce11ff8e2521c8c7dd0e5bfee4fb1789d76193124343\n",
            "Successfully built autocorrect\n",
            "Installing collected packages: autocorrect\n",
            "Successfully installed autocorrect-2.6.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4H4TaYE7REcI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "71e53828-1b2b-4d68-dc14-fb90f4b966e6"
      },
      "source": [
        "from autocorrect import Speller\n",
        "\n",
        "spell = Speller('en')\n",
        "\n",
        "print(spell('peoplle'))\n",
        "print(spell('peope'))\n",
        "print(spell('peopae'))\n"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "people\n",
            "people\n",
            "people\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VO03TM_KSL4V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c443a2a-0760-4942-956a-0eadc6a12fab"
      },
      "source": [
        "s = word_tokenize('Earlly biird catchess the womm')\n",
        "\n",
        "print(s)\n",
        "\n",
        "ss = ' '.join([spell(s) for s in s])\n",
        "\n",
        "print(ss)"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Earlly', 'biird', 'catchess', 'the', 'womm']\n",
            "Early bird catches the worm\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H4_svyI2myE3"
      },
      "source": [
        "### 언어의 단수화와 복수화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3tDE8Aeumxml",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa6b7422-37bb-40df-db44-e844b7ad6a5f"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "words = 'apples bananas oranges'\n",
        "tb = TextBlob(words)\n",
        "\n",
        "print(tb.words)\n",
        "print(tb.words.singularize())"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['apples', 'bananas', 'oranges']\n",
            "['apple', 'banana', 'orange']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLqLAg6Snfi8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e7c67f26-e36c-494d-a0ff-477e2d682a8f"
      },
      "source": [
        "words = 'car train airplane'\n",
        "tb = TextBlob(words)\n",
        "\n",
        "print(tb.words)\n",
        "print(tb.words.pluralize())\n"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['car', 'train', 'airplane']\n",
            "['cars', 'trains', 'airplanes']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-9i7ubNX0mD"
      },
      "source": [
        "### 어간(Stemming) 추출"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_28SNykTEbf"
      },
      "source": [
        "import nltk\n",
        "\n",
        "stemmer = nltk.stem.PorterStemmer()"
      ],
      "execution_count": 77,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCwnOPFNTP2v",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "bb9bf53e-a8dd-4a95-a982-622485cc7476"
      },
      "source": [
        "stemmer.stem('application')"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'applic'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O6O5VoWcTQuO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e78d2232-8ea5-4c51-d5d9-d48385bb0f2b"
      },
      "source": [
        "stemmer.stem('beginning')"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'begin'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aHvL3FirTUQO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "140b659a-7679-4ffd-cabb-3bcdf94d596e"
      },
      "source": [
        "stemmer.stem('catches')"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'catch'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4B7VXlfMTiAj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a45aab45-088c-422d-b0f2-e856c96c8195"
      },
      "source": [
        "stemmer.stem('education')"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'educ'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-zauW83XwJ8"
      },
      "source": [
        "### 표제어(Lemmatization) 추출"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FTHh2W06TlPm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dcf64f7c-ab23-49ef-ba7b-63ade49e3602"
      },
      "source": [
        "import nltk\n",
        "nltk.download('wordnet')\n",
        "from nltk.stem.wordnet import WordNetLemmatizer"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LF_jyPwaTzV5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "48c23986-2adc-4b51-878b-3fb799bbb293"
      },
      "source": [
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "lemmatizer.lemmatize('application')"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'application'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WrFilM4jWF9p",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "39d7e2f3-b55d-43d3-9d1f-e89e5241f3ae"
      },
      "source": [
        "lemmatizer.lemmatize('beginning')"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'beginning'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1pwiZSgWV8Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "694e237c-6116-434a-94f6-dee9254243a0"
      },
      "source": [
        "lemmatizer.lemmatize('catches')"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'catch'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MoKJNaJ8WT2x",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4c9bfc99-2401-412a-835b-ad892f1561b5"
      },
      "source": [
        "lemmatizer.lemmatize('education')"
      ],
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'education'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YauGaSZHWg3R"
      },
      "source": [
        "### 개체명 인식(Named Entity Recognition)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t5tcQw9FW8fi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95ef167a-7554-4518-f843-20157510c788"
      },
      "source": [
        "import nltk\n",
        "from nltk import word_tokenize\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/words.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WqWjHuthXWp4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4b068447-477b-43a3-aea1-37435d79eeac"
      },
      "source": [
        "s = 'Rome was not built in a day'\n",
        "print(s)"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Rome was not built in a day\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NnnBqd-YXkwa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e799812-c5dd-4aca-d818-f021404ab69e"
      },
      "source": [
        "tags = nltk.pos_tag(word_tokenize(s))\n",
        "print(tags)"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('Rome', 'NNP'), ('was', 'VBD'), ('not', 'RB'), ('built', 'VBN'), ('in', 'IN'), ('a', 'DT'), ('day', 'NN')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kjnWKrpPYWax",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "377cc12a-4703-4228-ae60-700c8cd65e37"
      },
      "source": [
        "entities = nltk.ne_chunk(tags, binary=True)\n",
        "print(entities)"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(S (NE Rome/NNP) was/VBD not/RB built/VBN in/IN a/DT day/NN)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOVaBRHZauv8"
      },
      "source": [
        "### 단어 중의성(Lexical Ambiguity)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9Y7u8Kqawho",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "70847e87-676c-479f-e67d-58f3348e331e"
      },
      "source": [
        "import nltk\n",
        "from nltk.wsd import lesk\n",
        "\n",
        "s = \"I saw bats.\"\n",
        "\n",
        "print(word_tokenize(s))\n",
        "print(lesk(word_tokenize(s), 'saw'))\n",
        "print(lesk(word_tokenize(s), 'bats'))"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['I', 'saw', 'bats', '.']\n",
            "Synset('saw.v.01')\n",
            "Synset('squash_racket.n.01')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FU4T_80TuSdp"
      },
      "source": [
        "## 한국어 처리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qeUwjUq3lkH"
      },
      "source": [
        "### 정규표현식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1HSV73e530v8"
      },
      "source": [
        "* 한국어 정규 표현식도 대부분의 문법은 영어 정규 표현식과 같음\n",
        "* 한국어는 자음과 모음이 분리되어 있기 때문에, 문법을 지정할 때는 자음과 모음을 동시에 고려해야 함"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DKYR7b3U44RM"
      },
      "source": [
        "#### match\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 문자열이 정규 표현식과 맞는지 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AgwMiWUe4zJN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cd1a6d73-27ba-4e92-b164-678a9c868577"
      },
      "source": [
        "import re\n",
        "check = '[ㄱ-ㅎ]+'\n",
        "\n",
        "print(re.match(check, 'ㅎ 안녕하세요.'))\n",
        "print(re.match(check, '안녕하세요. ㅎ'))"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 1), match='ㅎ'>\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVxRcA8Z457T"
      },
      "source": [
        "#### search\n",
        "\n",
        "* match와 다르게, search는 문자열의 전체를 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WWS4rgxq48N7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e8361982-c038-4e76-d4ee-d23b609513cb"
      },
      "source": [
        "check = '[ㄱ-ㅎ | ㅏ-ㅣ]+'\n",
        "\n",
        "print(re.search(check, 'ㄱ ㅏ 안녕하세요'))\n",
        "print(re.match(check, '안 ㄱ ㅏ'))\n",
        "print(re.search(check, '안ㄱ ㅏ'))\n"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 4), match='ㄱ ㅏ '>\n",
            "None\n",
            "<re.Match object; span=(1, 4), match='ㄱ ㅏ'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCWlVO4S48Ts"
      },
      "source": [
        "#### sub\n",
        "\n",
        "* 정규 표현식과 일치하는 부분을 다른 문자열로 교체"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Kv_oNRs48fU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be840a07-2a41-44f4-e340-5adae108710a"
      },
      "source": [
        "print(re.sub('[가-힣]','가나다라마바사','1'))\n",
        "print(re.sub('[^가-힣]','가나다라마바사','1'))\n"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "가나다라마바사\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pe_7bBBdvlXQ"
      },
      "source": [
        "### 토큰화(Tokenization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6aEfI5YHysxv"
      },
      "source": [
        "* 한국어를 학습 데이터를 사용할 때는 언어의 특성으로 인해 추가로 고려해야 할 사항이 존재\n",
        "* 한국어는 띄어쓰기를 준수하지 않아도 의미가 전달되는 경우가 많아 띄어쓰기가 지켜지지 않을 가능성이 존재\n",
        "* 띄어쓰기가 지켜지지 않으면 정상적인 토큰 분리가 이루어지지 않음\n",
        "* 한국어는 형태소라는 개념이 존재해 추가로 고려해주어야만 함\n",
        "* '그는', '그가' 등의 단어들은 같은 의미를 가리키지만 텍스트 처리에서는 다르게 받아들일 수 있어 처리를 해줘야 함"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "765_lRMqwlZg"
      },
      "source": [
        "#### 한국어 자연어 처리 konlpy와 형태소 분석기 MeCab 설치\n",
        "\n",
        "* https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8CoLF80pwnc9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "37d358a4-d037-44cf-8c0c-36e53e7efba1"
      },
      "source": [
        "%%bash\n",
        "apt-get update\n",
        "apt-get install g++ openjdk-8-jdk python-dev python3-dev\n",
        "pip3 install JPype1\n",
        "pip3 install konlpy"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease\n",
            "Hit:2 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease\n",
            "Hit:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease\n",
            "Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
            "Hit:5 https://r2u.stat.illinois.edu/ubuntu jammy InRelease\n",
            "Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n",
            "Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
            "Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n",
            "Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
            "Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
            "Reading package lists...\n",
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "Package python-dev is not available, but is referred to by another package.\n",
            "This may mean that the package is missing, has been obsoleted, or\n",
            "is only available from another source\n",
            "However the following packages replace it:\n",
            "  python2-dev python2 python-dev-is-python3\n",
            "\n",
            "Requirement already satisfied: JPype1 in /usr/local/lib/python3.10/dist-packages (1.5.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from JPype1) (24.1)\n",
            "Requirement already satisfied: konlpy in /usr/local/lib/python3.10/dist-packages (0.6.0)\n",
            "Requirement already satisfied: JPype1>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.5.0)\n",
            "Requirement already satisfied: lxml>=4.1.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (5.3.0)\n",
            "Requirement already satisfied: numpy>=1.6 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.26.4)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from JPype1>=0.7.0->konlpy) (24.1)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "W: Skipping acquire of configured file 'main/source/Sources' as repository 'https://r2u.stat.illinois.edu/ubuntu jammy InRelease' does not seem to provide it (sources.list entry misspelt?)\n",
            "E: Package 'python-dev' has no installation candidate\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Mecab 설치\n",
        "!apt-get update\n",
        "!apt-get install g++ openjdk-8-jdk\n",
        "!pip3 install konlpy JPype1-py3\n",
        "!bash <(curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh)\n",
        "\n",
        "# mecab-python의 버전 오류로 인해 아래 패키지를 설치하면 코랩에서 Mecab을 사용가능\n",
        "!pip install mecab-python3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Mv1orKcab_O2",
        "outputId": "782b8f97-55b4-46c5-cc9f-994d02545551"
      },
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\r0% [Working]\r            \rHit:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease\n",
            "\r0% [Waiting for headers] [Waiting for headers] [Connected to r2u.stat.illinois.edu (192.17.190.167)]\r                                                                                                    \rHit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease\n",
            "\r0% [Waiting for headers] [Waiting for headers] [Waiting for headers] [Connected to ppa.launchpadcont\r                                                                                                    \rHit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease\n",
            "\r0% [Waiting for headers] [Waiting for headers] [Connected to ppa.launchpadcontent.net (185.125.190.8\r                                                                                                    \rHit:4 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
            "\r                                                                                                    \r0% [Waiting for headers] [Connected to ppa.launchpadcontent.net (185.125.190.80)]\r                                                                                 \rHit:5 https://r2u.stat.illinois.edu/ubuntu jammy InRelease\n",
            "Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n",
            "Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
            "Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n",
            "Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
            "Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
            "Reading package lists... Done\n",
            "W: Skipping acquire of configured file 'main/source/Sources' as repository 'https://r2u.stat.illinois.edu/ubuntu jammy InRelease' does not seem to provide it (sources.list entry misspelt?)\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "g++ is already the newest version (4:11.2.0-1ubuntu1).\n",
            "g++ set to manually installed.\n",
            "The following additional packages will be installed:\n",
            "  fonts-dejavu-core fonts-dejavu-extra libatk-wrapper-java libatk-wrapper-java-jni libfontenc1\n",
            "  libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libice-dev librsvg2-common\n",
            "  libsm-dev libxkbfile1 libxt-dev libxtst6 libxxf86dga1 openjdk-8-jdk-headless openjdk-8-jre\n",
            "  openjdk-8-jre-headless x11-utils\n",
            "Suggested packages:\n",
            "  gvfs libice-doc libsm-doc libxt-doc openjdk-8-demo openjdk-8-source visualvm libnss-mdns\n",
            "  fonts-nanum fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei\n",
            "  fonts-indic mesa-utils\n",
            "The following NEW packages will be installed:\n",
            "  fonts-dejavu-core fonts-dejavu-extra libatk-wrapper-java libatk-wrapper-java-jni libfontenc1\n",
            "  libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libice-dev librsvg2-common\n",
            "  libsm-dev libxkbfile1 libxt-dev libxtst6 libxxf86dga1 openjdk-8-jdk openjdk-8-jdk-headless\n",
            "  openjdk-8-jre openjdk-8-jre-headless x11-utils\n",
            "0 upgraded, 22 newly installed, 0 to remove and 51 not upgraded.\n",
            "Need to get 50.0 MB of archives.\n",
            "After this operation, 169 MB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 fonts-dejavu-core all 2.37-2build1 [1,041 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 fonts-dejavu-extra all 2.37-2build1 [2,041 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfontenc1 amd64 1:1.1.4-1build3 [14.7 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxkbfile1 amd64 1:1.1.0-1build3 [71.8 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxtst6 amd64 2:1.2.3-1build4 [13.4 kB]\n",
            "Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxxf86dga1 amd64 2:1.1.5-0ubuntu3 [12.6 kB]\n",
            "Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 x11-utils amd64 7.7+5build2 [206 kB]\n",
            "Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libatk-wrapper-java all 0.38.0-5build1 [53.1 kB]\n",
            "Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libatk-wrapper-java-jni amd64 0.38.0-5build1 [49.0 kB]\n",
            "Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgtk2.0-common all 2.24.33-2ubuntu2.1 [125 kB]\n",
            "Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgtk2.0-0 amd64 2.24.33-2ubuntu2.1 [2,038 kB]\n",
            "Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgail18 amd64 2.24.33-2ubuntu2.1 [15.9 kB]\n",
            "Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgail-common amd64 2.24.33-2ubuntu2.1 [132 kB]\n",
            "Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgtk2.0-bin amd64 2.24.33-2ubuntu2.1 [7,936 B]\n",
            "Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libice-dev amd64 2:1.0.10-1build2 [51.4 kB]\n",
            "Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 librsvg2-common amd64 2.52.5+dfsg-3ubuntu0.2 [17.7 kB]\n",
            "Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsm-dev amd64 2:1.2.3-1build2 [18.1 kB]\n",
            "Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxt-dev amd64 1:1.2.1-1 [396 kB]\n",
            "Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jre-headless amd64 8u422-b05-1~22.04 [30.8 MB]\n",
            "Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jre amd64 8u422-b05-1~22.04 [75.5 kB]\n",
            "Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jdk-headless amd64 8u422-b05-1~22.04 [8,843 kB]\n",
            "Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jdk amd64 8u422-b05-1~22.04 [4,012 kB]\n",
            "Fetched 50.0 MB in 1s (69.4 MB/s)\n",
            "Selecting previously unselected package fonts-dejavu-core.\n",
            "(Reading database ... 123767 files and directories currently installed.)\n",
            "Preparing to unpack .../00-fonts-dejavu-core_2.37-2build1_all.deb ...\n",
            "Unpacking fonts-dejavu-core (2.37-2build1) ...\n",
            "Selecting previously unselected package fonts-dejavu-extra.\n",
            "Preparing to unpack .../01-fonts-dejavu-extra_2.37-2build1_all.deb ...\n",
            "Unpacking fonts-dejavu-extra (2.37-2build1) ...\n",
            "Selecting previously unselected package libfontenc1:amd64.\n",
            "Preparing to unpack .../02-libfontenc1_1%3a1.1.4-1build3_amd64.deb ...\n",
            "Unpacking libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Selecting previously unselected package libxkbfile1:amd64.\n",
            "Preparing to unpack .../03-libxkbfile1_1%3a1.1.0-1build3_amd64.deb ...\n",
            "Unpacking libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Selecting previously unselected package libxtst6:amd64.\n",
            "Preparing to unpack .../04-libxtst6_2%3a1.2.3-1build4_amd64.deb ...\n",
            "Unpacking libxtst6:amd64 (2:1.2.3-1build4) ...\n",
            "Selecting previously unselected package libxxf86dga1:amd64.\n",
            "Preparing to unpack .../05-libxxf86dga1_2%3a1.1.5-0ubuntu3_amd64.deb ...\n",
            "Unpacking libxxf86dga1:amd64 (2:1.1.5-0ubuntu3) ...\n",
            "Selecting previously unselected package x11-utils.\n",
            "Preparing to unpack .../06-x11-utils_7.7+5build2_amd64.deb ...\n",
            "Unpacking x11-utils (7.7+5build2) ...\n",
            "Selecting previously unselected package libatk-wrapper-java.\n",
            "Preparing to unpack .../07-libatk-wrapper-java_0.38.0-5build1_all.deb ...\n",
            "Unpacking libatk-wrapper-java (0.38.0-5build1) ...\n",
            "Selecting previously unselected package libatk-wrapper-java-jni:amd64.\n",
            "Preparing to unpack .../08-libatk-wrapper-java-jni_0.38.0-5build1_amd64.deb ...\n",
            "Unpacking libatk-wrapper-java-jni:amd64 (0.38.0-5build1) ...\n",
            "Selecting previously unselected package libgtk2.0-common.\n",
            "Preparing to unpack .../09-libgtk2.0-common_2.24.33-2ubuntu2.1_all.deb ...\n",
            "Unpacking libgtk2.0-common (2.24.33-2ubuntu2.1) ...\n",
            "Selecting previously unselected package libgtk2.0-0:amd64.\n",
            "Preparing to unpack .../10-libgtk2.0-0_2.24.33-2ubuntu2.1_amd64.deb ...\n",
            "Unpacking libgtk2.0-0:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Selecting previously unselected package libgail18:amd64.\n",
            "Preparing to unpack .../11-libgail18_2.24.33-2ubuntu2.1_amd64.deb ...\n",
            "Unpacking libgail18:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Selecting previously unselected package libgail-common:amd64.\n",
            "Preparing to unpack .../12-libgail-common_2.24.33-2ubuntu2.1_amd64.deb ...\n",
            "Unpacking libgail-common:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Selecting previously unselected package libgtk2.0-bin.\n",
            "Preparing to unpack .../13-libgtk2.0-bin_2.24.33-2ubuntu2.1_amd64.deb ...\n",
            "Unpacking libgtk2.0-bin (2.24.33-2ubuntu2.1) ...\n",
            "Selecting previously unselected package libice-dev:amd64.\n",
            "Preparing to unpack .../14-libice-dev_2%3a1.0.10-1build2_amd64.deb ...\n",
            "Unpacking libice-dev:amd64 (2:1.0.10-1build2) ...\n",
            "Selecting previously unselected package librsvg2-common:amd64.\n",
            "Preparing to unpack .../15-librsvg2-common_2.52.5+dfsg-3ubuntu0.2_amd64.deb ...\n",
            "Unpacking librsvg2-common:amd64 (2.52.5+dfsg-3ubuntu0.2) ...\n",
            "Selecting previously unselected package libsm-dev:amd64.\n",
            "Preparing to unpack .../16-libsm-dev_2%3a1.2.3-1build2_amd64.deb ...\n",
            "Unpacking libsm-dev:amd64 (2:1.2.3-1build2) ...\n",
            "Selecting previously unselected package libxt-dev:amd64.\n",
            "Preparing to unpack .../17-libxt-dev_1%3a1.2.1-1_amd64.deb ...\n",
            "Unpacking libxt-dev:amd64 (1:1.2.1-1) ...\n",
            "Selecting previously unselected package openjdk-8-jre-headless:amd64.\n",
            "Preparing to unpack .../18-openjdk-8-jre-headless_8u422-b05-1~22.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jre-headless:amd64 (8u422-b05-1~22.04) ...\n",
            "Selecting previously unselected package openjdk-8-jre:amd64.\n",
            "Preparing to unpack .../19-openjdk-8-jre_8u422-b05-1~22.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jre:amd64 (8u422-b05-1~22.04) ...\n",
            "Selecting previously unselected package openjdk-8-jdk-headless:amd64.\n",
            "Preparing to unpack .../20-openjdk-8-jdk-headless_8u422-b05-1~22.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jdk-headless:amd64 (8u422-b05-1~22.04) ...\n",
            "Selecting previously unselected package openjdk-8-jdk:amd64.\n",
            "Preparing to unpack .../21-openjdk-8-jdk_8u422-b05-1~22.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jdk:amd64 (8u422-b05-1~22.04) ...\n",
            "Setting up libice-dev:amd64 (2:1.0.10-1build2) ...\n",
            "Setting up libsm-dev:amd64 (2:1.2.3-1build2) ...\n",
            "Setting up libxtst6:amd64 (2:1.2.3-1build4) ...\n",
            "Setting up libxxf86dga1:amd64 (2:1.1.5-0ubuntu3) ...\n",
            "Setting up libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Setting up libxt-dev:amd64 (1:1.2.1-1) ...\n",
            "Setting up openjdk-8-jre-headless:amd64 (8u422-b05-1~22.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode\n",
            "Setting up fonts-dejavu-core (2.37-2build1) ...\n",
            "Setting up librsvg2-common:amd64 (2.52.5+dfsg-3ubuntu0.2) ...\n",
            "Setting up fonts-dejavu-extra (2.37-2build1) ...\n",
            "Setting up libgtk2.0-common (2.24.33-2ubuntu2.1) ...\n",
            "Setting up libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Setting up libgtk2.0-0:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Setting up openjdk-8-jdk-headless:amd64 (8u422-b05-1~22.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode\n",
            "Setting up libgail18:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Setting up libgtk2.0-bin (2.24.33-2ubuntu2.1) ...\n",
            "Setting up x11-utils (7.7+5build2) ...\n",
            "Setting up libatk-wrapper-java (0.38.0-5build1) ...\n",
            "Setting up libgail-common:amd64 (2.24.33-2ubuntu2.1) ...\n",
            "Setting up libatk-wrapper-java-jni:amd64 (0.38.0-5build1) ...\n",
            "Setting up openjdk-8-jre:amd64 (8u422-b05-1~22.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode\n",
            "Setting up openjdk-8-jdk:amd64 (8u422-b05-1~22.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode\n",
            "Processing triggers for hicolor-icon-theme (0.17-2) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtcm_debug.so.1 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtcm.so.1 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libhwloc.so.15 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libumf.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libgdk-pixbuf-2.0-0:amd64 (2.42.8+dfsg-1ubuntu0.3) ...\n",
            "Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...\n",
            "Requirement already satisfied: konlpy in /usr/local/lib/python3.10/dist-packages (0.6.0)\n",
            "Collecting JPype1-py3\n",
            "  Downloading JPype1-py3-0.5.5.4.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.4/88.4 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: JPype1>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.5.0)\n",
            "Requirement already satisfied: lxml>=4.1.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (5.3.0)\n",
            "Requirement already satisfied: numpy>=1.6 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.26.4)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from JPype1>=0.7.0->konlpy) (24.1)\n",
            "Building wheels for collected packages: JPype1-py3\n",
            "  Building wheel for JPype1-py3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for JPype1-py3: filename=JPype1_py3-0.5.5.4-cp310-cp310-linux_x86_64.whl size=3258012 sha256=3d614cf4a119bf3417af92dd8399265020c97834acad3a721765120e79f2d93e\n",
            "  Stored in directory: /root/.cache/pip/wheels/57/72/ea/b886a286a27c6e3c35ba9e9833b13abc5c5bdc0a9cad91e328\n",
            "Successfully built JPype1-py3\n",
            "Installing collected packages: JPype1-py3\n",
            "Successfully installed JPype1-py3-0.5.5.4\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "_jpype",
                  "jpype"
                ]
              },
              "id": "50c2b8aae2b04de4b1084f5d0d1ef1f9"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mecab-ko is already installed\n",
            "mecab-ko-dic is already installed\n",
            "mecab-python is already installed\n",
            "Done.\n",
            "Requirement already satisfied: mecab-python3 in /usr/local/lib/python3.10/dist-packages (1.0.10)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJhzBOckvqr9"
      },
      "source": [
        "#### 단어 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P-34oqwZwsC3"
      },
      "source": [
        "* 한국어는 공백으로 단어를 분리해도 조사, 접속사 등이 남아 분석에 어려움이 있음\n",
        "* 이를 해결해주는 한국어 토큰화는 조사, 접속사를 분리해주거나 제거\n",
        "* 한국어 토큰화를 사용하기 위해선 `konlpy`와 `mecab`, 'Okt'라는 라이브러리가 필요"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8m9Inqu0wrd5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "826fcad8-e4ec-4349-e372-2d0cd37c9a51"
      },
      "source": [
        "# KoNLPy의 Okt 불러와서 사용해보기\n",
        "from konlpy.tag import Okt\n",
        "\n",
        "Okt = Okt()\n",
        "\n",
        "# 예시 문장\n",
        "sentence = \"언제나 현재에 집중할 수 있다면 행복할것이다.!!!!\"\n",
        "Okt.pos(sentence)"
      ],
      "execution_count": 139,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('언제나', 'Adverb'),\n",
              " ('현재', 'Noun'),\n",
              " ('에', 'Josa'),\n",
              " ('집중', 'Noun'),\n",
              " ('할', 'Verb'),\n",
              " ('수', 'Noun'),\n",
              " ('있다면', 'Adjective'),\n",
              " ('행복할것이다', 'Adjective'),\n",
              " ('.!!!!', 'Punctuation')]"
            ]
          },
          "metadata": {},
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKg9p-SlwyRe"
      },
      "source": [
        "* 토큰화만 실행할 때는 `Okt.morphs()`라는 함수를 이용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ist7zW_RwvcO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "384d7526-36d2-4ab4-e453-b45582f68beb"
      },
      "source": [
        "Okt.morphs(sentence)"
      ],
      "execution_count": 140,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['언제나', '현재', '에', '집중', '할', '수', '있다면', '행복할것이다', '.!!!!']"
            ]
          },
          "metadata": {},
          "execution_count": 140
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7u6-78hw0gD"
      },
      "source": [
        "* 형태소만 사용하고 싶을 때는 `tagger.nouns()`라는 함수를 이용해 조사, 접속사 등을 제거 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1D_Bg0gewwm9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "63bee8b2-2334-456a-d0a3-b90f748f1dbf"
      },
      "source": [
        "Okt.nouns(sentence)"
      ],
      "execution_count": 141,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['현재', '집중', '수']"
            ]
          },
          "metadata": {},
          "execution_count": 141
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ThPaMDOGvsiW"
      },
      "source": [
        "#### 문장 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okKRCWhdxMbH"
      },
      "source": [
        "* 한국어 문장을 토큰화할 때는 kss(korean sentence splitter) 라이브러리 이용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dOKstGpcxv5-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f118c987-164d-4625-a192-d5ede28b4a94"
      },
      "source": [
        "!pip install kss"
      ],
      "execution_count": 142,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting kss\n",
            "  Downloading kss-6.0.4.tar.gz (1.1 MB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.1 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m41.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting emoji==1.2.0 (from kss)\n",
            "  Downloading emoji-1.2.0-py3-none-any.whl.metadata (4.3 kB)\n",
            "Collecting pecab (from kss)\n",
            "  Downloading pecab-1.0.8.tar.gz (26.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.4/26.4 MB\u001b[0m \u001b[31m64.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from kss) (3.4.2)\n",
            "Collecting jamo (from kss)\n",
            "  Downloading jamo-0.4.1-py3-none-any.whl.metadata (2.3 kB)\n",
            "Collecting hangul-jamo (from kss)\n",
            "  Downloading hangul_jamo-1.0.1-py3-none-any.whl.metadata (899 bytes)\n",
            "Collecting tossi (from kss)\n",
            "  Downloading tossi-0.3.1.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting distance (from kss)\n",
            "  Downloading Distance-0.1.3.tar.gz (180 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m180.3/180.3 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pyyaml==6.0 (from kss)\n",
            "  Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (2.0 kB)\n",
            "Collecting unidecode (from kss)\n",
            "  Downloading Unidecode-1.3.8-py3-none-any.whl.metadata (13 kB)\n",
            "Collecting cmudict (from kss)\n",
            "  Downloading cmudict-1.0.31-py3-none-any.whl.metadata (3.5 kB)\n",
            "Collecting koparadigm (from kss)\n",
            "  Downloading koparadigm-0.10.0-py3-none-any.whl.metadata (2.9 kB)\n",
            "Collecting kollocate (from kss)\n",
            "  Downloading kollocate-0.0.2-py3-none-any.whl.metadata (1.8 kB)\n",
            "Collecting bs4 (from kss)\n",
            "  Downloading bs4-0.0.2-py2.py3-none-any.whl.metadata (411 bytes)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from kss) (1.26.4)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.10/dist-packages (from kss) (7.4.4)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from kss) (1.13.1)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4->kss) (4.12.3)\n",
            "Requirement already satisfied: importlib-metadata>=5 in /usr/local/lib/python3.10/dist-packages (from cmudict->kss) (8.5.0)\n",
            "Requirement already satisfied: importlib-resources>=5 in /usr/local/lib/python3.10/dist-packages (from cmudict->kss) (6.4.5)\n",
            "Collecting whoosh (from kollocate->kss)\n",
            "  Downloading Whoosh-2.7.4-py2.py3-none-any.whl.metadata (3.1 kB)\n",
            "Collecting xlrd==1.2.0 (from koparadigm->kss)\n",
            "  Downloading xlrd-1.2.0-py2.py3-none-any.whl.metadata (1.3 kB)\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from pecab->kss) (17.0.0)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from pecab->kss) (2024.9.11)\n",
            "Requirement already satisfied: iniconfig in /usr/local/lib/python3.10/dist-packages (from pytest->kss) (2.0.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from pytest->kss) (24.1)\n",
            "Requirement already satisfied: pluggy<2.0,>=0.12 in /usr/local/lib/python3.10/dist-packages (from pytest->kss) (1.5.0)\n",
            "Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /usr/local/lib/python3.10/dist-packages (from pytest->kss) (1.2.2)\n",
            "Requirement already satisfied: tomli>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pytest->kss) (2.0.2)\n",
            "Collecting bidict (from tossi->kss)\n",
            "  Downloading bidict-0.23.1-py3-none-any.whl.metadata (8.7 kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from tossi->kss) (1.16.0)\n",
            "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=5->cmudict->kss) (3.20.2)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4->kss) (2.6)\n",
            "Downloading emoji-1.2.0-py3-none-any.whl (131 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m131.3/131.3 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m682.2/682.2 kB\u001b[0m \u001b[31m28.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bs4-0.0.2-py2.py3-none-any.whl (1.2 kB)\n",
            "Downloading cmudict-1.0.31-py3-none-any.whl (939 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m939.4/939.4 kB\u001b[0m \u001b[31m40.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading hangul_jamo-1.0.1-py3-none-any.whl (4.4 kB)\n",
            "Downloading jamo-0.4.1-py3-none-any.whl (9.5 kB)\n",
            "Downloading kollocate-0.0.2-py3-none-any.whl (72.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.2/72.2 MB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading koparadigm-0.10.0-py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m48.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading xlrd-1.2.0-py2.py3-none-any.whl (103 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m103.3/103.3 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading Unidecode-1.3.8-py3-none-any.whl (235 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/235.5 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bidict-0.23.1-py3-none-any.whl (32 kB)\n",
            "Downloading Whoosh-2.7.4-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.8/468.8 kB\u001b[0m \u001b[31m22.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: kss, distance, pecab, tossi\n",
            "  Building wheel for kss (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for kss: filename=kss-6.0.4-cp310-cp310-linux_x86_64.whl size=1446485 sha256=cc04e6b80bd48fda1f2d074a76c7693c87562776da1fb39400aeeb9dfe0dee26\n",
            "  Stored in directory: /root/.cache/pip/wheels/dd/70/d5/c9308346829b1eb9e7267d74696919d2453aee6ce350f98b3b\n",
            "  Building wheel for distance (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for distance: filename=Distance-0.1.3-py3-none-any.whl size=16256 sha256=7028fed1861b490f864c90c8d840badfc0bec5d7ee20cb4251f9694f8592465c\n",
            "  Stored in directory: /root/.cache/pip/wheels/e8/bb/de/f71bf63559ea9a921059a5405806f7ff6ed612a9231c4a9309\n",
            "  Building wheel for pecab (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pecab: filename=pecab-1.0.8-py3-none-any.whl size=26646665 sha256=b943863eb1ae385b9bf001b346821768cfc26a42e93bc5005d7f31965315382c\n",
            "  Stored in directory: /root/.cache/pip/wheels/5c/6f/b4/ab61b8863d7d8b1409def8ae31adcaa089fa91b8d022ec309d\n",
            "  Building wheel for tossi (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for tossi: filename=tossi-0.3.1-py3-none-any.whl size=12130 sha256=b24e89d89e51f48207e6124355fae142d42ea7ebbac249730845d38a6a15e795\n",
            "  Stored in directory: /root/.cache/pip/wheels/a7/18/60/1094a6fe93c8063efcd3e6700d09328216682e495a3c51af9f\n",
            "Successfully built kss distance pecab tossi\n",
            "Installing collected packages: whoosh, jamo, hangul-jamo, emoji, distance, xlrd, unidecode, pyyaml, kollocate, bidict, tossi, pecab, koparadigm, cmudict, bs4, kss\n",
            "  Attempting uninstall: xlrd\n",
            "    Found existing installation: xlrd 2.0.1\n",
            "    Uninstalling xlrd-2.0.1:\n",
            "      Successfully uninstalled xlrd-2.0.1\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 6.0.2\n",
            "    Uninstalling PyYAML-6.0.2:\n",
            "      Successfully uninstalled PyYAML-6.0.2\n",
            "Successfully installed bidict-0.23.1 bs4-0.0.2 cmudict-1.0.31 distance-0.1.3 emoji-1.2.0 hangul-jamo-1.0.1 jamo-0.4.1 kollocate-0.0.2 koparadigm-0.10.0 kss-6.0.4 pecab-1.0.8 pyyaml-6.0 tossi-0.3.1 unidecode-1.3.8 whoosh-2.7.4 xlrd-1.2.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1VGyGg5yN-d"
      },
      "source": [
        "* 라이브러리를 이용해도 한국어에는 전치 표현이 존재해 제대로 토큰화가 안됨\n",
        "* 좀 더 나은 학습을 위해 사용자는 해당 부분을 따로 처리해주어야만 함"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIlTDBv7xMg7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "580d7e48-0956-4c4e-ea80-268ff0acb327"
      },
      "source": [
        "import kss\n",
        "\n",
        "text = '진짜? 내일 뭐해? 애매모호한 문장도 가능? 밥은 먹었니? 나는..'\n",
        "print(kss.split_sentences(text))"
      ],
      "execution_count": 144,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:root:Oh! You have mecab in your environment. Kss will take this as a backend! :D\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['진짜? 내일 뭐해?', '애매모호한 문장도 가능?', '밥은 먹었니?', '나는..']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FAq7fR4r7cpA"
      },
      "source": [
        "#### 정규 표현식을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkCInLL67l9M"
      },
      "source": [
        "* 한국어도 정규 표현식을 이용해 토큰화 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lyscjy5a7c4r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "37d09d25-4367-47d2-d2ee-ac042f085109"
      },
      "source": [
        "from nltk.tokenize import RegexpTokenizer\n",
        "\n",
        "sentence = '안녕하세요 ㅋㅋ 저는 자연어 처리(Natural Language Processing)를ㄹ 배우고 있습니다.'\n",
        "\n",
        "tokenizer = RegexpTokenizer(\"[가-힣]+\")\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 145,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['안녕하세요', '저는', '자연어', '처리', '를', '배우고', '있습니다']"
            ]
          },
          "metadata": {},
          "execution_count": 145
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gmXhfXCC7c0u",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e69e4ddf-9486-4ae8-eb17-4d01ed12cbcb"
      },
      "source": [
        "tokenizer = RegexpTokenizer(\"[ㄱ-ㅎ]+\", gaps=True)\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 146,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['안녕하세요 ', ' 저는 자연어 처리(Natural Language Processing)를', ' 배우고 있습니다.']"
            ]
          },
          "metadata": {},
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZdfxeFsn25k"
      },
      "source": [
        "#### 케라스를 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-_Zn15pNn25o",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f66c19d6-314c-43ac-aa1c-9a21b08fc5dc"
      },
      "source": [
        "from tensorflow.keras.preprocessing.text import text_to_word_sequence\n",
        "\n",
        "sentence = '성공의 비결은 단 한가지, 잘할 수 있는 일에 광적으로 집중하는 것이다.'\n",
        "\n",
        "text_to_word_sequence(sentence)"
      ],
      "execution_count": 148,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['성공의', '비결은', '단', '한가지', '잘할', '수', '있는', '일에', '광적으로', '집중하는', '것이다']"
            ]
          },
          "metadata": {},
          "execution_count": 148
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pzEFiZt7n25z"
      },
      "source": [
        "#### TextBlob을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tgzHxLPyn252",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "89785699-4b22-46a4-f803-5475c37be28b"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "blob = TextBlob(sentence)\n",
        "blob.words"
      ],
      "execution_count": 149,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "WordList(['성공의', '비결은', '단', '한가지', '잘할', '수', '있는', '일에', '광적으로', '집중하는', '것이다'])"
            ]
          },
          "metadata": {},
          "execution_count": 149
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xaQF79AaKiN"
      },
      "source": [
        "## Bag of Words(BoW)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1-FpBEoS3CBY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6ec6c79b-9a35-499d-ff1c-bd884d76f27f"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "corpus = [\"Think like a man of action and act like man of thought\"]\n",
        "\n",
        "vector = CountVectorizer()\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 150,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 1 2 2 2 1 1]]\n",
            "{'think': 6, 'like': 3, 'man': 4, 'of': 5, 'action': 1, 'and': 2, 'act': 0, 'thought': 7}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIjY4WXr48FE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8391f21a-5856-4e8f-972f-302e7b3ebeb4"
      },
      "source": [
        "vector = CountVectorizer(stop_words='english')\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 151,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 2 2 1 1]]\n",
            "{'think': 4, 'like': 2, 'man': 3, 'action': 1, 'act': 0, 'thought': 5}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OMeaVigC4eE6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9232ba81-861c-4f09-fe54-2629fbfec701"
      },
      "source": [
        "corpus = [\"평생 살 것처럼 꿈을 꾸어라. 그리고 내일 죽을 것처럼 오늘을 살아라\"]\n",
        "\n",
        "vector = CountVectorizer()\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 152,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[2 1 1 1 1 1 1 1 1]]\n",
            "{'평생': 8, '것처럼': 0, '꿈을': 3, '꾸어라': 2, '그리고': 1, '내일': 4, '죽을': 7, '오늘을': 6, '살아라': 5}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2ud_G3LqIpw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e564f32-ce7f-4831-fc22-750b4ab96654"
      },
      "source": [
        "import re\n",
        "from konlpy.tag import Okt\n",
        "\n",
        "Okt = Okt()\n",
        "\n",
        "corpus = \"평생 살 것처럼 꿈을 꾸어라. 그리고 내일 죽을 것처럼 오늘을 살아라\"\n",
        "\n",
        "tokens = Okt.morphs(re.sub(\"(\\.)\",\"\",corpus))\n",
        "vocab = {}\n",
        "bow = []\n",
        "\n",
        "for tok in tokens:\n",
        "  if tok not in vocab.keys():\n",
        "    vocab[tok] = len(vocab)\n",
        "    bow.insert(len(vocab)-1,1)\n",
        "  else:\n",
        "    index = vocab.get(tok)\n",
        "    bow[index] = bow[index]+1\n",
        "\n",
        "print(bow)\n",
        "print(vocab)"
      ],
      "execution_count": 154,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1, 1, 2, 2, 1, 3, 1, 1, 1, 1, 1, 1]\n",
            "{'평생': 0, '살': 1, '것': 2, '처럼': 3, '꿈': 4, '을': 5, '꾸어라': 6, '그리고': 7, '내일': 8, '죽': 9, '오늘': 10, '살아라': 11}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5n2gkHk17NyC"
      },
      "source": [
        "## 문서 단어 행렬(DTM)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F87rTUZs7S2p"
      },
      "source": [
        "* 문서 단어 행렬(Document-Term Matrix)은 문서에 등장하는 여러 단어들의 빈도를 행렬로 표현\n",
        "* 각 문서에 대한 BoW를 하나의 행렬로 표현한 것"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c4pNYz6V7SO9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f896cab-5130-41fb-bf8b-80ff3d0c8d0f"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "corpus = [\"Think like a man of action and act like man of thought.\",\n",
        "          \"Try not to become a man of success but rather try to become a man of value.\",\n",
        "          \"Give me liberty, of give me death\"]\n",
        "\n",
        "vector = CountVectorizer(stop_words='english')\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 156,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 0 0 2 2 0 1 1 0 0]\n",
            " [0 0 0 0 0 2 1 0 0 2 1]\n",
            " [0 0 1 1 0 0 0 0 0 0 0]]\n",
            "{'think': 7, 'like': 4, 'man': 5, 'action': 1, 'act': 0, 'thought': 8, 'try': 9, 'success': 6, 'value': 10, 'liberty': 3, 'death': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "columns = []\n",
        "for k, v in sorted(vector.vocabulary_.items(), key=lambda item: item[1]):\n",
        "  columns.append(k)\n",
        "\n",
        "df = pd.DataFrame(bow.toarray(), columns = columns)\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 163
        },
        "id": "38EMofo4iFqa",
        "outputId": "a895a66d-73a6-4fea-d55f-68a149a4f1e7"
      },
      "execution_count": 161,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   act  action  death  liberty  like  man  success  think  thought  try  value\n",
              "0    1       1      0        0     2    2        0      1        1    0      0\n",
              "1    0       0      0        0     0    2        1      0        0    2      1\n",
              "2    0       0      1        1     0    0        0      0        0    0      0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-efbbc40b-7a17-4b26-861c-f1aebcf5cb24\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>act</th>\n",
              "      <th>action</th>\n",
              "      <th>death</th>\n",
              "      <th>liberty</th>\n",
              "      <th>like</th>\n",
              "      <th>man</th>\n",
              "      <th>success</th>\n",
              "      <th>think</th>\n",
              "      <th>thought</th>\n",
              "      <th>try</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-efbbc40b-7a17-4b26-861c-f1aebcf5cb24')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-efbbc40b-7a17-4b26-861c-f1aebcf5cb24 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-efbbc40b-7a17-4b26-861c-f1aebcf5cb24');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-61235ad7-9078-474d-8239-50d68ead2bef\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-61235ad7-9078-474d-8239-50d68ead2bef')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-61235ad7-9078-474d-8239-50d68ead2bef button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_5e970b5e-02bb-4dcb-a073-d6f495ccc2d1\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_5e970b5e-02bb-4dcb-a073-d6f495ccc2d1 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"act\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"action\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"death\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"liberty\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"like\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1,\n        \"min\": 0,\n        \"max\": 2,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          2\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"man\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1,\n        \"min\": 0,\n        \"max\": 2,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          2\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"success\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"think\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"thought\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"try\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1,\n        \"min\": 0,\n        \"max\": 2,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          2,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"value\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 161
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z-bPw94FTGb"
      },
      "source": [
        "## 어휘 빈도-문서 역빈도(TF-IDF) 분석"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-0f-hz5Fgai"
      },
      "source": [
        "* 어휘 빈도-문서 역빈도(TF-IDF; Term Frequency-Inverse Docunment Frequency)는 단순히 빈도수가 높은 단어가 핵심어가 아닌, 특정 문서에서만 집중적으로 등장할 때 해당 단어가 문서의 주제를 잘 담고 있는 핵심어라고 가정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3kEwiZBJSSB"
      },
      "source": [
        "* 특정 문서에서 특정단어가 많이 등장하고 그 단어가 다른 문서에서 적게 등장할 때, 그 단어를 특정 문서의 핵심어로 간주\n",
        "* 어휘 빈도-문서 역빈도는 어휘 빈도와 역문서 빈도를 곱해 계산 가능"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s7wTnMY8JEDC"
      },
      "source": [
        "* **어휘 빈도**는 특정 문서에서 특정 단어가 많이 등장하는 것을 의미\n",
        "\n",
        "$$ tf_{x,y} $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8289-mSJF1p"
      },
      "source": [
        "* **역문서 빈도**는 다른 문서에서 등장하지 않는 단어 빈도를 의미\n",
        "\n",
        "$$ log(N/df_x) $$      "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MyZYLgRJH8R"
      },
      "source": [
        "* **어휘 빈도-문서 역빈도**는 다음과 같이 표현\n",
        "\n",
        "$$ W_{x,y} = tf_{x,y} * log(N/df_x) $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cv_23ugcKMjS"
      },
      "source": [
        "* tf-idf를 편리하게 계산하기 위해 `scikit-learn`의 `tfidfvectorizer`를 이용\n",
        "* 앞서 계산한 단어 빈도 수를 입력하여 tf-idf로 변환"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JH25kuy-J8cj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6c3d2fcd-553e-4785-8f70-4a5efd219726"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "tfidf = TfidfVectorizer(stop_words='english').fit(corpus)\n",
        "\n",
        "print(tfidf.transform(corpus).toarray())\n",
        "print(tfidf.vocabulary_)"
      ],
      "execution_count": 163,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.311383   0.311383   0.         0.         0.62276601 0.4736296\n",
            "  0.         0.311383   0.311383   0.         0.        ]\n",
            " [0.         0.         0.         0.         0.         0.52753275\n",
            "  0.34682109 0.         0.         0.69364217 0.34682109]\n",
            " [0.         0.         0.70710678 0.70710678 0.         0.\n",
            "  0.         0.         0.         0.         0.        ]]\n",
            "{'think': 7, 'like': 4, 'man': 5, 'action': 1, 'act': 0, 'thought': 8, 'try': 9, 'success': 6, 'value': 10, 'liberty': 3, 'death': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoGurU81KqYD"
      },
      "source": [
        "* 좀 더 편리하게 확인하기 위해 데이터프레임으로 변환"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uSjYYeDBKGEr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 163
        },
        "outputId": "35812472-b311-484e-8aa0-63d6aeab2506"
      },
      "source": [
        "columns = []\n",
        "for k, v in sorted(tfidf.vocabulary_.items(), key=lambda item: item[1]):\n",
        "  columns.append(k)\n",
        "\n",
        "df = pd.DataFrame(tfidf.transform(corpus).toarray(), columns = columns)\n",
        "df"
      ],
      "execution_count": 164,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        act    action     death   liberty      like       man   success  \\\n",
              "0  0.311383  0.311383  0.000000  0.000000  0.622766  0.473630  0.000000   \n",
              "1  0.000000  0.000000  0.000000  0.000000  0.000000  0.527533  0.346821   \n",
              "2  0.000000  0.000000  0.707107  0.707107  0.000000  0.000000  0.000000   \n",
              "\n",
              "      think   thought       try     value  \n",
              "0  0.311383  0.311383  0.000000  0.000000  \n",
              "1  0.000000  0.000000  0.693642  0.346821  \n",
              "2  0.000000  0.000000  0.000000  0.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3e36bba2-e2b6-4a6b-8e7d-19b9b2291fc1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>act</th>\n",
              "      <th>action</th>\n",
              "      <th>death</th>\n",
              "      <th>liberty</th>\n",
              "      <th>like</th>\n",
              "      <th>man</th>\n",
              "      <th>success</th>\n",
              "      <th>think</th>\n",
              "      <th>thought</th>\n",
              "      <th>try</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.622766</td>\n",
              "      <td>0.473630</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.527533</td>\n",
              "      <td>0.346821</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.693642</td>\n",
              "      <td>0.346821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.707107</td>\n",
              "      <td>0.707107</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3e36bba2-e2b6-4a6b-8e7d-19b9b2291fc1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3e36bba2-e2b6-4a6b-8e7d-19b9b2291fc1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3e36bba2-e2b6-4a6b-8e7d-19b9b2291fc1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-bae82110-fbed-4045-b067-30c8b4d57661\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-bae82110-fbed-4045-b067-30c8b4d57661')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-bae82110-fbed-4045-b067-30c8b4d57661 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_1afd6d1a-049d-4799-a045-b813552ceecf\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_1afd6d1a-049d-4799-a045-b813552ceecf button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"act\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17977706113233077,\n        \"min\": 0.0,\n        \"max\": 0.31138300391661294,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.0,\n          0.31138300391661294\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"action\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17977706113233077,\n        \"min\": 0.0,\n        \"max\": 0.31138300391661294,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.0,\n          0.31138300391661294\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"death\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.408248290463863,\n        \"min\": 0.0,\n        \"max\": 0.7071067811865476,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.7071067811865476,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"liberty\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.408248290463863,\n        \"min\": 0.0,\n        \"max\": 0.7071067811865476,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.7071067811865476,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"like\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.35955412226466155,\n        \"min\": 0.0,\n        \"max\": 0.6227660078332259,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.0,\n          0.6227660078332259\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"man\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.2902646357121108,\n        \"min\": 0.0,\n        \"max\": 0.5275327514657516,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.4736296010332684,\n          0.5275327514657516\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"success\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.20023724763542283,\n        \"min\": 0.0,\n        \"max\": 0.34682108647230336,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.34682108647230336,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"think\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17977706113233077,\n        \"min\": 0.0,\n        \"max\": 0.31138300391661294,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.0,\n          0.31138300391661294\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"thought\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17977706113233077,\n        \"min\": 0.0,\n        \"max\": 0.31138300391661294,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.0,\n          0.31138300391661294\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"try\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.40047449527084567,\n        \"min\": 0.0,\n        \"max\": 0.6936421729446067,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.6936421729446067,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"value\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.20023724763542283,\n        \"min\": 0.0,\n        \"max\": 0.34682108647230336,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.34682108647230336,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 164
        }
      ]
    }
  ]
}