Text_preprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carregando documentos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "import glob \n",
    "import codecs\n",
    "import nltk\n",
    "import re\n",
    "import spacy\n",
    "from nltk.stem import SnowballStemmer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import unicodedata\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\"\"\"Naruto (ナルト?) é uma série de mangá escrita e ilustrada por Masashi Kishimoto, \n",
    "que conta a história de Naruto Uzumaki, um jovem ninja que constantemente procura por \n",
    "reconhecimento e sonha em se tornar Hokage, o ninja líder de sua vila. \n",
    "A história é dividida em duas partes, a primeira parte se passa nos anos da pré-adolescência de Naruto, \n",
    "e a segunda parte se passa em sua adolescência. A série é baseada em dois mangás one-shots de Kishimoto: \n",
    "Karakuri (1995), e que por causa dele Kishimoto ganhou uma menção honrosa no prêmio Hop Step da Shueisha \n",
    "no ano seguinte, e Naruto (1997). A editora Panini Comics licenciou e publicou o mangá em \n",
    "três versões no Brasil, e em Portugal é a editora Devir Manga quem licencia e publica o mangá. \n",
    "A história de Naruto continua com o seu filho, Boruto Uzumaki em Boruto: Naruto Next Generations.\"\"\",\n",
    "     \n",
    "\"\"\"Serial Experiments Lain (シリアルエクスペリメンツレイン Shiriaru Ekusuperimentsu Rein?) é uma série anime \n",
    "lançada pelo estúdio Triangle Staff, dirigida por Ryutaro Nakamura, criação de cenários por \n",
    "Chiaki J. Konaka e desenhada por Yoshitoshi ABe. A série foi transmitida pela TV Tokyo de Julho a \n",
    "Setembro de 1998 e teve um jogo para PlayStation lançado no mesmo ano.\"\"\",\n",
    "     \n",
    "\"\"\"Fullmetal Alchemist (鋼の錬金術師 Hagane no Renkinjutsushi?, lit. \"Alquimista de Aço\") \n",
    "é um anime baseado no mangá shōnen escrito e ilustrado por Hiromu Arakawa. O mangá foi\n",
    "serializado na revista mensal japonesa Monthly Shōnen Gangan entre agosto de 2001 e junho de 2010, \n",
    "com os seus 108 capítulos individuais compilados em 27 volumes em formato tankōbon e publicados pela \n",
    "editora Square Enix. O mundo de Fullmetal Alchemist é baseado no período após a Revolução Industrial \n",
    "Europeia. Situado em um universo ficcional em que a alquimia é uma das mais avançadas técnicas científicas \n",
    "conhecidas pelo homem, a história centra-se nos irmãos Edward Elric e Alphonse Elric, que estão procurando \n",
    "a pedra filosofal para restaurar seus corpos após uma desastrosa tentativa de trazer a mãe falecida de volta à \n",
    "vida através da alquimia.\"\"\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # Pré-processamento"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## limpando o texto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_clean1 = [ unicodedata.normalize('NFKD', s).encode('ISO-8859-1', 'ignore').decode('ISO-8859-1') \n",
    "                 for s in corpus]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "single_char = 'ê'\n",
    "multiple_chars = '\\N{LATIN SMALL LETTER E}\\N{COMBINING CIRCUMFLEX ACCENT}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ê\n",
      "ê\n"
     ]
    }
   ],
   "source": [
    "print(single_char)\n",
    "print(multiple_chars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Naruto (?) e uma serie de manga escrita e ilustrada por Masashi Kishimoto, \\nque conta a historia de Naruto Uzumaki, um jovem ninja que constantemente procura por \\nreconhecimento e sonha em se tornar Hokage, o ninja lider de sua vila. \\nA historia e dividida em duas partes, a primeira parte se passa nos anos da pre-adolescencia de Naruto, \\ne a segunda parte se passa em sua adolescencia. A serie e baseada em dois mangas one-shots de Kishimoto: \\nKarakuri (1995), e que por causa dele Kishimoto ganhou uma mencao honrosa no premio Hop Step da Shueisha \\nno ano seguinte, e Naruto (1997). A editora Panini Comics licenciou e publicou o manga em \\ntres versoes no Brasil, e em Portugal e a editora Devir Manga quem licencia e publica o manga. \\nA historia de Naruto continua com o seu filho, Boruto Uzumaki em Boruto: Naruto Next Generations.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unicodedata.normalize('NFKD', corpus_clean1[0]).encode('ISO-8859-1', 'ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_clean = [ re.sub(r'[^a-zA-Z ]',r' ',doc) for doc in corpus_clean1 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Naruto     e uma serie de manga escrita e ilustrada por Masashi Kishimoto   que conta a historia de Naruto Uzumaki  um jovem ninja que constantemente procura por  reconhecimento e sonha em se tornar Hokage  o ninja lider de sua vila   A historia e dividida em duas partes  a primeira parte se passa nos anos da pre adolescencia de Naruto   e a segunda parte se passa em sua adolescencia  A serie e baseada em dois mangas one shots de Kishimoto   Karakuri         e que por causa dele Kishimoto ganhou uma mencao honrosa no premio Hop Step da Shueisha  no ano seguinte  e Naruto         A editora Panini Comics licenciou e publicou o manga em  tres versoes no Brasil  e em Portugal e a editora Devir Manga quem licencia e publica o manga   A historia de Naruto continua com o seu filho  Boruto Uzumaki em Boruto  Naruto Next Generations '"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_clean[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_list = nltk.corpus.stopwords.words('portuguese')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['de', 'a', 'o', 'que', 'e', 'do', 'da', 'em', 'um', 'para', 'com', 'não', 'uma', 'os', 'no', 'se', 'na', 'por', 'mais', 'as', 'dos', 'como', 'mas', 'ao', 'ele', 'das', 'à', 'seu', 'sua', 'ou', 'quando', 'muito', 'nos', 'já', 'eu', 'também', 'só', 'pelo', 'pela', 'até', 'isso', 'ela', 'entre', 'depois', 'sem', 'mesmo', 'aos', 'seus', 'quem', 'nas', 'me', 'esse', 'eles', 'você', 'essa', 'num', 'nem', 'suas', 'meu', 'às', 'minha', 'numa', 'pelos', 'elas', 'qual', 'nós', 'lhe', 'deles', 'essas', 'esses', 'pelas', 'este', 'dele', 'tu', 'te', 'vocês', 'vos', 'lhes', 'meus', 'minhas', 'teu', 'tua', 'teus', 'tuas', 'nosso', 'nossa', 'nossos', 'nossas', 'dela', 'delas', 'esta', 'estes', 'estas', 'aquele', 'aquela', 'aqueles', 'aquelas', 'isto', 'aquilo', 'estou', 'está', 'estamos', 'estão', 'estive', 'esteve', 'estivemos', 'estiveram', 'estava', 'estávamos', 'estavam', 'estivera', 'estivéramos', 'esteja', 'estejamos', 'estejam', 'estivesse', 'estivéssemos', 'estivessem', 'estiver', 'estivermos', 'estiverem', 'hei', 'há', 'havemos', 'hão', 'houve', 'houvemos', 'houveram', 'houvera', 'houvéramos', 'haja', 'hajamos', 'hajam', 'houvesse', 'houvéssemos', 'houvessem', 'houver', 'houvermos', 'houverem', 'houverei', 'houverá', 'houveremos', 'houverão', 'houveria', 'houveríamos', 'houveriam', 'sou', 'somos', 'são', 'era', 'éramos', 'eram', 'fui', 'foi', 'fomos', 'foram', 'fora', 'fôramos', 'seja', 'sejamos', 'sejam', 'fosse', 'fôssemos', 'fossem', 'for', 'formos', 'forem', 'serei', 'será', 'seremos', 'serão', 'seria', 'seríamos', 'seriam', 'tenho', 'tem', 'temos', 'tém', 'tinha', 'tínhamos', 'tinham', 'tive', 'teve', 'tivemos', 'tiveram', 'tivera', 'tivéramos', 'tenha', 'tenhamos', 'tenham', 'tivesse', 'tivéssemos', 'tivessem', 'tiver', 'tivermos', 'tiverem', 'terei', 'terá', 'teremos', 'terão', 'teria', 'teríamos', 'teriam']\n"
     ]
    }
   ],
   "source": [
    "print(stop_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_tokens = [nltk.word_tokenize(doc) for doc in corpus_clean]   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Naruto', 'e', 'uma', 'serie', 'de', 'manga', 'escrita', 'e', 'ilustrada', 'por', 'Masashi', 'Kishimoto', 'que', 'conta', 'a', 'historia', 'de', 'Naruto', 'Uzumaki', 'um', 'jovem', 'ninja', 'que', 'constantemente', 'procura', 'por', 'reconhecimento', 'e', 'sonha', 'em', 'se', 'tornar', 'Hokage', 'o', 'ninja', 'lider', 'de', 'sua', 'vila', 'A', 'historia', 'e', 'dividida', 'em', 'duas', 'partes', 'a', 'primeira', 'parte', 'se', 'passa', 'nos', 'anos', 'da', 'pre', 'adolescencia', 'de', 'Naruto', 'e', 'a', 'segunda', 'parte', 'se', 'passa', 'em', 'sua', 'adolescencia', 'A', 'serie', 'e', 'baseada', 'em', 'dois', 'mangas', 'one', 'shots', 'de', 'Kishimoto', 'Karakuri', 'e', 'que', 'por', 'causa', 'dele', 'Kishimoto', 'ganhou', 'uma', 'mencao', 'honrosa', 'no', 'premio', 'Hop', 'Step', 'da', 'Shueisha', 'no', 'ano', 'seguinte', 'e', 'Naruto', 'A', 'editora', 'Panini', 'Comics', 'licenciou', 'e', 'publicou', 'o', 'manga', 'em', 'tres', 'versoes', 'no', 'Brasil', 'e', 'em', 'Portugal', 'e', 'a', 'editora', 'Devir', 'Manga', 'quem', 'licencia', 'e', 'publica', 'o', 'manga', 'A', 'historia', 'de', 'Naruto', 'continua', 'com', 'o', 'seu', 'filho', 'Boruto', 'Uzumaki', 'em', 'Boruto', 'Naruto', 'Next', 'Generations']\n"
     ]
    }
   ],
   "source": [
    "print(corpus_tokens[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stopwords(tokens):\n",
    "    return [token for token in tokens if token.lower() not in stop_list]  \n",
    "\n",
    "corpus_stop = [' '.join(remove_stopwords(doc)) for doc in corpus_tokens]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Naruto serie manga escrita ilustrada Masashi Kishimoto conta historia Naruto Uzumaki jovem ninja constantemente procura reconhecimento sonha tornar Hokage ninja lider vila historia dividida duas partes primeira parte passa anos pre adolescencia Naruto segunda parte passa adolescencia serie baseada dois mangas one shots Kishimoto Karakuri causa Kishimoto ganhou mencao honrosa premio Hop Step Shueisha ano seguinte Naruto editora Panini Comics licenciou publicou manga tres versoes Brasil Portugal editora Devir Manga licencia publica manga historia Naruto continua filho Boruto Uzumaki Boruto Naruto Next Generations\n"
     ]
    }
   ],
   "source": [
    "print(corpus_stop[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_tokens2 = [nltk.word_tokenize(doc) for doc in corpus_clean]   \n",
    "ps = SnowballStemmer('portuguese')\n",
    "def do_stem(tokens):\n",
    "    return [ps.stem(word) for word in tokens]\n",
    "\n",
    "corpus_stem = [' '.join(do_stem(doc)) for doc in corpus_tokens2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'narut e uma seri de mang escrit e ilustr por masash kishimot que cont a histor de narut uzumak um jov ninj que constant procur por reconhec e sonh em se torn hokag o ninj lid de sua vil a histor e divid em duas part a primeir part se pass nos anos da pre adolescenc de narut e a segund part se pass em sua adolescenc a seri e bas em dois mang one shots de kishimot karakur e que por caus del kishimot ganh uma menca honros no premi hop step da shueish no ano seguint e narut a editor panin comics licenc e public o mang em tres verso no brasil e em portugal e a editor dev mang quem licenc e public o mang a histor de narut continu com o seu filh borut uzumak em borut narut next generations'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_stem[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'narut e uma seri de mang escrit e ilustr por masash kishimot que cont a histor de narut uzumak um jov ninj que constant procur por reconhec e sonh em se torn hokag o ninj lid de sua vil a histor e divid em duas part a primeir part se pass nos anos da pre adolescenc de narut e a segund part se pass em sua adolescenc a seri e bas em dois mang one shots de kishimot karakur e que por caus del kishimot ganh uma menca honros no premi hop step da shueish no ano seguint e narut a editor panin comics licenc e public o mang em tres verso no brasil e em portugal e a editor dev mang quem licenc e public o mang a histor de narut continu com o seu filh borut uzumak em borut narut next generations'"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_stem[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lematizador\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sp = spacy.load('en_core_web_sm', disable=['parser','ner'])\n",
    "#sp = spacy.load('en_core_web_sm')\n",
    "sp = spacy.load(\"pt_core_news_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Naruto ( ナルト ? ) ser umar série de mangar escrito e ilustrar por Masashi Kishimoto , \\n que contar o história de Naruto Uzumaki , um jovem ninja que constantemente procurar por \\n reconhecimento e sonhar em se tornar Hokage , o ninja líder de suar vila . \\n A história ser dividir em dois partir , o primeiro partir se passar o ano da pré-adolescência de Naruto , \\n e o segundo partir se passar em suar adolescência . A série ser basear em dois mangás one-shots de Kishimoto : \\n Karakuri ( 1995 ) , e que por causar d ele Kishimoto ganhar umar menção honroso o prêmio Hop Step da Shueisha \\n o ano seguinte , e Naruto ( 1997 ) . A editorar Panini Comics licenciar e publicar o mangar em \\n três versão o Brasil , e em Portugal ser o editorar Devir Manga quem licenciar e publicar o mangar . \\n A história de Naruto continuar com o seu filhar , Boruto Uzumaki em Boruto : Naruto Next Generations .'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = sp(corpus[0])\n",
    "' '.join([token.lemma_ for token in doc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Naruto (ナルト?) é uma série de mangá escrita e ilustrada por Masashi Kishimoto, \n",
      "que conta a história de Naruto Uzumaki, um jovem ninja que constantemente procura por \n",
      "reconhecimento e sonha em se tornar Hokage, o ninja líder de sua vila. \n",
      "A história é dividida em duas partes, a primeira parte se passa nos anos da pré-adolescência de Naruto, \n",
      "e a segunda parte se passa em sua adolescência. A série é baseada em dois mangás one-shots de Kishimoto: \n",
      "Karakuri (1995), e que por causa dele Kishimoto ganhou uma menção honrosa no prêmio Hop Step da Shueisha \n",
      "no ano seguinte, e Naruto (1997). A editora Panini Comics licenciou e publicou o mangá em \n",
      "três versões no Brasil, e em Portugal é a editora Devir Manga quem licencia e publica o mangá. \n",
      "A história de Naruto continua com o seu filho, Boruto Uzumaki em Boruto: Naruto Next Generations.\n"
     ]
    }
   ],
   "source": [
    "print(corpus[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Criando Matrizes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matriz Esparsa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t1\n",
      "  (0, 2)\t2\n",
      "  (1, 2)\t3\n",
      "  (2, 0)\t4\n",
      "  (2, 1)\t5\n",
      "  (2, 2)\t6\n"
     ]
    }
   ],
   "source": [
    "from scipy.sparse import csr_matrix\n",
    "import numpy as np\n",
    "\n",
    "indptr = np.array([0,2,3,6])\n",
    "indices = np.array([0,2,2,0,1,2])\n",
    "data = np.array([1,2,3,4,5,6])\n",
    "S = csr_matrix( (data,indices,indptr))\n",
    "print(S)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[1, 0, 2],\n",
       "        [0, 0, 3],\n",
       "        [4, 5, 6]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "S.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando matrizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 194)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer()\n",
    "X = count_vect.fit_transform(corpus)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "names = count_vect.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32),\n",
       " array([115, 192, 182, 169,  43, 104,  59,  82, 132, 106,  93, 142,  37,\n",
       "         77, 184, 181,  89, 117,  36, 135, 144, 164,  56, 151, 174,  78,\n",
       "        101, 168, 187,  49,  51, 125, 134, 124, 126, 119,  16,  41, 137,\n",
       "          8, 153,  22,  50, 105, 120, 160,  92,   1,  27,  44,  73, 108,\n",
       "         80, 118, 138,  81, 167, 161,  15, 152,   2,  52, 122,  33,  99,\n",
       "        141, 178, 185,  25, 133,  47, 103, 143,  98, 139,  38,  32, 157,\n",
       "         67,  24, 116,  74], dtype=int32))"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.getrow(0).nonzero()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('naruto', 115), ('ナルト', 192), ('uma', 182), ('série', 169), ('de', 43), ('mangá', 104), ('escrita', 59), ('ilustrada', 82), ('por', 132), ('masashi', 106), ('kishimoto', 93), ('que', 142), ('conta', 37), ('história', 77), ('uzumaki', 184), ('um', 181), ('jovem', 89), ('ninja', 117), ('constantemente', 36), ('procura', 135), ('reconhecimento', 144), ('sonha', 164), ('em', 56), ('se', 151), ('tornar', 174), ('hokage', 78), ('líder', 101), ('sua', 168), ('vila', 187), ('dividida', 49), ('duas', 51), ('partes', 125), ('primeira', 134), ('parte', 124), ('passa', 126), ('nos', 119), ('anos', 16), ('da', 41), ('pré', 137), ('adolescência', 8), ('segunda', 153), ('baseada', 22), ('dois', 50), ('mangás', 105), ('one', 120), ('shots', 160), ('karakuri', 92), ('1995', 1), ('causa', 27), ('dele', 44), ('ganhou', 73), ('menção', 108), ('honrosa', 80), ('no', 118), ('prêmio', 138), ('hop', 81), ('step', 167), ('shueisha', 161), ('ano', 15), ('seguinte', 152), ('1997', 2), ('editora', 52), ('panini', 122), ('comics', 33), ('licenciou', 99), ('publicou', 141), ('três', 178), ('versões', 185), ('brasil', 25), ('portugal', 133), ('devir', 47), ('manga', 103), ('quem', 143), ('licencia', 98), ('publica', 139), ('continua', 38), ('com', 32), ('seu', 157), ('filho', 67), ('boruto', 24), ('next', 116), ('generations', 74)]\n"
     ]
    }
   ],
   "source": [
    "print( [ (names[w],w) for w in X.getrow(0).nonzero()[1]] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 194)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_vect = TfidfVectorizer()\n",
    "X2 = tfidf_vect.fit_transform(corpus)\n",
    "X2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 10000)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hash_vect = HashingVectorizer(n_features=10000)\n",
    "X3 = hash_vect.fit_transform(corpus)\n",
    "X3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'HashingVectorizer' object has no attribute 'vocabulary_'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-60-2579c45f8c68>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mhash_vect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'HashingVectorizer' object has no attribute 'vocabulary_'"
     ]
    }
   ],
   "source": [
    "hash_vect.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'HashingVectorizer' object has no attribute 'vocabulary_'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-59-46e55322302f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcount_vect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'about'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtfidf_vect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'about'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhash_vect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'about'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'HashingVectorizer' object has no attribute 'vocabulary_'"
     ]
    }
   ],
   "source": [
    "count_vect.vocabulary_['about'], tfidf_vect.vocabulary_['about'], hash_vect.vocabulary_['about']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessando com scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# carregando os dados\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "categories = ['alt.atheism', 'talk.religion.misc',\n",
    "               'comp.graphics', 'sci.space']\n",
    "\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', remove=('headers','footers','quotes')\n",
    "                                       , categories=categories)\n",
    "newsgroups_test = fetch_20newsgroups(subset='test', remove=('headers','footers','quotes')\n",
    "                                       , categories=categories)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_test.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 26879)\n",
      "(1353, 21544)\n"
     ]
    }
   ],
   "source": [
    "count_vect = TfidfVectorizer()\n",
    "X_train = count_vect.fit_transform(newsgroups_train.data)\n",
    "print(X_train.shape)\n",
    "\n",
    "tfidf_vect = TfidfVectorizer()\n",
    "X_test = tfidf_vect.fit_transform(newsgroups_test.data)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## limpando o texto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(299, 8372)\n"
     ]
    }
   ],
   "source": [
    "tfidf_vect = TfidfVectorizer(strip_accents='unicode', lowercase=True, )\n",
    "X2 = tfidf_vect.fit_transform(corpus)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 29277)\n"
     ]
    }
   ],
   "source": [
    "from nltk import word_tokenize          \n",
    "from nltk.stem import WordNetLemmatizer \n",
    "\n",
    "class LemmaTokenizer:\n",
    "    def __init__(self):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "    def __call__(self, doc):\n",
    "        return [self.wnl.lemmatize(t) for t in word_tokenize(doc)]\n",
    "\n",
    "vect = TfidfVectorizer(tokenizer=LemmaTokenizer(), strip_accents='unicode')\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/thiagodepaulo/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['becau'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 19081)\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "porter_stemmer = PorterStemmer()\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "\n",
    "def stemming_tokenizer(str_input):\n",
    "    words = re.sub(r\"[^A-Za-z0-9\\-]\", \" \", str_input).lower().split()\n",
    "    words = [\"#NUMBER\" if token[0].isdigit() else token for token in words]\n",
    "    words = [porter_stemmer.stem(word) for word in words]\n",
    "    return words\n",
    "\n",
    "stem_stopwords = stemming_tokenizer(' '.join(stopwords.words('english')))\n",
    "vect = TfidfVectorizer(stop_words = stem_stopwords, tokenizer=stemming_tokenizer, strip_accents='unicode')\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['rych', 'somewher', 'avail', 'cel', 'format', 'read', 'abl',\n",
       "       'like', 'rule', 'textur', 'manual', 'said', 'explicitli', 'noth',\n",
       "       'store', 'inform', 'know', 'anyon', 'preserv', 'prj', 'orient',\n",
       "       'default', 'given', 'restart', 'reload', 'file', '#number', 'care',\n",
       "       'posit', 'plane', 'map', 'model', 'save', 'notic'], dtype='<U83')"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vect.inverse_transform(X2)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Hi,\\n\\nI've noticed that if you only save a model (with all your mapping planes\\npositioned carefully) to a .3DS file that when you reload it after restarting\\n3DS, they are given a default position and orientation.  But if you save\\nto a .PRJ file their positions/orientation are preserved.  Does anyone\\nknow why this information is not stored in the .3DS file?  Nothing is\\nexplicitly said in the manual about saving texture rules in the .PRJ file. \\nI'd like to be able to read the texture rule information, does anyone have \\nthe format for the .PRJ file?\\n\\nIs the .CEL file format available from somewhere?\\n\\nRych\""
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_train.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 19077)\n"
     ]
    }
   ],
   "source": [
    "vect = CountVectorizer(max_df = 0.3, stop_words = stem_stopwords, tokenizer=stemming_tokenizer, )\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'#number'}"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vect.stop_words_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 8408)\n"
     ]
    }
   ],
   "source": [
    "vect = TfidfVectorizer(min_df=2, stop_words = stem_stopwords, tokenizer=stemming_tokenizer, )\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 181266)\n"
     ]
    }
   ],
   "source": [
    "vect = TfidfVectorizer(ngram_range=(1,2), stop_words = stem_stopwords, tokenizer=stemming_tokenizer, )\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "print(X2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "0",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-99-3b7a27f1f636>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 0"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 48 candidates, totalling 240 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   11.5s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:  1.1min\n",
      "[Parallel(n_jobs=-1)]: Done 240 out of 240 | elapsed:  1.5min finished\n",
      "/home/thiagodepaulo/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_stochastic_gradient.py:573: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=Pipeline(steps=[('vect', CountVectorizer()),\n",
       "                                       ('tfidf', TfidfTransformer()),\n",
       "                                       ('clf', SGDClassifier())]),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'clf__alpha': (1e-05, 1e-06), 'clf__max_iter': (20,),\n",
       "                         'clf__penalty': ('l2', 'elasticnet'),\n",
       "                         'tfidf__norm': ('l1', 'l2'),\n",
       "                         'vect__max_df': (0.5, 0.75, 1.0),\n",
       "                         'vect__ngram_range': ((1, 1), (1, 2))},\n",
       "             refit='f1_macro',\n",
       "             scoring={'f1_weighted', 'f1_micro', 'f1_macro', 'accuracy'},\n",
       "             verbose=1)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier()),\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'vect__max_df': (0.5, 0.75, 1.0),\n",
    "    # 'vect__max_features': (None, 5000, 10000, 50000),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
    "    # 'tfidf__use_idf': (True, False),\n",
    "    'tfidf__norm': ('l1', 'l2'),\n",
    "    'clf__max_iter': (20,),\n",
    "    'clf__alpha': (0.00001, 0.000001),\n",
    "    'clf__penalty': ('l2', 'elasticnet'),    \n",
    "}\n",
    "\n",
    "scoring = {\n",
    "        'accuracy',\n",
    "        'f1_micro',\n",
    "        'f1_macro',\n",
    "        'f1_weighted'\n",
    "    }\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, \n",
    "                           return_train_score=False, scoring=scoring, refit='f1_macro')\n",
    "grid_search.fit(newsgroups_train.data, newsgroups_train.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_clf__alpha</th>\n",
       "      <th>param_clf__max_iter</th>\n",
       "      <th>param_clf__penalty</th>\n",
       "      <th>param_tfidf__norm</th>\n",
       "      <th>param_vect__max_df</th>\n",
       "      <th>param_vect__ngram_range</th>\n",
       "      <th>...</th>\n",
       "      <th>std_test_f1_macro</th>\n",
       "      <th>rank_test_f1_macro</th>\n",
       "      <th>split0_test_accuracy</th>\n",
       "      <th>split1_test_accuracy</th>\n",
       "      <th>split2_test_accuracy</th>\n",
       "      <th>split3_test_accuracy</th>\n",
       "      <th>split4_test_accuracy</th>\n",
       "      <th>mean_test_accuracy</th>\n",
       "      <th>std_test_accuracy</th>\n",
       "      <th>rank_test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.781466</td>\n",
       "      <td>0.075570</td>\n",
       "      <td>0.218755</td>\n",
       "      <td>0.040950</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006572</td>\n",
       "      <td>3</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.839901</td>\n",
       "      <td>0.838251</td>\n",
       "      <td>0.004734</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.730455</td>\n",
       "      <td>0.084023</td>\n",
       "      <td>0.310166</td>\n",
       "      <td>0.081767</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.021929</td>\n",
       "      <td>46</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.788698</td>\n",
       "      <td>0.783784</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.790640</td>\n",
       "      <td>0.799897</td>\n",
       "      <td>0.015176</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.762487</td>\n",
       "      <td>0.178918</td>\n",
       "      <td>0.104586</td>\n",
       "      <td>0.013136</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.005963</td>\n",
       "      <td>11</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.835300</td>\n",
       "      <td>0.003112</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.709784</td>\n",
       "      <td>0.205260</td>\n",
       "      <td>0.283460</td>\n",
       "      <td>0.046596</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.010832</td>\n",
       "      <td>43</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.791155</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.812808</td>\n",
       "      <td>0.805313</td>\n",
       "      <td>0.007422</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.803316</td>\n",
       "      <td>0.104262</td>\n",
       "      <td>0.138345</td>\n",
       "      <td>0.011105</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007361</td>\n",
       "      <td>12</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.832512</td>\n",
       "      <td>0.834807</td>\n",
       "      <td>0.004268</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3.331783</td>\n",
       "      <td>0.293534</td>\n",
       "      <td>0.428598</td>\n",
       "      <td>0.069150</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.019061</td>\n",
       "      <td>45</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.800983</td>\n",
       "      <td>0.791155</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.780788</td>\n",
       "      <td>0.799892</td>\n",
       "      <td>0.012747</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.844691</td>\n",
       "      <td>0.156000</td>\n",
       "      <td>0.227587</td>\n",
       "      <td>0.030258</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.005049</td>\n",
       "      <td>27</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.817734</td>\n",
       "      <td>0.821532</td>\n",
       "      <td>0.006382</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3.293817</td>\n",
       "      <td>0.251922</td>\n",
       "      <td>0.380334</td>\n",
       "      <td>0.088700</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007373</td>\n",
       "      <td>23</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.812808</td>\n",
       "      <td>0.823987</td>\n",
       "      <td>0.007195</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.004079</td>\n",
       "      <td>0.047236</td>\n",
       "      <td>0.191473</td>\n",
       "      <td>0.037258</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011355</td>\n",
       "      <td>14</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.830049</td>\n",
       "      <td>0.826944</td>\n",
       "      <td>0.008842</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3.313298</td>\n",
       "      <td>0.113640</td>\n",
       "      <td>0.450777</td>\n",
       "      <td>0.171966</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006378</td>\n",
       "      <td>6</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.837438</td>\n",
       "      <td>0.833335</td>\n",
       "      <td>0.005628</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.948686</td>\n",
       "      <td>0.085920</td>\n",
       "      <td>0.193932</td>\n",
       "      <td>0.025754</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015406</td>\n",
       "      <td>34</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.790640</td>\n",
       "      <td>0.817588</td>\n",
       "      <td>0.015592</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3.462354</td>\n",
       "      <td>0.041177</td>\n",
       "      <td>0.409927</td>\n",
       "      <td>0.095733</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.014686</td>\n",
       "      <td>25</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.823998</td>\n",
       "      <td>0.009932</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1.414981</td>\n",
       "      <td>0.101897</td>\n",
       "      <td>0.185269</td>\n",
       "      <td>0.031396</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007077</td>\n",
       "      <td>5</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.845209</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.837757</td>\n",
       "      <td>0.004957</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5.448619</td>\n",
       "      <td>0.500948</td>\n",
       "      <td>0.435626</td>\n",
       "      <td>0.054068</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015129</td>\n",
       "      <td>44</td>\n",
       "      <td>0.793612</td>\n",
       "      <td>0.791155</td>\n",
       "      <td>0.788698</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.810345</td>\n",
       "      <td>0.800398</td>\n",
       "      <td>0.011693</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1.534923</td>\n",
       "      <td>0.121580</td>\n",
       "      <td>0.230644</td>\n",
       "      <td>0.051575</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011258</td>\n",
       "      <td>9</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.845209</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.842365</td>\n",
       "      <td>0.835303</td>\n",
       "      <td>0.007377</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>5.671826</td>\n",
       "      <td>0.402508</td>\n",
       "      <td>0.455926</td>\n",
       "      <td>0.070089</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.024869</td>\n",
       "      <td>47</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.769042</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.783251</td>\n",
       "      <td>0.796454</td>\n",
       "      <td>0.017202</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.418050</td>\n",
       "      <td>0.191503</td>\n",
       "      <td>0.198604</td>\n",
       "      <td>0.026325</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.004322</td>\n",
       "      <td>17</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.827586</td>\n",
       "      <td>0.830874</td>\n",
       "      <td>0.002956</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5.558293</td>\n",
       "      <td>0.408692</td>\n",
       "      <td>0.442215</td>\n",
       "      <td>0.136093</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.032609</td>\n",
       "      <td>48</td>\n",
       "      <td>0.788698</td>\n",
       "      <td>0.739558</td>\n",
       "      <td>0.786241</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.795567</td>\n",
       "      <td>0.783683</td>\n",
       "      <td>0.023361</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1.256030</td>\n",
       "      <td>0.098152</td>\n",
       "      <td>0.207394</td>\n",
       "      <td>0.035849</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.010962</td>\n",
       "      <td>33</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.832924</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.807882</td>\n",
       "      <td>0.817596</td>\n",
       "      <td>0.009471</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>4.150820</td>\n",
       "      <td>0.115423</td>\n",
       "      <td>0.407977</td>\n",
       "      <td>0.113387</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.005565</td>\n",
       "      <td>21</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.820197</td>\n",
       "      <td>0.824482</td>\n",
       "      <td>0.004082</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1.195833</td>\n",
       "      <td>0.088259</td>\n",
       "      <td>0.179329</td>\n",
       "      <td>0.038830</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011358</td>\n",
       "      <td>28</td>\n",
       "      <td>0.847666</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.810345</td>\n",
       "      <td>0.823494</td>\n",
       "      <td>0.013404</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>4.263381</td>\n",
       "      <td>0.268481</td>\n",
       "      <td>0.415344</td>\n",
       "      <td>0.086465</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007390</td>\n",
       "      <td>16</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.826455</td>\n",
       "      <td>0.005387</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1.144487</td>\n",
       "      <td>0.099848</td>\n",
       "      <td>0.181845</td>\n",
       "      <td>0.030771</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.012469</td>\n",
       "      <td>29</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.827586</td>\n",
       "      <td>0.821537</td>\n",
       "      <td>0.009345</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>3.887525</td>\n",
       "      <td>0.163775</td>\n",
       "      <td>0.429371</td>\n",
       "      <td>0.059713</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011681</td>\n",
       "      <td>24</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.842365</td>\n",
       "      <td>0.824984</td>\n",
       "      <td>0.010797</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1.004613</td>\n",
       "      <td>0.112963</td>\n",
       "      <td>0.207150</td>\n",
       "      <td>0.049645</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.013751</td>\n",
       "      <td>1</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.852580</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.839901</td>\n",
       "      <td>0.836285</td>\n",
       "      <td>0.013755</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3.489990</td>\n",
       "      <td>0.152679</td>\n",
       "      <td>0.399019</td>\n",
       "      <td>0.074832</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017671</td>\n",
       "      <td>10</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.857494</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.834317</td>\n",
       "      <td>0.013595</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.980913</td>\n",
       "      <td>0.058161</td>\n",
       "      <td>0.198800</td>\n",
       "      <td>0.026794</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015433</td>\n",
       "      <td>13</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.844828</td>\n",
       "      <td>0.829899</td>\n",
       "      <td>0.013498</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>3.353435</td>\n",
       "      <td>0.218558</td>\n",
       "      <td>0.378743</td>\n",
       "      <td>0.055678</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017672</td>\n",
       "      <td>19</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.850123</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.832512</td>\n",
       "      <td>0.827928</td>\n",
       "      <td>0.013064</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1.011428</td>\n",
       "      <td>0.108064</td>\n",
       "      <td>0.170097</td>\n",
       "      <td>0.026682</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006866</td>\n",
       "      <td>2</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.837438</td>\n",
       "      <td>0.836284</td>\n",
       "      <td>0.005712</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>3.476389</td>\n",
       "      <td>0.151603</td>\n",
       "      <td>0.413923</td>\n",
       "      <td>0.066623</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.022398</td>\n",
       "      <td>26</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.796069</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.837438</td>\n",
       "      <td>0.824982</td>\n",
       "      <td>0.016340</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0.879326</td>\n",
       "      <td>0.046263</td>\n",
       "      <td>0.189003</td>\n",
       "      <td>0.029447</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.012078</td>\n",
       "      <td>39</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.798526</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.815271</td>\n",
       "      <td>0.814651</td>\n",
       "      <td>0.010487</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>3.199194</td>\n",
       "      <td>0.415425</td>\n",
       "      <td>0.402208</td>\n",
       "      <td>0.070496</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006197</td>\n",
       "      <td>30</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.825123</td>\n",
       "      <td>0.822518</td>\n",
       "      <td>0.005856</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>0.903133</td>\n",
       "      <td>0.117907</td>\n",
       "      <td>0.175164</td>\n",
       "      <td>0.015804</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009830</td>\n",
       "      <td>40</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.798526</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.805419</td>\n",
       "      <td>0.810715</td>\n",
       "      <td>0.009755</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>3.186427</td>\n",
       "      <td>0.160274</td>\n",
       "      <td>0.441265</td>\n",
       "      <td>0.080052</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017691</td>\n",
       "      <td>37</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.793612</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.814661</td>\n",
       "      <td>0.014176</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>1.052073</td>\n",
       "      <td>0.109593</td>\n",
       "      <td>0.205265</td>\n",
       "      <td>0.059111</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.004097</td>\n",
       "      <td>36</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.812808</td>\n",
       "      <td>0.816616</td>\n",
       "      <td>0.006192</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3.484935</td>\n",
       "      <td>0.460935</td>\n",
       "      <td>0.399847</td>\n",
       "      <td>0.060045</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>l2</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.014912</td>\n",
       "      <td>35</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.800983</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.802956</td>\n",
       "      <td>0.815628</td>\n",
       "      <td>0.014660</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>1.216375</td>\n",
       "      <td>0.074438</td>\n",
       "      <td>0.179980</td>\n",
       "      <td>0.023812</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009513</td>\n",
       "      <td>7</td>\n",
       "      <td>0.840295</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.817734</td>\n",
       "      <td>0.832343</td>\n",
       "      <td>0.010007</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>5.649092</td>\n",
       "      <td>0.354507</td>\n",
       "      <td>0.450916</td>\n",
       "      <td>0.148370</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016995</td>\n",
       "      <td>15</td>\n",
       "      <td>0.835381</td>\n",
       "      <td>0.847666</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.812808</td>\n",
       "      <td>0.828901</td>\n",
       "      <td>0.014562</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>1.516918</td>\n",
       "      <td>0.210002</td>\n",
       "      <td>0.206745</td>\n",
       "      <td>0.032923</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.008610</td>\n",
       "      <td>8</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.845209</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.834975</td>\n",
       "      <td>0.832843</td>\n",
       "      <td>0.006746</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>6.146172</td>\n",
       "      <td>0.434750</td>\n",
       "      <td>0.546013</td>\n",
       "      <td>0.154832</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.014432</td>\n",
       "      <td>22</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.842752</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.830049</td>\n",
       "      <td>0.826452</td>\n",
       "      <td>0.010111</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>1.718571</td>\n",
       "      <td>0.222621</td>\n",
       "      <td>0.215502</td>\n",
       "      <td>0.037446</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.013580</td>\n",
       "      <td>4</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.845209</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.844828</td>\n",
       "      <td>0.834813</td>\n",
       "      <td>0.010396</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>6.146915</td>\n",
       "      <td>0.568203</td>\n",
       "      <td>0.429336</td>\n",
       "      <td>0.072391</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l1</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009655</td>\n",
       "      <td>32</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.823096</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.832512</td>\n",
       "      <td>0.822522</td>\n",
       "      <td>0.008750</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>1.234500</td>\n",
       "      <td>0.132468</td>\n",
       "      <td>0.220534</td>\n",
       "      <td>0.069370</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.005122</td>\n",
       "      <td>41</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.805419</td>\n",
       "      <td>0.811698</td>\n",
       "      <td>0.005831</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>4.512287</td>\n",
       "      <td>0.286832</td>\n",
       "      <td>0.423238</td>\n",
       "      <td>0.089565</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.013655</td>\n",
       "      <td>20</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.828010</td>\n",
       "      <td>0.844828</td>\n",
       "      <td>0.823020</td>\n",
       "      <td>0.012394</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>1.194708</td>\n",
       "      <td>0.057217</td>\n",
       "      <td>0.194088</td>\n",
       "      <td>0.021766</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015468</td>\n",
       "      <td>31</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.805897</td>\n",
       "      <td>0.845209</td>\n",
       "      <td>0.830049</td>\n",
       "      <td>0.820064</td>\n",
       "      <td>0.015360</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3.980653</td>\n",
       "      <td>0.176984</td>\n",
       "      <td>0.386150</td>\n",
       "      <td>0.039602</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>0.75</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011457</td>\n",
       "      <td>18</td>\n",
       "      <td>0.815725</td>\n",
       "      <td>0.830467</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.837838</td>\n",
       "      <td>0.830049</td>\n",
       "      <td>0.825469</td>\n",
       "      <td>0.009410</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>1.207627</td>\n",
       "      <td>0.046130</td>\n",
       "      <td>0.216191</td>\n",
       "      <td>0.045195</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009871</td>\n",
       "      <td>42</td>\n",
       "      <td>0.813268</td>\n",
       "      <td>0.825553</td>\n",
       "      <td>0.800983</td>\n",
       "      <td>0.808354</td>\n",
       "      <td>0.802956</td>\n",
       "      <td>0.810223</td>\n",
       "      <td>0.008786</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>2.937894</td>\n",
       "      <td>0.316478</td>\n",
       "      <td>0.233103</td>\n",
       "      <td>0.028524</td>\n",
       "      <td>1e-06</td>\n",
       "      <td>20</td>\n",
       "      <td>elasticnet</td>\n",
       "      <td>l2</td>\n",
       "      <td>1</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>...</td>\n",
       "      <td>0.008590</td>\n",
       "      <td>38</td>\n",
       "      <td>0.820639</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.803440</td>\n",
       "      <td>0.822660</td>\n",
       "      <td>0.813672</td>\n",
       "      <td>0.007077</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0        1.781466      0.075570         0.218755        0.040950   \n",
       "1        2.730455      0.084023         0.310166        0.081767   \n",
       "2        0.762487      0.178918         0.104586        0.013136   \n",
       "3        2.709784      0.205260         0.283460        0.046596   \n",
       "4        0.803316      0.104262         0.138345        0.011105   \n",
       "5        3.331783      0.293534         0.428598        0.069150   \n",
       "6        0.844691      0.156000         0.227587        0.030258   \n",
       "7        3.293817      0.251922         0.380334        0.088700   \n",
       "8        1.004079      0.047236         0.191473        0.037258   \n",
       "9        3.313298      0.113640         0.450777        0.171966   \n",
       "10       0.948686      0.085920         0.193932        0.025754   \n",
       "11       3.462354      0.041177         0.409927        0.095733   \n",
       "12       1.414981      0.101897         0.185269        0.031396   \n",
       "13       5.448619      0.500948         0.435626        0.054068   \n",
       "14       1.534923      0.121580         0.230644        0.051575   \n",
       "15       5.671826      0.402508         0.455926        0.070089   \n",
       "16       1.418050      0.191503         0.198604        0.026325   \n",
       "17       5.558293      0.408692         0.442215        0.136093   \n",
       "18       1.256030      0.098152         0.207394        0.035849   \n",
       "19       4.150820      0.115423         0.407977        0.113387   \n",
       "20       1.195833      0.088259         0.179329        0.038830   \n",
       "21       4.263381      0.268481         0.415344        0.086465   \n",
       "22       1.144487      0.099848         0.181845        0.030771   \n",
       "23       3.887525      0.163775         0.429371        0.059713   \n",
       "24       1.004613      0.112963         0.207150        0.049645   \n",
       "25       3.489990      0.152679         0.399019        0.074832   \n",
       "26       0.980913      0.058161         0.198800        0.026794   \n",
       "27       3.353435      0.218558         0.378743        0.055678   \n",
       "28       1.011428      0.108064         0.170097        0.026682   \n",
       "29       3.476389      0.151603         0.413923        0.066623   \n",
       "30       0.879326      0.046263         0.189003        0.029447   \n",
       "31       3.199194      0.415425         0.402208        0.070496   \n",
       "32       0.903133      0.117907         0.175164        0.015804   \n",
       "33       3.186427      0.160274         0.441265        0.080052   \n",
       "34       1.052073      0.109593         0.205265        0.059111   \n",
       "35       3.484935      0.460935         0.399847        0.060045   \n",
       "36       1.216375      0.074438         0.179980        0.023812   \n",
       "37       5.649092      0.354507         0.450916        0.148370   \n",
       "38       1.516918      0.210002         0.206745        0.032923   \n",
       "39       6.146172      0.434750         0.546013        0.154832   \n",
       "40       1.718571      0.222621         0.215502        0.037446   \n",
       "41       6.146915      0.568203         0.429336        0.072391   \n",
       "42       1.234500      0.132468         0.220534        0.069370   \n",
       "43       4.512287      0.286832         0.423238        0.089565   \n",
       "44       1.194708      0.057217         0.194088        0.021766   \n",
       "45       3.980653      0.176984         0.386150        0.039602   \n",
       "46       1.207627      0.046130         0.216191        0.045195   \n",
       "47       2.937894      0.316478         0.233103        0.028524   \n",
       "\n",
       "   param_clf__alpha param_clf__max_iter param_clf__penalty param_tfidf__norm  \\\n",
       "0             1e-05                  20                 l2                l1   \n",
       "1             1e-05                  20                 l2                l1   \n",
       "2             1e-05                  20                 l2                l1   \n",
       "3             1e-05                  20                 l2                l1   \n",
       "4             1e-05                  20                 l2                l1   \n",
       "5             1e-05                  20                 l2                l1   \n",
       "6             1e-05                  20                 l2                l2   \n",
       "7             1e-05                  20                 l2                l2   \n",
       "8             1e-05                  20                 l2                l2   \n",
       "9             1e-05                  20                 l2                l2   \n",
       "10            1e-05                  20                 l2                l2   \n",
       "11            1e-05                  20                 l2                l2   \n",
       "12            1e-05                  20         elasticnet                l1   \n",
       "13            1e-05                  20         elasticnet                l1   \n",
       "14            1e-05                  20         elasticnet                l1   \n",
       "15            1e-05                  20         elasticnet                l1   \n",
       "16            1e-05                  20         elasticnet                l1   \n",
       "17            1e-05                  20         elasticnet                l1   \n",
       "18            1e-05                  20         elasticnet                l2   \n",
       "19            1e-05                  20         elasticnet                l2   \n",
       "20            1e-05                  20         elasticnet                l2   \n",
       "21            1e-05                  20         elasticnet                l2   \n",
       "22            1e-05                  20         elasticnet                l2   \n",
       "23            1e-05                  20         elasticnet                l2   \n",
       "24            1e-06                  20                 l2                l1   \n",
       "25            1e-06                  20                 l2                l1   \n",
       "26            1e-06                  20                 l2                l1   \n",
       "27            1e-06                  20                 l2                l1   \n",
       "28            1e-06                  20                 l2                l1   \n",
       "29            1e-06                  20                 l2                l1   \n",
       "30            1e-06                  20                 l2                l2   \n",
       "31            1e-06                  20                 l2                l2   \n",
       "32            1e-06                  20                 l2                l2   \n",
       "33            1e-06                  20                 l2                l2   \n",
       "34            1e-06                  20                 l2                l2   \n",
       "35            1e-06                  20                 l2                l2   \n",
       "36            1e-06                  20         elasticnet                l1   \n",
       "37            1e-06                  20         elasticnet                l1   \n",
       "38            1e-06                  20         elasticnet                l1   \n",
       "39            1e-06                  20         elasticnet                l1   \n",
       "40            1e-06                  20         elasticnet                l1   \n",
       "41            1e-06                  20         elasticnet                l1   \n",
       "42            1e-06                  20         elasticnet                l2   \n",
       "43            1e-06                  20         elasticnet                l2   \n",
       "44            1e-06                  20         elasticnet                l2   \n",
       "45            1e-06                  20         elasticnet                l2   \n",
       "46            1e-06                  20         elasticnet                l2   \n",
       "47            1e-06                  20         elasticnet                l2   \n",
       "\n",
       "   param_vect__max_df param_vect__ngram_range  ... std_test_f1_macro  \\\n",
       "0                 0.5                  (1, 1)  ...          0.006572   \n",
       "1                 0.5                  (1, 2)  ...          0.021929   \n",
       "2                0.75                  (1, 1)  ...          0.005963   \n",
       "3                0.75                  (1, 2)  ...          0.010832   \n",
       "4                   1                  (1, 1)  ...          0.007361   \n",
       "5                   1                  (1, 2)  ...          0.019061   \n",
       "6                 0.5                  (1, 1)  ...          0.005049   \n",
       "7                 0.5                  (1, 2)  ...          0.007373   \n",
       "8                0.75                  (1, 1)  ...          0.011355   \n",
       "9                0.75                  (1, 2)  ...          0.006378   \n",
       "10                  1                  (1, 1)  ...          0.015406   \n",
       "11                  1                  (1, 2)  ...          0.014686   \n",
       "12                0.5                  (1, 1)  ...          0.007077   \n",
       "13                0.5                  (1, 2)  ...          0.015129   \n",
       "14               0.75                  (1, 1)  ...          0.011258   \n",
       "15               0.75                  (1, 2)  ...          0.024869   \n",
       "16                  1                  (1, 1)  ...          0.004322   \n",
       "17                  1                  (1, 2)  ...          0.032609   \n",
       "18                0.5                  (1, 1)  ...          0.010962   \n",
       "19                0.5                  (1, 2)  ...          0.005565   \n",
       "20               0.75                  (1, 1)  ...          0.011358   \n",
       "21               0.75                  (1, 2)  ...          0.007390   \n",
       "22                  1                  (1, 1)  ...          0.012469   \n",
       "23                  1                  (1, 2)  ...          0.011681   \n",
       "24                0.5                  (1, 1)  ...          0.013751   \n",
       "25                0.5                  (1, 2)  ...          0.017671   \n",
       "26               0.75                  (1, 1)  ...          0.015433   \n",
       "27               0.75                  (1, 2)  ...          0.017672   \n",
       "28                  1                  (1, 1)  ...          0.006866   \n",
       "29                  1                  (1, 2)  ...          0.022398   \n",
       "30                0.5                  (1, 1)  ...          0.012078   \n",
       "31                0.5                  (1, 2)  ...          0.006197   \n",
       "32               0.75                  (1, 1)  ...          0.009830   \n",
       "33               0.75                  (1, 2)  ...          0.017691   \n",
       "34                  1                  (1, 1)  ...          0.004097   \n",
       "35                  1                  (1, 2)  ...          0.014912   \n",
       "36                0.5                  (1, 1)  ...          0.009513   \n",
       "37                0.5                  (1, 2)  ...          0.016995   \n",
       "38               0.75                  (1, 1)  ...          0.008610   \n",
       "39               0.75                  (1, 2)  ...          0.014432   \n",
       "40                  1                  (1, 1)  ...          0.013580   \n",
       "41                  1                  (1, 2)  ...          0.009655   \n",
       "42                0.5                  (1, 1)  ...          0.005122   \n",
       "43                0.5                  (1, 2)  ...          0.013655   \n",
       "44               0.75                  (1, 1)  ...          0.015468   \n",
       "45               0.75                  (1, 2)  ...          0.011457   \n",
       "46                  1                  (1, 1)  ...          0.009871   \n",
       "47                  1                  (1, 2)  ...          0.008590   \n",
       "\n",
       "    rank_test_f1_macro  split0_test_accuracy  split1_test_accuracy  \\\n",
       "0                    3              0.830467              0.835381   \n",
       "1                   46              0.815725              0.788698   \n",
       "2                   11              0.835381              0.840295   \n",
       "3                   43              0.805897              0.808354   \n",
       "4                   12              0.830467              0.842752   \n",
       "5                   45              0.810811              0.800983   \n",
       "6                   27              0.828010              0.810811   \n",
       "7                   23              0.830467              0.828010   \n",
       "8                   14              0.823096              0.840295   \n",
       "9                    6              0.825553              0.835381   \n",
       "10                  34              0.813268              0.832924   \n",
       "11                  25              0.813268              0.830467   \n",
       "12                   5              0.830467              0.840295   \n",
       "13                  44              0.793612              0.791155   \n",
       "14                   9              0.825553              0.845209   \n",
       "15                  47              0.810811              0.769042   \n",
       "16                  17              0.828010              0.830467   \n",
       "17                  48              0.788698              0.739558   \n",
       "18                  33              0.815725              0.832924   \n",
       "19                  21              0.820639              0.830467   \n",
       "20                  28              0.847666              0.828010   \n",
       "21                  16              0.828010              0.828010   \n",
       "22                  29              0.808354              0.835381   \n",
       "23                  24              0.818182              0.810811   \n",
       "24                   1              0.837838              0.852580   \n",
       "25                  10              0.828010              0.857494   \n",
       "26                  13              0.828010              0.840295   \n",
       "27                  19              0.823096              0.850123   \n",
       "28                   2              0.837838              0.842752   \n",
       "29                  26              0.820639              0.842752   \n",
       "30                  39              0.808354              0.823096   \n",
       "31                  30              0.825553              0.825553   \n",
       "32                  40              0.810811              0.828010   \n",
       "33                  37              0.805897              0.823096   \n",
       "34                  36              0.828010              0.813268   \n",
       "35                  35              0.808354              0.828010   \n",
       "36                   7              0.840295              0.842752   \n",
       "37                  15              0.835381              0.847666   \n",
       "38                   8              0.828010              0.845209   \n",
       "39                  22              0.815725              0.842752   \n",
       "40                   4              0.837838              0.845209   \n",
       "41                  32              0.818182              0.823096   \n",
       "42                  41              0.820639              0.810811   \n",
       "43                  20              0.813268              0.818182   \n",
       "44                  31              0.805897              0.813268   \n",
       "45                  18              0.815725              0.830467   \n",
       "46                  42              0.813268              0.825553   \n",
       "47                  38              0.820639              0.810811   \n",
       "\n",
       "    split2_test_accuracy  split3_test_accuracy  split4_test_accuracy  \\\n",
       "0               0.842752              0.842752              0.839901   \n",
       "1               0.783784              0.820639              0.790640   \n",
       "2               0.835381              0.830467              0.834975   \n",
       "3               0.791155              0.808354              0.812808   \n",
       "4               0.835381              0.832924              0.832512   \n",
       "5               0.791155              0.815725              0.780788   \n",
       "6               0.825553              0.825553              0.817734   \n",
       "7               0.818182              0.830467              0.812808   \n",
       "8               0.828010              0.813268              0.830049   \n",
       "9               0.828010              0.840295              0.837438   \n",
       "10              0.818182              0.832924              0.790640   \n",
       "11              0.830467              0.810811              0.834975   \n",
       "12              0.845209              0.837838              0.834975   \n",
       "13              0.788698              0.818182              0.810345   \n",
       "14              0.830467              0.832924              0.842365   \n",
       "15              0.810811              0.808354              0.783251   \n",
       "16              0.835381              0.832924              0.827586   \n",
       "17              0.786241              0.808354              0.795567   \n",
       "18              0.823096              0.808354              0.807882   \n",
       "19              0.823096              0.828010              0.820197   \n",
       "20              0.815725              0.815725              0.810345   \n",
       "21              0.820639              0.820639              0.834975   \n",
       "22              0.820639              0.815725              0.827586   \n",
       "23              0.823096              0.830467              0.842365   \n",
       "24              0.810811              0.840295              0.839901   \n",
       "25              0.815725              0.835381              0.834975   \n",
       "26              0.805897              0.830467              0.844828   \n",
       "27              0.810811              0.823096              0.832512   \n",
       "28              0.825553              0.837838              0.837438   \n",
       "29              0.796069              0.828010              0.837438   \n",
       "30              0.798526              0.828010              0.815271   \n",
       "31              0.810811              0.825553              0.825123   \n",
       "32              0.798526              0.810811              0.805419   \n",
       "33              0.815725              0.793612              0.834975   \n",
       "34              0.810811              0.818182              0.812808   \n",
       "35              0.800983              0.837838              0.802956   \n",
       "36              0.823096              0.837838              0.817734   \n",
       "37              0.810811              0.837838              0.812808   \n",
       "38              0.828010              0.828010              0.834975   \n",
       "39              0.815725              0.828010              0.830049   \n",
       "40              0.818182              0.828010              0.844828   \n",
       "41              0.808354              0.830467              0.832512   \n",
       "42              0.805897              0.815725              0.805419   \n",
       "43              0.810811              0.828010              0.844828   \n",
       "44              0.805897              0.845209              0.830049   \n",
       "45              0.813268              0.837838              0.830049   \n",
       "46              0.800983              0.808354              0.802956   \n",
       "47              0.810811              0.803440              0.822660   \n",
       "\n",
       "    mean_test_accuracy  std_test_accuracy  rank_test_accuracy  \n",
       "0             0.838251           0.004734                   1  \n",
       "1             0.799897           0.015176                  45  \n",
       "2             0.835300           0.003112                   6  \n",
       "3             0.805313           0.007422                  43  \n",
       "4             0.834807           0.004268                   8  \n",
       "5             0.799892           0.012747                  46  \n",
       "6             0.821532           0.006382                  31  \n",
       "7             0.823987           0.007195                  25  \n",
       "8             0.826944           0.008842                  17  \n",
       "9             0.833335           0.005628                  10  \n",
       "10            0.817588           0.015592                  34  \n",
       "11            0.823998           0.009932                  24  \n",
       "12            0.837757           0.004957                   2  \n",
       "13            0.800398           0.011693                  44  \n",
       "14            0.835303           0.007377                   5  \n",
       "15            0.796454           0.017202                  47  \n",
       "16            0.830874           0.002956                  13  \n",
       "17            0.783683           0.023361                  48  \n",
       "18            0.817596           0.009471                  33  \n",
       "19            0.824482           0.004082                  23  \n",
       "20            0.823494           0.013404                  26  \n",
       "21            0.826455           0.005387                  18  \n",
       "22            0.821537           0.009345                  30  \n",
       "23            0.824984           0.010797                  21  \n",
       "24            0.836285           0.013755                   3  \n",
       "25            0.834317           0.013595                   9  \n",
       "26            0.829899           0.013498                  14  \n",
       "27            0.827928           0.013064                  16  \n",
       "28            0.836284           0.005712                   4  \n",
       "29            0.824982           0.016340                  22  \n",
       "30            0.814651           0.010487                  38  \n",
       "31            0.822518           0.005856                  29  \n",
       "32            0.810715           0.009755                  41  \n",
       "33            0.814661           0.014176                  37  \n",
       "34            0.816616           0.006192                  35  \n",
       "35            0.815628           0.014660                  36  \n",
       "36            0.832343           0.010007                  12  \n",
       "37            0.828901           0.014562                  15  \n",
       "38            0.832843           0.006746                  11  \n",
       "39            0.826452           0.010111                  19  \n",
       "40            0.834813           0.010396                   7  \n",
       "41            0.822522           0.008750                  28  \n",
       "42            0.811698           0.005831                  40  \n",
       "43            0.823020           0.012394                  27  \n",
       "44            0.820064           0.015360                  32  \n",
       "45            0.825469           0.009410                  20  \n",
       "46            0.810223           0.008786                  42  \n",
       "47            0.813672           0.007077                  39  \n",
       "\n",
       "[48 rows x 43 columns]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(grid_search.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Pipeline(steps=[('vect', CountVectorizer(max_df=0.5)),\n",
       "                 ('tfidf', TfidfTransformer(norm='l1')),\n",
       "                 ('clf', SGDClassifier(alpha=1e-06, max_iter=20))]),\n",
       " 0.8247948617768188)"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_ , grid_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for ExtraTreesClassifier.\n",
      "Fitting 10 folds for each of 2 candidates, totalling 20 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:   11.9s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for RandomForestClassifier.\n",
      "Fitting 10 folds for each of 2 candidates, totalling 20 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    5.3s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for AdaBoostClassifier.\n",
      "Fitting 10 folds for each of 2 candidates, totalling 20 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:   13.2s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for GradientBoostingClassifier.\n",
      "Fitting 10 folds for each of 4 candidates, totalling 40 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  40 out of  40 | elapsed:  2.6min finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for SVC.\n",
      "Fitting 10 folds for each of 6 candidates, totalling 60 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   52.5s\n",
      "[Parallel(n_jobs=-1)]: Done  60 out of  60 | elapsed:  1.4min finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for MultinomialNB.\n",
      "Fitting 10 folds for each of 3 candidates, totalling 30 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed:    0.3s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running GridSearchCV for KNN.\n",
      "Fitting 10 folds for each of 4 candidates, totalling 40 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  40 out of  40 | elapsed:    0.7s finished\n"
     ]
    }
   ],
   "source": [
    "from estimator_select import EstimatorSelectionHelper\n",
    "from sklearn.ensemble import (ExtraTreesClassifier, RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier)\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "vect = TfidfVectorizer()\n",
    "X2 = vect.fit_transform(newsgroups_train.data)\n",
    "\n",
    "models1 = {\n",
    "        'ExtraTreesClassifier': ExtraTreesClassifier(),\n",
    "        'RandomForestClassifier': RandomForestClassifier(),\n",
    "        'AdaBoostClassifier': AdaBoostClassifier(),\n",
    "        'GradientBoostingClassifier': GradientBoostingClassifier(),\n",
    "        'SVC': SVC(),\n",
    "        'MultinomialNB': MultinomialNB(),\n",
    "        'KNN': KNeighborsClassifier()\n",
    "    }\n",
    "\n",
    "params1 = {\n",
    "        'ExtraTreesClassifier': {'n_estimators': [16, 32]},\n",
    "        'RandomForestClassifier': {'n_estimators': [16, 32]},\n",
    "        'AdaBoostClassifier': {'n_estimators': [16, 32]},\n",
    "        'GradientBoostingClassifier': {'n_estimators': [16, 32], 'learning_rate': [0.8, 1.0]},\n",
    "        'SVC': [\n",
    "            {'kernel': ['linear'], 'C': [1, 10]},\n",
    "            {'kernel': ['rbf'], 'C': [1, 10], 'gamma': [0.001, 0.0001]},\n",
    "        ],\n",
    "        'MultinomialNB': {'alpha': [0.1, 0.01, 0.001]},\n",
    "        'KNN': {'n_neighbors': [1,3,5,7], 'metric': ['cosine'] }\n",
    "    }\n",
    "\n",
    "scoring = {\n",
    "        'accuracy',\n",
    "        'f1_micro',\n",
    "        'f1_macro',\n",
    "        'f1_weighted'\n",
    "    }\n",
    "\n",
    "helper1 = EstimatorSelectionHelper(models1, params1)\n",
    "helper1.fit(X2, newsgroups_train.target, scoring=scoring, n_jobs=-1, cv=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_n_estimators</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_f1_micro</th>\n",
       "      <th>split1_test_f1_micro</th>\n",
       "      <th>split2_test_f1_micro</th>\n",
       "      <th>split3_test_f1_micro</th>\n",
       "      <th>...</th>\n",
       "      <th>std_test_f1_macro</th>\n",
       "      <th>rank_test_f1_macro</th>\n",
       "      <th>estimator</th>\n",
       "      <th>param_learning_rate</th>\n",
       "      <th>param_C</th>\n",
       "      <th>param_kernel</th>\n",
       "      <th>param_gamma</th>\n",
       "      <th>param_alpha</th>\n",
       "      <th>param_metric</th>\n",
       "      <th>param_n_neighbors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.462394</td>\n",
       "      <td>0.285814</td>\n",
       "      <td>0.019146</td>\n",
       "      <td>0.002338</td>\n",
       "      <td>16</td>\n",
       "      <td>{'n_estimators': 16}</td>\n",
       "      <td>0.730392</td>\n",
       "      <td>0.710784</td>\n",
       "      <td>0.715686</td>\n",
       "      <td>0.671569</td>\n",
       "      <td>...</td>\n",
       "      <td>0.025448</td>\n",
       "      <td>2</td>\n",
       "      <td>ExtraTreesClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.666805</td>\n",
       "      <td>0.556729</td>\n",
       "      <td>0.016941</td>\n",
       "      <td>0.002848</td>\n",
       "      <td>32</td>\n",
       "      <td>{'n_estimators': 32}</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.720588</td>\n",
       "      <td>0.740196</td>\n",
       "      <td>0.740196</td>\n",
       "      <td>...</td>\n",
       "      <td>0.024955</td>\n",
       "      <td>1</td>\n",
       "      <td>ExtraTreesClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.956224</td>\n",
       "      <td>0.122629</td>\n",
       "      <td>0.011058</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>16</td>\n",
       "      <td>{'n_estimators': 16}</td>\n",
       "      <td>0.622549</td>\n",
       "      <td>0.647059</td>\n",
       "      <td>0.622549</td>\n",
       "      <td>0.647059</td>\n",
       "      <td>...</td>\n",
       "      <td>0.033185</td>\n",
       "      <td>2</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.481530</td>\n",
       "      <td>0.046704</td>\n",
       "      <td>0.016678</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>32</td>\n",
       "      <td>{'n_estimators': 32}</td>\n",
       "      <td>0.725490</td>\n",
       "      <td>0.720588</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.696078</td>\n",
       "      <td>...</td>\n",
       "      <td>0.025999</td>\n",
       "      <td>1</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.869839</td>\n",
       "      <td>0.164228</td>\n",
       "      <td>0.016074</td>\n",
       "      <td>0.003906</td>\n",
       "      <td>16</td>\n",
       "      <td>{'n_estimators': 16}</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>0.509804</td>\n",
       "      <td>0.490196</td>\n",
       "      <td>0.563725</td>\n",
       "      <td>...</td>\n",
       "      <td>0.032408</td>\n",
       "      <td>2</td>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.342742</td>\n",
       "      <td>0.988452</td>\n",
       "      <td>0.029473</td>\n",
       "      <td>0.009303</td>\n",
       "      <td>32</td>\n",
       "      <td>{'n_estimators': 32}</td>\n",
       "      <td>0.656863</td>\n",
       "      <td>0.573529</td>\n",
       "      <td>0.553922</td>\n",
       "      <td>0.549020</td>\n",
       "      <td>...</td>\n",
       "      <td>0.039440</td>\n",
       "      <td>1</td>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15.776915</td>\n",
       "      <td>0.405385</td>\n",
       "      <td>0.010098</td>\n",
       "      <td>0.000645</td>\n",
       "      <td>16</td>\n",
       "      <td>{'learning_rate': 0.8, 'n_estimators': 16}</td>\n",
       "      <td>0.715686</td>\n",
       "      <td>0.676471</td>\n",
       "      <td>0.661765</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>...</td>\n",
       "      <td>0.034531</td>\n",
       "      <td>2</td>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>0.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>32.970417</td>\n",
       "      <td>1.218046</td>\n",
       "      <td>0.010846</td>\n",
       "      <td>0.000849</td>\n",
       "      <td>32</td>\n",
       "      <td>{'learning_rate': 0.8, 'n_estimators': 32}</td>\n",
       "      <td>0.671569</td>\n",
       "      <td>0.710784</td>\n",
       "      <td>0.696078</td>\n",
       "      <td>0.715686</td>\n",
       "      <td>...</td>\n",
       "      <td>0.035063</td>\n",
       "      <td>1</td>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>0.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>17.689266</td>\n",
       "      <td>0.444532</td>\n",
       "      <td>0.011198</td>\n",
       "      <td>0.001233</td>\n",
       "      <td>16</td>\n",
       "      <td>{'learning_rate': 1.0, 'n_estimators': 16}</td>\n",
       "      <td>0.700980</td>\n",
       "      <td>0.681373</td>\n",
       "      <td>0.637255</td>\n",
       "      <td>0.700980</td>\n",
       "      <td>...</td>\n",
       "      <td>0.026341</td>\n",
       "      <td>4</td>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>29.268113</td>\n",
       "      <td>5.704844</td>\n",
       "      <td>0.009681</td>\n",
       "      <td>0.001979</td>\n",
       "      <td>32</td>\n",
       "      <td>{'learning_rate': 1.0, 'n_estimators': 32}</td>\n",
       "      <td>0.715686</td>\n",
       "      <td>0.651961</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>...</td>\n",
       "      <td>0.028321</td>\n",
       "      <td>3</td>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.905316</td>\n",
       "      <td>0.481384</td>\n",
       "      <td>0.507987</td>\n",
       "      <td>0.028179</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 1, 'kernel': 'linear'}</td>\n",
       "      <td>0.843137</td>\n",
       "      <td>0.828431</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.019811</td>\n",
       "      <td>2</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>linear</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.221086</td>\n",
       "      <td>0.257588</td>\n",
       "      <td>0.494068</td>\n",
       "      <td>0.032859</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 10, 'kernel': 'linear'}</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.818627</td>\n",
       "      <td>0.828431</td>\n",
       "      <td>0.823529</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017135</td>\n",
       "      <td>1</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>linear</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.352297</td>\n",
       "      <td>0.265537</td>\n",
       "      <td>0.698165</td>\n",
       "      <td>0.044233</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}</td>\n",
       "      <td>0.289216</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000520</td>\n",
       "      <td>3</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>rbf</td>\n",
       "      <td>0.001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.546276</td>\n",
       "      <td>0.272409</td>\n",
       "      <td>0.779521</td>\n",
       "      <td>0.077969</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}</td>\n",
       "      <td>0.289216</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000520</td>\n",
       "      <td>3</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>rbf</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.417966</td>\n",
       "      <td>0.256561</td>\n",
       "      <td>0.807117</td>\n",
       "      <td>0.067140</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}</td>\n",
       "      <td>0.289216</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000520</td>\n",
       "      <td>3</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>rbf</td>\n",
       "      <td>0.001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7.836699</td>\n",
       "      <td>1.700634</td>\n",
       "      <td>0.593195</td>\n",
       "      <td>0.143004</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}</td>\n",
       "      <td>0.289216</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000520</td>\n",
       "      <td>3</td>\n",
       "      <td>SVC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>rbf</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.012449</td>\n",
       "      <td>0.003891</td>\n",
       "      <td>0.005492</td>\n",
       "      <td>0.000892</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'alpha': 0.1}</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.843137</td>\n",
       "      <td>0.857843</td>\n",
       "      <td>0.838235</td>\n",
       "      <td>...</td>\n",
       "      <td>0.025239</td>\n",
       "      <td>3</td>\n",
       "      <td>MultinomialNB</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.011357</td>\n",
       "      <td>0.001760</td>\n",
       "      <td>0.006012</td>\n",
       "      <td>0.001324</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'alpha': 0.01}</td>\n",
       "      <td>0.852941</td>\n",
       "      <td>0.862745</td>\n",
       "      <td>0.887255</td>\n",
       "      <td>0.843137</td>\n",
       "      <td>...</td>\n",
       "      <td>0.032481</td>\n",
       "      <td>1</td>\n",
       "      <td>MultinomialNB</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.011954</td>\n",
       "      <td>0.001884</td>\n",
       "      <td>0.005458</td>\n",
       "      <td>0.001340</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'alpha': 0.001}</td>\n",
       "      <td>0.857843</td>\n",
       "      <td>0.852941</td>\n",
       "      <td>0.867647</td>\n",
       "      <td>0.828431</td>\n",
       "      <td>...</td>\n",
       "      <td>0.026514</td>\n",
       "      <td>2</td>\n",
       "      <td>MultinomialNB</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.006448</td>\n",
       "      <td>0.002194</td>\n",
       "      <td>0.115800</td>\n",
       "      <td>0.027151</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'metric': 'cosine', 'n_neighbors': 1}</td>\n",
       "      <td>0.710784</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.696078</td>\n",
       "      <td>0.735294</td>\n",
       "      <td>...</td>\n",
       "      <td>0.039518</td>\n",
       "      <td>1</td>\n",
       "      <td>KNN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cosine</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.004688</td>\n",
       "      <td>0.000746</td>\n",
       "      <td>0.084065</td>\n",
       "      <td>0.004373</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'metric': 'cosine', 'n_neighbors': 3}</td>\n",
       "      <td>0.676471</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.686275</td>\n",
       "      <td>0.715686</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042532</td>\n",
       "      <td>2</td>\n",
       "      <td>KNN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cosine</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.004569</td>\n",
       "      <td>0.000856</td>\n",
       "      <td>0.093554</td>\n",
       "      <td>0.006508</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'metric': 'cosine', 'n_neighbors': 5}</td>\n",
       "      <td>0.681373</td>\n",
       "      <td>0.691176</td>\n",
       "      <td>0.671569</td>\n",
       "      <td>0.710784</td>\n",
       "      <td>...</td>\n",
       "      <td>0.040713</td>\n",
       "      <td>4</td>\n",
       "      <td>KNN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cosine</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.004844</td>\n",
       "      <td>0.000851</td>\n",
       "      <td>0.093401</td>\n",
       "      <td>0.005622</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'metric': 'cosine', 'n_neighbors': 7}</td>\n",
       "      <td>0.691176</td>\n",
       "      <td>0.681373</td>\n",
       "      <td>0.686275</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>...</td>\n",
       "      <td>0.037518</td>\n",
       "      <td>3</td>\n",
       "      <td>KNN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cosine</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23 rows × 66 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       1.462394      0.285814         0.019146        0.002338   \n",
       "1       2.666805      0.556729         0.016941        0.002848   \n",
       "0       0.956224      0.122629         0.011058        0.000535   \n",
       "1       1.481530      0.046704         0.016678        0.001770   \n",
       "0       1.869839      0.164228         0.016074        0.003906   \n",
       "1       4.342742      0.988452         0.029473        0.009303   \n",
       "0      15.776915      0.405385         0.010098        0.000645   \n",
       "1      32.970417      1.218046         0.010846        0.000849   \n",
       "2      17.689266      0.444532         0.011198        0.001233   \n",
       "3      29.268113      5.704844         0.009681        0.001979   \n",
       "0       7.905316      0.481384         0.507987        0.028179   \n",
       "1       7.221086      0.257588         0.494068        0.032859   \n",
       "2       8.352297      0.265537         0.698165        0.044233   \n",
       "3       8.546276      0.272409         0.779521        0.077969   \n",
       "4       9.417966      0.256561         0.807117        0.067140   \n",
       "5       7.836699      1.700634         0.593195        0.143004   \n",
       "0       0.012449      0.003891         0.005492        0.000892   \n",
       "1       0.011357      0.001760         0.006012        0.001324   \n",
       "2       0.011954      0.001884         0.005458        0.001340   \n",
       "0       0.006448      0.002194         0.115800        0.027151   \n",
       "1       0.004688      0.000746         0.084065        0.004373   \n",
       "2       0.004569      0.000856         0.093554        0.006508   \n",
       "3       0.004844      0.000851         0.093401        0.005622   \n",
       "\n",
       "  param_n_estimators                                       params  \\\n",
       "0                 16                         {'n_estimators': 16}   \n",
       "1                 32                         {'n_estimators': 32}   \n",
       "0                 16                         {'n_estimators': 16}   \n",
       "1                 32                         {'n_estimators': 32}   \n",
       "0                 16                         {'n_estimators': 16}   \n",
       "1                 32                         {'n_estimators': 32}   \n",
       "0                 16   {'learning_rate': 0.8, 'n_estimators': 16}   \n",
       "1                 32   {'learning_rate': 0.8, 'n_estimators': 32}   \n",
       "2                 16   {'learning_rate': 1.0, 'n_estimators': 16}   \n",
       "3                 32   {'learning_rate': 1.0, 'n_estimators': 32}   \n",
       "0                NaN                 {'C': 1, 'kernel': 'linear'}   \n",
       "1                NaN                {'C': 10, 'kernel': 'linear'}   \n",
       "2                NaN    {'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}   \n",
       "3                NaN   {'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}   \n",
       "4                NaN   {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}   \n",
       "5                NaN  {'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}   \n",
       "0                NaN                               {'alpha': 0.1}   \n",
       "1                NaN                              {'alpha': 0.01}   \n",
       "2                NaN                             {'alpha': 0.001}   \n",
       "0                NaN       {'metric': 'cosine', 'n_neighbors': 1}   \n",
       "1                NaN       {'metric': 'cosine', 'n_neighbors': 3}   \n",
       "2                NaN       {'metric': 'cosine', 'n_neighbors': 5}   \n",
       "3                NaN       {'metric': 'cosine', 'n_neighbors': 7}   \n",
       "\n",
       "   split0_test_f1_micro  split1_test_f1_micro  split2_test_f1_micro  \\\n",
       "0              0.730392              0.710784              0.715686   \n",
       "1              0.750000              0.720588              0.740196   \n",
       "0              0.622549              0.647059              0.622549   \n",
       "1              0.725490              0.720588              0.705882   \n",
       "0              0.583333              0.509804              0.490196   \n",
       "1              0.656863              0.573529              0.553922   \n",
       "0              0.715686              0.676471              0.661765   \n",
       "1              0.671569              0.710784              0.696078   \n",
       "2              0.700980              0.681373              0.637255   \n",
       "3              0.715686              0.651961              0.666667   \n",
       "0              0.843137              0.828431              0.833333   \n",
       "1              0.833333              0.818627              0.828431   \n",
       "2              0.289216              0.294118              0.294118   \n",
       "3              0.289216              0.294118              0.294118   \n",
       "4              0.289216              0.294118              0.294118   \n",
       "5              0.289216              0.294118              0.294118   \n",
       "0              0.833333              0.843137              0.857843   \n",
       "1              0.852941              0.862745              0.887255   \n",
       "2              0.857843              0.852941              0.867647   \n",
       "0              0.710784              0.750000              0.696078   \n",
       "1              0.676471              0.705882              0.686275   \n",
       "2              0.681373              0.691176              0.671569   \n",
       "3              0.691176              0.681373              0.686275   \n",
       "\n",
       "   split3_test_f1_micro  ...  std_test_f1_macro  rank_test_f1_macro  \\\n",
       "0              0.671569  ...           0.025448                   2   \n",
       "1              0.740196  ...           0.024955                   1   \n",
       "0              0.647059  ...           0.033185                   2   \n",
       "1              0.696078  ...           0.025999                   1   \n",
       "0              0.563725  ...           0.032408                   2   \n",
       "1              0.549020  ...           0.039440                   1   \n",
       "0              0.705882  ...           0.034531                   2   \n",
       "1              0.715686  ...           0.035063                   1   \n",
       "2              0.700980  ...           0.026341                   4   \n",
       "3              0.705882  ...           0.028321                   3   \n",
       "0              0.833333  ...           0.019811                   2   \n",
       "1              0.823529  ...           0.017135                   1   \n",
       "2              0.294118  ...           0.000520                   3   \n",
       "3              0.294118  ...           0.000520                   3   \n",
       "4              0.294118  ...           0.000520                   3   \n",
       "5              0.294118  ...           0.000520                   3   \n",
       "0              0.838235  ...           0.025239                   3   \n",
       "1              0.843137  ...           0.032481                   1   \n",
       "2              0.828431  ...           0.026514                   2   \n",
       "0              0.735294  ...           0.039518                   1   \n",
       "1              0.715686  ...           0.042532                   2   \n",
       "2              0.710784  ...           0.040713                   4   \n",
       "3              0.705882  ...           0.037518                   3   \n",
       "\n",
       "                    estimator  param_learning_rate  param_C  param_kernel  \\\n",
       "0        ExtraTreesClassifier                  NaN      NaN           NaN   \n",
       "1        ExtraTreesClassifier                  NaN      NaN           NaN   \n",
       "0      RandomForestClassifier                  NaN      NaN           NaN   \n",
       "1      RandomForestClassifier                  NaN      NaN           NaN   \n",
       "0          AdaBoostClassifier                  NaN      NaN           NaN   \n",
       "1          AdaBoostClassifier                  NaN      NaN           NaN   \n",
       "0  GradientBoostingClassifier                  0.8      NaN           NaN   \n",
       "1  GradientBoostingClassifier                  0.8      NaN           NaN   \n",
       "2  GradientBoostingClassifier                    1      NaN           NaN   \n",
       "3  GradientBoostingClassifier                    1      NaN           NaN   \n",
       "0                         SVC                  NaN        1        linear   \n",
       "1                         SVC                  NaN       10        linear   \n",
       "2                         SVC                  NaN        1           rbf   \n",
       "3                         SVC                  NaN        1           rbf   \n",
       "4                         SVC                  NaN       10           rbf   \n",
       "5                         SVC                  NaN       10           rbf   \n",
       "0               MultinomialNB                  NaN      NaN           NaN   \n",
       "1               MultinomialNB                  NaN      NaN           NaN   \n",
       "2               MultinomialNB                  NaN      NaN           NaN   \n",
       "0                         KNN                  NaN      NaN           NaN   \n",
       "1                         KNN                  NaN      NaN           NaN   \n",
       "2                         KNN                  NaN      NaN           NaN   \n",
       "3                         KNN                  NaN      NaN           NaN   \n",
       "\n",
       "   param_gamma  param_alpha  param_metric  param_n_neighbors  \n",
       "0          NaN          NaN           NaN                NaN  \n",
       "1          NaN          NaN           NaN                NaN  \n",
       "0          NaN          NaN           NaN                NaN  \n",
       "1          NaN          NaN           NaN                NaN  \n",
       "0          NaN          NaN           NaN                NaN  \n",
       "1          NaN          NaN           NaN                NaN  \n",
       "0          NaN          NaN           NaN                NaN  \n",
       "1          NaN          NaN           NaN                NaN  \n",
       "2          NaN          NaN           NaN                NaN  \n",
       "3          NaN          NaN           NaN                NaN  \n",
       "0          NaN          NaN           NaN                NaN  \n",
       "1          NaN          NaN           NaN                NaN  \n",
       "2        0.001          NaN           NaN                NaN  \n",
       "3       0.0001          NaN           NaN                NaN  \n",
       "4        0.001          NaN           NaN                NaN  \n",
       "5       0.0001          NaN           NaN                NaN  \n",
       "0          NaN          0.1           NaN                NaN  \n",
       "1          NaN         0.01           NaN                NaN  \n",
       "2          NaN        0.001           NaN                NaN  \n",
       "0          NaN          NaN        cosine                  1  \n",
       "1          NaN          NaN        cosine                  3  \n",
       "2          NaN          NaN        cosine                  5  \n",
       "3          NaN          NaN        cosine                  7  \n",
       "\n",
       "[23 rows x 66 columns]"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fix it, writ in a file\n",
    "helper1.score_summary(sort_by='min_score')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}