evaluation-mixed-metre.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation on mixed-metre poetry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Notebook contains the evaluation metrics for [`Rantanplan`](https://pypi.org/project/rantanplan/0.4.3/) v0.4.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last run: June 19 2020 - 08:39:51\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "print(f\"Last run: {datetime.utcnow().strftime('%B %d %Y - %H:%M:%S')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the evaluation of mixed-metre poetry we used Antonio Carvajal's annotated _Extravagante jerarquía (1958-1982)_, Madrid: Hiperión, 1983. Due to copyright issues we cannot redistribute the corpus. Therefore, the next analysis uses the outputs from both Navarro-Colorado's system and Rantanplan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q pandas numpy \"spacy<2.3.0\" spacy_affixes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash --out _\n",
    "# pip install https://github.com/explosion/spacy-models/archive/es_core_news_md-2.2.5.zip\n",
    "python -m spacy download es_core_news_md\n",
    "python -m spacy_affixes download es"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q \"rantanplan==0.4.3\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from glob import glob\n",
    "from xml.etree import ElementTree\n",
    "\n",
    "\n",
    "def clean_text(string):\n",
    "    output = string.strip()\n",
    "    # replacements = ((\"“\", '\"'), (\"”\", '\"'), (\"//\", \"\"), (\"«\", '\"'), (\"»\",'\"'))\n",
    "    replacements = ((\"“\", ''), (\"”\", ''), (\"//\", \"\"), (\"«\", ''), (\"»\",''))\n",
    "    for replacement in replacements:\n",
    "        output = output.replace(*replacement)\n",
    "    output = re.sub(r'(?is)\\s+', ' ', output)\n",
    "    output = re.sub(r\"(\\w)-(\\w)\", r\"\\1\\2\", output)  # \"Villa-nueva\" breaks Navarro-Colorado's system\n",
    "    return output\n",
    "\n",
    "def num2sym(metric, length):\n",
    "    if \"/\" in metric:\n",
    "        hemi1, hemi2 = metric.split(\"/\")\n",
    "        return num2sym(hemi1, math.floor(length / 2)) + num2sym(hemi2, math.ceil(length / 2))\n",
    "    else:\n",
    "        symbols = int(length) * [\"-\"]\n",
    "        for i in metric.split(\"-\"):\n",
    "            symbols[int(i) - 1] = \"+\"\n",
    "        return \"\".join(symbols)\n",
    "\n",
    "def load_tei(filename):\n",
    "    lines = []\n",
    "    with open(filename, \"r\") as xml:\n",
    "        contents = xml.read()\n",
    "        tree = ElementTree.fromstring(contents)\n",
    "        tags = tree.findall(\".//{http://www.tei-c.org/ns/1.0}l\")\n",
    "        for tag in tags:\n",
    "            text = clean_text(tag.text)\n",
    "            lines.append((text, tag.attrib['met']))\n",
    "    return pd.DataFrame(lines, columns=[\"line_text\", \"metrical_pattern\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corpus provided does not have any information about the texts, just metrical patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "carvajal = pd.read_csv(\"data/carvajal.csv\")\n",
    "carvajal.Length = carvajal.Length.astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "carvajal[\"MetricSymbol\"] = carvajal[[\"Metric\", \"Length\"]].apply(\n",
    "    lambda row: num2sym(row[\"Metric\"].strip(), float(row[\"Length\"])), axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leaving next cell for reference."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "!mkdir -p carvajal\n",
    "carvajal.Verses = carvajal.Verses.apply(clean_text)\n",
    "with open(\"carvajal/carvajal.txt\", \"w\") as file:\n",
    "    file.write(\"\\n\".join(carvajal.Verses.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navarro-Colorado"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next cells illustrate how to run the code with the actual Carvajal's corpus. However, we can only release the results after running Navarro-Colorado's system and not the text of the corpus."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "%%bash --err carvajal_timing --out carvajal_output\n",
    "time -p docker run -v $(pwd)/carvajal:/adso/data_in -v $(pwd)/outputs:/adso/data_out linhdpostdata/adso"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "navarro_colorado_carvajal_times = dict(pair.split(\" \") for pair in carvajal_timing.strip().split(\"\\n\")[-3:])"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "navarro_colorado_carvajal = load_tei(\"outputs/carvajal.xml\")\n",
    "navarro_colorado_stress = navarro_colorado_carvajal.line_text.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we'll use the pre-calculated results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/navarro_colorado_carvajal.txt\") as file:\n",
    "    navarro_colorado_stress = np.array(file.read().split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy on Carvajal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Navarro-Colorado on Carvajal: 0.4938\n"
     ]
    }
   ],
   "source": [
    "correct = sum(navarro_colorado_stress == carvajal.MetricSymbol.values)\n",
    "navarro_colorado_accuracy = correct / carvajal.MetricSymbol.size\n",
    "print(f\"Navarro-Colorado on Carvajal: {navarro_colorado_accuracy:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rantanplan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rantanplan import get_scansion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leaving next cell for reference"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "rantanplan_carvajal = get_scansion(\"\\n\".join(carvajal.Verses.values))\n",
    "rantanplan_carvajal_stress = np.array([line[\"rhythm\"][\"stress\"] for line in rantanplan_carvajal])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we load the results from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/rantanplan_carvajal.txt\") as file:\n",
    "    rantanplan_carvajal_stress = np.array(file.read().split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy on Carvajal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rantanplan on on Carvajal: 0.6763\n"
     ]
    }
   ],
   "source": [
    "correct = sum(rantanplan_carvajal_stress == carvajal.MetricSymbol.values)\n",
    "rantanplan_carvajal_accuracy = correct / carvajal.MetricSymbol.size\n",
    "print(f\"Rantanplan on on Carvajal: {rantanplan_carvajal_accuracy:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Model</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Navarro-Colorado</td>\n",
       "      <td>0.493833</td>\n",
       "      <td>7484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Rantanplan</td>\n",
       "      <td>0.676336</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "# Using pre-calculated time values\n",
    "display(HTML(\n",
    "    pd.DataFrame([\n",
    "        [\"Navarro-Colorado\", navarro_colorado_accuracy, 7484],\n",
    "        [\"Rantanplan\", rantanplan_carvajal_accuracy, 27]\n",
    "    ], columns=[\"Model\", \"Accuracy\", \"Time\"]).to_html(index=False)\n",
    "))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}