6-UN-speeches-text-summarisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7daf3c55",
   "metadata": {},
   "source": [
    "## TEXT SUMMARISATION \n",
    "* Explore 3 different text summarisation models on a subsample of Australia's un-speech for 2015\n",
    "    * TF-IDF\n",
    "    * LSA Algoirthm\n",
    "    * TextRank Algoirthm\n",
    "* Apply the most robust of these methods for long text to summarise each of Australia's speeches from 1970-2015\n",
    "    * export the top (5) sentences for each year to either txt or word file for reading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3afeb197",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9a998a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# packages \n",
    "## other specific packages are imported in their relevant sections \n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk import tokenize\n",
    "import matplotlib.pyplot as plt\n",
    "import html\n",
    "import re\n",
    "import collections "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "68c4e537",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read pickled DF from notebook #1 (initial exploratory data visuals)\n",
    "df = pd.read_pickle('df_un_general_debates.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b6dac829",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>country_name</th>\n",
       "      <th>speaker</th>\n",
       "      <th>position</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "      <th>tokens</th>\n",
       "      <th>len_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>ALB</td>\n",
       "      <td>Albania</td>\n",
       "      <td>Mr. NAS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33: May I first convey to our President the co...</td>\n",
       "      <td>51419</td>\n",
       "      <td>[may, first, convey, president, congratulation...</td>\n",
       "      <td>4125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>ARG</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>Mr. DE PABLO PARDO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>177.\\t : It is a fortunate coincidence that pr...</td>\n",
       "      <td>29286</td>\n",
       "      <td>[fortunate, coincidence, precisely, time, unit...</td>\n",
       "      <td>2327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Mr. McMAHON</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.\\t  It is a pleasure for me to extend to y...</td>\n",
       "      <td>31839</td>\n",
       "      <td>[pleasure, extend, mr, president, warmest, con...</td>\n",
       "      <td>2545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>AUT</td>\n",
       "      <td>Austria</td>\n",
       "      <td>Mr. KIRCHSCHLAEGER</td>\n",
       "      <td>NaN</td>\n",
       "      <td>155.\\t  May I begin by expressing to Ambassado...</td>\n",
       "      <td>26616</td>\n",
       "      <td>[may, begin, expressing, ambassador, hambro, b...</td>\n",
       "      <td>2135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>BEL</td>\n",
       "      <td>Belgium</td>\n",
       "      <td>Mr. HARMEL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>176. No doubt each of us, before coming up to ...</td>\n",
       "      <td>25911</td>\n",
       "      <td>[doubt, us, coming, rostrum, wonders, usefulne...</td>\n",
       "      <td>2025</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   session  year country country_name             speaker position  \\\n",
       "0       25  1970     ALB      Albania             Mr. NAS      NaN   \n",
       "1       25  1970     ARG    Argentina  Mr. DE PABLO PARDO      NaN   \n",
       "2       25  1970     AUS    Australia         Mr. McMAHON      NaN   \n",
       "3       25  1970     AUT      Austria  Mr. KIRCHSCHLAEGER      NaN   \n",
       "4       25  1970     BEL      Belgium          Mr. HARMEL      NaN   \n",
       "\n",
       "                                                text  length  \\\n",
       "0  33: May I first convey to our President the co...   51419   \n",
       "1  177.\\t : It is a fortunate coincidence that pr...   29286   \n",
       "2  100.\\t  It is a pleasure for me to extend to y...   31839   \n",
       "3  155.\\t  May I begin by expressing to Ambassado...   26616   \n",
       "4  176. No doubt each of us, before coming up to ...   25911   \n",
       "\n",
       "                                              tokens  len_tokens  \n",
       "0  [may, first, convey, president, congratulation...        4125  \n",
       "1  [fortunate, coincidence, precisely, time, unit...        2327  \n",
       "2  [pleasure, extend, mr, president, warmest, con...        2545  \n",
       "3  [may, begin, expressing, ambassador, hambro, b...        2135  \n",
       "4  [doubt, us, coming, rostrum, wonders, usefulne...        2025  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# E.G. of data set again (same as from notebook (1) and cleaned in notebook (2)) etc.\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05cddd97",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dfe3f37",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7805fa2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>country_name</th>\n",
       "      <th>speaker</th>\n",
       "      <th>position</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "      <th>tokens</th>\n",
       "      <th>len_tokens</th>\n",
       "      <th>paragraphs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7322</th>\n",
       "      <td>70</td>\n",
       "      <td>2015</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Ms. Julie Bishop</td>\n",
       "      <td>Minister for Foreign Affairs</td>\n",
       "      <td>We meet this day at an important time for the ...</td>\n",
       "      <td>12832</td>\n",
       "      <td>[meet, day, important, time, united, nations, ...</td>\n",
       "      <td>1095</td>\n",
       "      <td>[We meet this day at an important time for the...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      session  year country country_name           speaker  \\\n",
       "7322       70  2015     AUS    Australia  Ms. Julie Bishop   \n",
       "\n",
       "                          position  \\\n",
       "7322  Minister for Foreign Affairs   \n",
       "\n",
       "                                                   text  length  \\\n",
       "7322  We meet this day at an important time for the ...   12832   \n",
       "\n",
       "                                                 tokens  len_tokens  \\\n",
       "7322  [meet, day, important, time, united, nations, ...        1095   \n",
       "\n",
       "                                             paragraphs  \n",
       "7322  [We meet this day at an important time for the...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#E.G. for the 3 different models testing, will try on: Australia & year = 2015\n",
    "df.query('country==\"AUS\" & year==2015')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "1c58b071",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_australia_2015 = df.query('country==\"AUS\" & year==2015')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0b7e8dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "## re-use earlier regex clean function from notebook #2\n",
    "\n",
    "def regex_clean(text):\n",
    "    # convert html escapes like &amp; to characters.\n",
    "    text = html.unescape(text) \n",
    "    # tags like <tab>\n",
    "    text = re.sub(r'<[^<>]*>', ' ', text)\n",
    "    # markdown URLs like [Some text](https://....)\n",
    "    text = re.sub(r'\\[([^\\[\\]]*)\\]\\([^\\(\\)]*\\)', r'\\1', text)\n",
    "    # text or code in brackets like [0]\n",
    "    text = re.sub(r'\\[[^\\[\\]]*\\]', ' ', text)\n",
    "    # standalone sequences of specials, matches &# but not #cool\n",
    "    text = re.sub(r'(?:^|\\s)[&#<>{}\\[\\]+|\\\\:-]{1,}(?:\\s|$)', ' ', text)\n",
    "    # standalone sequences of hyphens like --- or ==\n",
    "    text = re.sub(r'(?:^|\\s)[\\-=\\+]{2,}(?:\\s|$)', ' ', text)\n",
    "    # sequences of white spaces\n",
    "    text = re.sub(r'\\s+', ' ', text)\n",
    "    \n",
    "    return text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "af52fca8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jason\\AppData\\Local\\Temp/ipykernel_30644/3074268155.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_australia_2015['text_clean'] = df_australia_2015['text'].apply(regex_clean)\n"
     ]
    }
   ],
   "source": [
    "# clean text apply\n",
    "df_australia_2015['text_clean'] = df_australia_2015['text'].apply(regex_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf6d57e1",
   "metadata": {},
   "source": [
    "* full text of 2015 if desired\n",
    "    * https://www.foreignminister.gov.au/minister/julie-bishop/speech/national-statement-united-nations-general-assembly-70th-session\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70b8c313",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2b1191a7",
   "metadata": {},
   "source": [
    "## TF-IDF Summarisation\n",
    "* simplest approach is to summarise sentences based on aggregate TF-IDF values of words in that sentence\n",
    "\n",
    "\n",
    "* apply the TF-IDF vectorisation, then aggregate values to sentence level\n",
    "    * generate a scroe for each sentence as a sum of the TF-IDF values for each word in that sentence\n",
    "    * meaning a sentence with a high score contains many important words compared to other sentences in the doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "431c114f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# quickly tokenize text for the tf-idf\n",
    "sentences = tokenize.sent_tokenize(df_australia_2015.iloc[0]['text_clean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "9c96fb18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise tfidf vectorizer object and apply to tokenized sentences\n",
    "tfidfVectorizer = TfidfVectorizer()\n",
    "words_tfidf = tfidfVectorizer.fit_transform(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "feddfe49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence Number: 13:  under that article, united nations members pledge to take action individually and jointly to promote higher standards of living; solutions to international economic, social, health and related problems; and universal respect for, and observance of, human rights and fundamental freedoms. \n",
      "\n",
      "Sentence Number: 15:  the 2030 agenda for sustainable development (resolution 70/1), adopted unanimously last friday by the general assembly, is a manifestation of the 16/23 15-29595 29/09/2015 a/70/pv.18 australia pledge and a testament to the fundamental role of the organization. \n",
      "\n",
      "Sentence Number: 43:  human rights have been at the very centre of the united nations over the past 70 years, from the united nations charter in 1945 to the sustainable development goals in 2015. with the rise of terrorist groups like daesh, the continuing depredations of the north korean regime, and the persistence of forced labour and other contemporary forms of slavery, the need for the united nations to prosecute a strong human rights agenda has never been more pressing or urgent. \n",
      "\n",
      "Sentence Number: 53:  as a non-permanent 15-29595 17/23 a/70/pv.18 29/09/2015 member of the security council, we experienced first- hand the difficulties the council faces in responding to the crises and conflicts in syria, iraq, libya, yemen, south sudan, the central african republic and mali. \n",
      "\n",
      "Sentence Number: 73:  consistent with our record over decades of resettling permanently thousands of refugees, australia was pleased to announce recently that we would resettle permanently an additional 12,000 refugees from syria and iraq — registered by the office of the united nations high commissioner for refugees — particularly those who are unlikely to have a home to which they might return. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Parameter to specify number of summary sentences required\n",
    "num_summary_sentence = 5\n",
    "\n",
    "# Sort the sentences in descending order by the sum of TF-IDF values\n",
    "sent_sum = words_tfidf.sum(axis=1)\n",
    "important_sent = np.argsort(sent_sum, axis=0)[::-1]\n",
    "\n",
    "# Print three most important sentences in the order they appear in the article\n",
    "for i in range(0, len(sentences)):\n",
    "    if i in important_sent[:num_summary_sentence]:\n",
    "        print (f\"Sentence Number: {i}: \", sentences[i], '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "644b12d6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d35b657a",
   "metadata": {},
   "source": [
    "* Just for curiosity, print the 3 least important sentences\n",
    "    * they are indeed, pretty broad and not particularly useful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "265903c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence Number: 8:  terrorism today is a global threat. \n",
      "\n",
      "Sentence Number: 29:  there is an inescapable truth. \n",
      "\n",
      "Sentence Number: 80:  the role of peacekeeping is fundamental. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Parameter to specify number of summary sentences required\n",
    "num_summary_sentence = 3\n",
    "\n",
    "# Sort the sentences in descending order by the sum of TF-IDF values\n",
    "sent_sum = words_tfidf.sum(axis=1)\n",
    "important_sent = np.argsort(sent_sum, axis=0)\n",
    "\n",
    "# Print three most (least) sentences in the order they appear in the article\n",
    "for i in range(0, len(sentences)):\n",
    "    if i in important_sent[:num_summary_sentence]:\n",
    "        print (f\"Sentence Number: {i}: \", sentences[i], '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "264c819d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e3ffb0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7834cd4d",
   "metadata": {},
   "source": [
    "## LSA ALGORITHM\n",
    "* (Latent Semantic Analysis) \n",
    "* LSA assumes words that are close in meaning will occur in the same documents\n",
    "\n",
    "\n",
    "* First is representing the entire doc in a sentence-term matrix\n",
    "    * each row = sentence\n",
    "    * each column = word \n",
    "    * the value of each cell in this matrix is the word frequency, often scaled as TF-IDF weights\n",
    "    \n",
    "    \n",
    "* Objective is to reduce all words to a few topics by creating a modified representation of the sentence-term matrix\n",
    "    * to create this: apply method of non-negative matrix factorisation (NMF) that expresses this matrix as the product of 2 new decomposed matrices with fewer rows/columns\n",
    "    \n",
    "    \n",
    "* use `sumy` package; which has integrated stop words + tokenizer and stemmer functionality from nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f1afbbb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install sumy\n",
    "from sumy.parsers.plaintext import PlaintextParser\n",
    "from sumy.nlp.tokenizers import Tokenizer\n",
    "from sumy.nlp.stemmers import Stemmer\n",
    "from sumy.utils import get_stop_words\n",
    "from sumy.summarizers.lsa import LsaSummarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d15c43e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "LANGUAGE = \"english\"\n",
    "stemmer = Stemmer(LANGUAGE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "98150a23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Australia recognizes the dedicated, often courageous work of the thousands of United Nations personnel in the field who protect vulnerable citizens, deliver vital humanitarian assistance, rebuild damaged societies and support development. \n",
      "\n",
      "Under that Article, United Nations Members pledge to take action individually and jointly to promote higher standards of living; solutions to international economic, social, health and related problems; and universal respect for, and observance of, human rights and fundamental freedoms. \n",
      "\n",
      "Last week the Australian Government announced a new domestic policy, a $100 million women’s safety package, improving front-line support services and providing educational resources to help change community attitudes to violence and abuse. \n",
      "\n",
      "With the rise of terrorist groups like Daesh, the continuing depredations of the North Korean regime, and the persistence of forced labour and other contemporary forms of slavery, the need for the United Nations to prosecute a strong human rights agenda has never been more pressing or urgent. \n",
      "\n",
      "Daesh is wreaking unspeakable havoc and harm on individuals, families and communities and destroying the world’s cultural heritage in gross violation of international law and against the fundamental concepts of morality. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# quickly manually specify\n",
    "num_summary_sentence = 5\n",
    "\n",
    "# parser = PlaintextParser.from_string(article1['text'], Tokenizer(LANGUAGE))\n",
    "parser = PlaintextParser.from_string(df_australia_2015.iloc[0]['text_clean'], Tokenizer(LANGUAGE))\n",
    "\n",
    "summarizer = LsaSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "    print (str(sentence), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64696e7a",
   "metadata": {},
   "source": [
    "* There is some crossover with tf-idf, e.g. 2nd sentence here is the top tf-idf sentence\n",
    "* While tf-idf seems to have an inclination for facts, LSA seems to represent concepts better\n",
    "* LSA also picked up on a significant section of the talk about incorporating improved policies etc. for Women, which tf-idf did not pick up - perhaps because of the commoness of the gender pronouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb72960f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cd946bf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "278cc70c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "14da7c08",
   "metadata": {},
   "source": [
    "## TextRank Algorithm\n",
    "* Models a document like a graph, utilising sentences as nodes\n",
    "\n",
    "\n",
    "* A function is required to compute similarity of sentences, which build the edges (node connectors) \n",
    "    * This function weights the graph edges, the higher the similarity between sentences, the more important the edge will be between them in the graph\n",
    "    \n",
    "    \n",
    "    \n",
    "* The TextRank Algo determines the relation of similarity between sentences based on the content they both share\n",
    "    * This overlap is calculated as: number of common tokens between them\n",
    "        * Then divided by the length of each (to avoid over-promoting long sentences)\n",
    "        \n",
    "        \n",
    "        \n",
    "* The result is a dense graph representing the document\n",
    "    * The most significant sentences are selected and presented in the same order they appear in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9cf2dbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: also uses some packages imported above in the LSA section\n",
    "from sumy.summarizers.text_rank import TextRankSummarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "3943fded",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Australia recognizes the dedicated, often courageous work of the thousands of United Nations personnel in the field who protect vulnerable citizens, deliver vital humanitarian assistance, rebuild damaged societies and support development. \n",
      "\n",
      "The United Nations women, peace and security agenda has changed our collective thinking on the role of women in conflict. \n",
      "\n",
      "Human rights have been at the very centre of the United Nations over the past 70 years, from the United Nations Charter in 1945 to the Sustainable Development Goals in 2015. \n",
      "\n",
      "Australia is standing for a seat on the United Nations Human Rights Council for the 2018-2020 term. \n",
      "\n",
      "Our term on the Human Rights Council would reflect Australia’s inclusive, diverse society and build on the Australian Government’s strong domestic human rights agenda. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_summary_sentence = 5 \n",
    "\n",
    "# implement with sumy package: textrank implementation \n",
    "parser = PlaintextParser.from_string(df_australia_2015.iloc[0]['text_clean'], Tokenizer(LANGUAGE))\n",
    "summarizer = TextRankSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "# loop through summarizer object (takes # of summary sentences (int) as parameter))\n",
    "for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "    print (str(sentence), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1188bb52",
   "metadata": {},
   "source": [
    "* from human perspective reading the article, seems to capture the main essence of Australia's overall speech better\n",
    "    * particularly the notion of Australia applying for a seat and what they plan to do once there\n",
    "* Although, the speech covers many different topics, so it is hard to capture all of it -- but the textrank algo does a pretty decent job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d53c354d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "001658e0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "324e565a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "314c819b",
   "metadata": {},
   "source": [
    "## Deconstructing 1 Country over Time (1970-2015)\n",
    "* Pick 1 country and extract salient points each year over duration\n",
    "    * Will use Australia for this example\n",
    "* extract top sentences for each year\n",
    "    * export those sentences to a document for a human to read and gain a quick broad overview of Australia's UN speeches content for the last 45 + years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0a93f2a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify DF with just australia for all years in data\n",
    "df_australia = df.query('country==\"AUS\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d44a17c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jason\\AppData\\Local\\Temp/ipykernel_20120/3605202302.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_australia['text_clean'] = df_australia['text'].apply(regex_clean)\n"
     ]
    }
   ],
   "source": [
    "# apply text cleaning function (if text_clean doesn't already exist)\n",
    "df_australia['text_clean'] = df_australia['text'].apply(regex_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "72e934a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>country_name</th>\n",
       "      <th>speaker</th>\n",
       "      <th>position</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "      <th>tokens</th>\n",
       "      <th>len_tokens</th>\n",
       "      <th>text_clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Mr. McMAHON</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.\\t  It is a pleasure for me to extend to y...</td>\n",
       "      <td>31839</td>\n",
       "      <td>[pleasure, extend, mr, president, warmest, con...</td>\n",
       "      <td>2545</td>\n",
       "      <td>100. It is a pleasure for me to extend to you,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>26</td>\n",
       "      <td>1971</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Mr. BOWEN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.\\t  I should like, on behalf of Australia,,...</td>\n",
       "      <td>20857</td>\n",
       "      <td>[like, behalf, australia, extend, congratulati...</td>\n",
       "      <td>1674</td>\n",
       "      <td>38. I should like, on behalf of Australia,, to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>27</td>\n",
       "      <td>1972</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Mr. Bo wen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mr. President, I should like, on behalf of the...</td>\n",
       "      <td>24700</td>\n",
       "      <td>[mr, president, like, behalf, australian, dele...</td>\n",
       "      <td>2032</td>\n",
       "      <td>Mr. President, I should like, on behalf of the...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     session  year country country_name      speaker position  \\\n",
       "2         25  1970     AUS    Australia  Mr. McMAHON      NaN   \n",
       "73        26  1971     AUS    Australia    Mr. BOWEN      NaN   \n",
       "190       27  1972     AUS    Australia  Mr. Bo wen       NaN   \n",
       "\n",
       "                                                  text  length  \\\n",
       "2    100.\\t  It is a pleasure for me to extend to y...   31839   \n",
       "73   38.\\t  I should like, on behalf of Australia,,...   20857   \n",
       "190  Mr. President, I should like, on behalf of the...   24700   \n",
       "\n",
       "                                                tokens  len_tokens  \\\n",
       "2    [pleasure, extend, mr, president, warmest, con...        2545   \n",
       "73   [like, behalf, australia, extend, congratulati...        1674   \n",
       "190  [mr, president, like, behalf, australian, dele...        2032   \n",
       "\n",
       "                                            text_clean  \n",
       "2    100. It is a pleasure for me to extend to you,...  \n",
       "73   38. I should like, on behalf of Australia,, to...  \n",
       "190  Mr. President, I should like, on behalf of the...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#E.G. \n",
    "df_australia.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649898be",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e60419a0",
   "metadata": {},
   "source": [
    "* loop through the `df_australia` and apply the text-rank algo to summarise top (5) sentences for each year\n",
    "    * append those sentences to the `collections.defaultdict` object\n",
    "    * convert dict to DF\n",
    "    * (optional) export DF to text or word document to read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "68e99783",
   "metadata": {},
   "outputs": [],
   "source": [
    "## colletions.dict method:\n",
    "sum_sents_year_df = collections.defaultdict(list)\n",
    "\n",
    "# \n",
    "num_summary_sentence = 5 \n",
    "\n",
    "for i in df_australia[['year', 'text_clean']].index:\n",
    "    # sentences = tokenize.sent_tokenize(df_australia['text_clean'][i]) # not needed for textrank actually\n",
    "    \n",
    "    # initialise textrank option + stop words etc.\n",
    "    parser = PlaintextParser.from_string(df_australia['text_clean'][i], Tokenizer(LANGUAGE))\n",
    "    summarizer = TextRankSummarizer(stemmer)\n",
    "    summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "    \n",
    "    # append top sentences to collections.dict\n",
    "    for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "        sum_sents_year_df['year'].append(df_australia['year'][i])\n",
    "        sum_sents_year_df['sent'].append(sentence)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adcba689",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3790b676",
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct DF from dict \n",
    "summary_sents_aus = pd.DataFrame.from_dict(sum_sents_year_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c91a8d12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>sent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1970</td>\n",
       "      <td>After the traumatic experience of the Second W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1970</td>\n",
       "      <td>In its twenty-five years the United Nations ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1970</td>\n",
       "      <td>In our view, the United Nations has a signific...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1970</td>\n",
       "      <td>The United Nations has a related responsibilit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1970</td>\n",
       "      <td>I have reaffirmed Australia's support for an a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>225</th>\n",
       "      <td>2015</td>\n",
       "      <td>Australia recognizes the dedicated, often cour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>2015</td>\n",
       "      <td>The United Nations women, peace and security a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>2015</td>\n",
       "      <td>Human rights have been at the very centre of t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>2015</td>\n",
       "      <td>Australia is standing for a seat on the United...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>2015</td>\n",
       "      <td>Our term on the Human Rights Council would ref...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>230 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     year                                               sent\n",
       "0    1970  After the traumatic experience of the Second W...\n",
       "1    1970  In its twenty-five years the United Nations ha...\n",
       "2    1970  In our view, the United Nations has a signific...\n",
       "3    1970  The United Nations has a related responsibilit...\n",
       "4    1970  I have reaffirmed Australia's support for an a...\n",
       "..    ...                                                ...\n",
       "225  2015  Australia recognizes the dedicated, often cour...\n",
       "226  2015  The United Nations women, peace and security a...\n",
       "227  2015  Human rights have been at the very centre of t...\n",
       "228  2015  Australia is standing for a seat on the United...\n",
       "229  2015  Our term on the Human Rights Council would ref...\n",
       "\n",
       "[230 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# E.G. of DF \n",
    "summary_sents_aus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "id": "a502c104",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Sentence: Our term on the Human Rights Council would reflect Australia’s inclusive, diverse society and build on the Australian Government’s strong domestic human rights agenda.>"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# E.G. of a sentence from the last row \n",
    "summary_sents_aus.iloc[229].sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "47c986ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# can restructure the sentences to group by year; but doesn't matter if outputing to text or word file\n",
    "\n",
    "# pd.DataFrame(summary_sents_aus.set_index('year').stack([0]), columns=['sentences'])\n",
    "# australia_summaries_df = pd.DataFrame(summary_sents_aus.set_index('year').stack([0]), columns=['sentences'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3cea8b8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# write all years summary sentences to text file\n",
    "summary_sents_aus.to_csv(r'summary_sents_aus_text.txt', header=True, index=False, sep='\\t', mode='w+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67d42863",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4caae2d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "facc27ee",
   "metadata": {},
   "source": [
    "## (Optional) Save Output to Microsoft Word Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ec387ec6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install python-docx\n",
    "\n",
    "import docx\n",
    "from docx.shared import Cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "f7f318fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify doc using docx \n",
    "doc = docx.Document('summary_sentences_australia.docx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5f91a7d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add table in shape of current DF dimensions\n",
    "t = doc.add_table(summary_sents_aus.shape[0]+1, df.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "3407d5ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# optional - refit cell width\n",
    "t.allow_autofit = False\n",
    "\n",
    "for cell in t.columns[0].cells:\n",
    "    cell.width = Cm(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "7613e5eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the header rows.\n",
    "for j in range(summary_sents_aus.shape[-1]):\n",
    "    t.cell(0,j).text = summary_sents_aus.columns[j]\n",
    "\n",
    "# add the rest of the data frame\n",
    "for i in range(summary_sents_aus.shape[0]):\n",
    "    for j in range(summary_sents_aus.shape[-1]):\n",
    "        t.cell(i+1,j).text = str(summary_sents_aus.values[i,j])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "5eba3f3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save to word document\n",
    "doc.save('summary_sentences_australia.docx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fff2d37",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50d0bdef",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db7e751a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8a446b9c",
   "metadata": {},
   "source": [
    "## Possible Future Scope of Works:\n",
    "* maybe try implementing the `rogue_scorer` to determine actual accuracy of the summarisation techniques\n",
    "    * would require a human-written summary to compare against or another way to label a proper comparison piece of text "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}