1_intro.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "1-intro.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JcWQTb62xuV0"
      },
      "source": [
        "# Week 1 - Retreiving and Preparing Text for Machines\n",
        "\n",
        "This week, we begin by \"begging, borrowing and stealing\" text from several\n",
        "contexts of human communication (e.g., PDFs, HTML, Word) and preparing it for\n",
        "machines to \"read\" and analyze. This notebook outlines scraping text from the\n",
        "web, PDF and Word documents. Then we detail \"spidering\" or walking\n",
        "through hyperlinks to build samples of online content, and using APIs,\n",
        "Application Programming Interfaces, provided by webservices to access their\n",
        "content. Along the way, we will use regular expressions, outlined in the\n",
        "reading, to remove unwanted formatting and ornamentation. Finally, we discuss\n",
        "various text encodings, filtering and data structures in which text can be\n",
        "placed for analysis.\n",
        "\n",
        "For this notebook we will be using the following packages:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ja7aN1vfxuV5"
      },
      "source": [
        "We made a python package just for this course: lucem_illud. If you haven't installed this package, you should run the following code first. You don't need to install the package later; all you need to do is just to import the package with: import lucem_illud. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IEshSMkexuV6"
      },
      "source": [
        "!pip install git+git://github.com/UChicago-Computational-Content-Analysis/lucem_illud.git\n",
        "#installing lucem_illud package"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TpqDkHtmygUi",
        "outputId": "2a364db2-743d-4032-c83a-7f2735931f4b"
      },
      "source": [
        "!git clone https://github.com/ming-cui/Homework-Notebooks.git"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'Homework-Notebooks'...\n",
            "remote: Enumerating objects: 4123, done.\u001b[K\n",
            "remote: Total 4123 (delta 0), reused 0 (delta 0), pack-reused 4123\u001b[K\n",
            "Receiving objects: 100% (4123/4123), 241.68 MiB | 27.67 MiB/s, done.\n",
            "Resolving deltas: 100% (147/147), done.\n",
            "Checking out files: 100% (4059/4059), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-TQAWyx-xuV6"
      },
      "source": [
        "If you're not familiar with jupyter notebook, you may wonder what the exclamation mark(!) at the beginning of the command does (or even what pip means). The exclamation mark enables us to execute Terminal commands in the notebook cells. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CbpTR75LxuV6"
      },
      "source": [
        "#Special module written for this class\n",
        "#This provides access to data and to helper functions from previous weeks\n",
        "import lucem_illud #pip install git+git://github.com/UChicago-Computational-Content-Analysis/lucem_illud.git\n",
        "\n",
        "#All these packages need to be installed from pip\n",
        "import requests #for http requests\n",
        "import bs4 #called `beautifulsoup4`, an html parser\n",
        "import pandas #gives us DataFrames\n",
        "import docx #reading MS doc files, install as `python-docx`\n",
        "\n",
        "#Stuff for pdfs\n",
        "#Install as `pdfminer2`\n",
        "import pdfminer.pdfinterp\n",
        "import pdfminer.converter\n",
        "import pdfminer.layout\n",
        "import pdfminer.pdfpage\n",
        "\n",
        "#These come with Python\n",
        "import re #for regexs\n",
        "import urllib.parse #For joining urls\n",
        "import io #for making http requests look like files\n",
        "import json #For Tumblr API responses\n",
        "import os.path #For checking if files exist\n",
        "import os #For making directories"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LU-FBdsjxuV7"
      },
      "source": [
        "We will also be working on the following files/urls"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H03aPOjUxuV7"
      },
      "source": [
        "wikipedia_base_url = 'https://en.wikipedia.org'\n",
        "wikipedia_content_analysis = 'https://en.wikipedia.org/wiki/Content_analysis'\n",
        "content_analysis_save = 'wikipedia_content_analysis.html'\n",
        "example_text_file = 'sometextfile.txt'\n",
        "information_extraction_pdf = 'https://github.com/Computational-Content-Analysis-2018/Data-Files/raw/master/1-intro/Content%20Analysis%2018.pdf'\n",
        "example_docx = 'https://github.com/Computational-Content-Analysis-2018/Data-Files/raw/master/1-intro/macs6000_connecting_to_midway.docx'\n",
        "example_docx_save = 'example.docx'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvHcX7eFxuV7"
      },
      "source": [
        "# Scraping\n",
        "\n",
        "Before we can start analyzing content we need to obtain it. Sometimes it will be\n",
        "provided to us from a pre-curated text archive, but sometimes we will need to\n",
        "download it. As a starting example we will attempt to download the wikipedia\n",
        "page on content analysis. The page is located at [https://en.wikipedia.org/wiki/\n",
        "Content_analysis](https://en.wikipedia.org/wiki/Content_analysis) so lets start\n",
        "with that.\n",
        "\n",
        "We can do this by making an HTTP GET request to that url, a GET request is\n",
        "simply a request to the server to provide the contents given by some url. The\n",
        "other request we will be using in this class is called a POST request and\n",
        "requests the server to take some content we provide. While the Python standard\n",
        "library does have the ability do make GET requests we will be using the\n",
        "[_requests_](http://docs.python-requests.org/en/master/) package as it is _'the\n",
        "only Non-GMO HTTP library for Python'_...also it provides a nicer interface."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-1p0lqqYxuV8",
        "outputId": "78232811-1613-49bd-f1f5-b8e20833c30b"
      },
      "source": [
        "#wikipedia_content_analysis = 'https://en.wikipedia.org/wiki/Content_analysis'\n",
        "requests.get(wikipedia_content_analysis)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Response [200]>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMGeler4xuV8"
      },
      "source": [
        "`'Response [200]'` means the server responded with what we asked for. If you get\n",
        "another number (e.g. 404) it likely means there was some kind of error, these\n",
        "codes are called HTTP response codes and a list of them can be found\n",
        "[here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). The response\n",
        "object contains all the data the server sent including the website's contents\n",
        "and the HTTP header. We are interested in the contents which we can access with\n",
        "the `.text` attribute."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "io_ZAa2kxuV8",
        "outputId": "7106546a-55e9-467c-b5e3-2705f5fa3cda"
      },
      "source": [
        "wikiContentRequest = requests.get(wikipedia_content_analysis)\n",
        "print(wikiContentRequest.text[:1000])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<!DOCTYPE html>\n",
            "<html class=\"client-nojs\" lang=\"en\" dir=\"ltr\">\n",
            "<head>\n",
            "<meta charset=\"UTF-8\"/>\n",
            "<title>Content analysis - Wikipedia</title>\n",
            "<script>document.documentElement.className=\"client-js\";RLCONF={\"wgBreakFrames\":!1,\"wgSeparatorTransformTable\":[\"\",\"\"],\"wgDigitTransformTable\":[\"\",\"\"],\"wgDefaultDateFormat\":\"dmy\",\"wgMonthNames\":[\"\",\"January\",\"February\",\"March\",\"April\",\"May\",\"June\",\"July\",\"August\",\"September\",\"October\",\"November\",\"December\"],\"wgRequestId\":\"YE33wXUQYVbmJhxrAGUGlAAAAEs\",\"wgCSPNonce\":!1,\"wgCanonicalNamespace\":\"\",\"wgCanonicalSpecialPageName\":!1,\"wgNamespaceNumber\":0,\"wgPageName\":\"Content_analysis\",\"wgTitle\":\"Content analysis\",\"wgCurRevisionId\":991020438,\"wgRevisionId\":991020438,\"wgArticleId\":473317,\"wgIsArticle\":!0,\"wgIsRedirect\":!1,\"wgAction\":\"view\",\"wgUserName\":null,\"wgUserGroups\":[\"*\"],\"wgCategories\":[\"Articles needing expert attention with no reason or talk parameter\",\"Articles needing expert attention from April 2008\",\"All articles needing expert attention\",\"Sociology\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4oUYGMtExuV8"
      },
      "source": [
        "This is not what we were looking for, because it is the start of the HTML that\n",
        "makes up the website. This is HTML and is meant to be read by computers. Luckily\n",
        "we have a computer to parse it for us. To do the parsing we will use [_Beautiful\n",
        "Soup_](https://www.crummy.com/software/BeautifulSoup/) which is a better parser\n",
        "than the one in the standard library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W29B1scyxuV9"
      },
      "source": [
        "But before we proceed to Beautiful Soup, a digression about Python syntax, especially about objects and functions.\n",
        "For those who are not familiar with the syntax of python (or, if you're familiar with R programming), you might wonder what requests.get or wikiContentRequest.text mean. To understand this, you need to first understand what objects are. You may have heard that Python is an object oriented programming language (unlike the procedure oriented programming language, an example of which is R). Object is a set of variables (or, data) and functions into which you pass your data. So, in object oriented programming languages, like python, variables and functions are bunleded into objects.\n",
        "\n",
        "For example, let's look at wikiContentRequest. We use dir() function, which returns the list of attributes and functions of objects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XTvhCYOfxuV9",
        "outputId": "57e67387-1c5a-4823-da35-6d2e8e1028ef"
      },
      "source": [
        " dir(wikiContentRequest)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['__attrs__',\n",
              " '__bool__',\n",
              " '__class__',\n",
              " '__delattr__',\n",
              " '__dict__',\n",
              " '__dir__',\n",
              " '__doc__',\n",
              " '__enter__',\n",
              " '__eq__',\n",
              " '__exit__',\n",
              " '__format__',\n",
              " '__ge__',\n",
              " '__getattribute__',\n",
              " '__getstate__',\n",
              " '__gt__',\n",
              " '__hash__',\n",
              " '__init__',\n",
              " '__init_subclass__',\n",
              " '__iter__',\n",
              " '__le__',\n",
              " '__lt__',\n",
              " '__module__',\n",
              " '__ne__',\n",
              " '__new__',\n",
              " '__nonzero__',\n",
              " '__reduce__',\n",
              " '__reduce_ex__',\n",
              " '__repr__',\n",
              " '__setattr__',\n",
              " '__setstate__',\n",
              " '__sizeof__',\n",
              " '__str__',\n",
              " '__subclasshook__',\n",
              " '__weakref__',\n",
              " '_content',\n",
              " '_content_consumed',\n",
              " '_next',\n",
              " 'apparent_encoding',\n",
              " 'close',\n",
              " 'connection',\n",
              " 'content',\n",
              " 'cookies',\n",
              " 'elapsed',\n",
              " 'encoding',\n",
              " 'headers',\n",
              " 'history',\n",
              " 'is_permanent_redirect',\n",
              " 'is_redirect',\n",
              " 'iter_content',\n",
              " 'iter_lines',\n",
              " 'json',\n",
              " 'links',\n",
              " 'next',\n",
              " 'ok',\n",
              " 'raise_for_status',\n",
              " 'raw',\n",
              " 'reason',\n",
              " 'request',\n",
              " 'status_code',\n",
              " 'text',\n",
              " 'url']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6E6hFAaTxuV-"
      },
      "source": [
        "There's 'text' here. We used 'wikiContentRequest.text' to access 'text.' In other words, we use .(dot notation) to access functions from objects. wikiContentRequest has a set of functions, as shown above, and we used 'wikiContentRequest.text' to access one of them. By the way, dot notations do not necessarily refer to functions--it refers to anything that the entity contains. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yetT401-xuV-"
      },
      "source": [
        "Moving on to the next step: BeautifulSoup, a Python library which extracts data from HTML and XML, and transforms HTML files into Python objects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UGli8YxYxuV-",
        "outputId": "27579fff-5d21-4e21-ad56-aebaaa73694a"
      },
      "source": [
        "wikiContentSoup = bs4.BeautifulSoup(wikiContentRequest.text, 'html.parser')\n",
        "print(wikiContentSoup.text[:200])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Content analysis - Wikipedia\n",
            "document.documentElement.className=\"client-js\";RLCONF={\"wgBreakFrames\":!1,\"wgSeparatorTransformTable\":[\"\",\"\"],\"wgDigitTransformTable\":[\"\",\"\"],\"wgDefaultDateFormat\":\"dm\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZTsG1V8lxuV-"
      },
      "source": [
        "This is better but there's still random whitespace and we have more than just\n",
        "the text of the article. This is because what we requested is the whole webpage,\n",
        "not just the text for the article.\n",
        "\n",
        "We want to extract only the text we care about, and in order to do this we will\n",
        "need to inspect the html. One way to do this is simply to go to the website with\n",
        "a browser and use its inspection or view source tool. If javascript or other\n",
        "dynamic loading occurs on the page, however, it is likely that what Python\n",
        "receives is not what you will see, so we will need to inspect what Python\n",
        "receives. To do this we can save the html `requests` obtained."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2dChASnIxuV_"
      },
      "source": [
        "#content_analysis_save = 'wikipedia_content_analysis.html'\n",
        "\n",
        "with open(content_analysis_save, mode='w', encoding='utf-8') as f:\n",
        "    f.write(wikiContentRequest.text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sRWzWDNVxuV_"
      },
      "source": [
        "open() is a function which literally opens and returns the file. This function has multiple modes, and, here, we used mode as 'w', which means: open a file for writing. And then, we use 'write' function to write on the empty file (content_analysis_save) that we created using open(content_analysis_save, mode='w', encoding='utf-8').} What did we write on this file? The text we got from wikiContentRequest.text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jPToT-0SxuV_"
      },
      "source": [
        "Now let's open the file (`wikipedia_content_analysis.html`) we just created with\n",
        "a web browser. It should look sort of like the original but without the images\n",
        "and formatting.\n",
        "\n",
        "As there is very little standardization on structuring webpages, figuring out\n",
        "how best to extract what you want is an art. Looking at this page it looks like\n",
        "all the main textual content is inside `<p>`(paragraph) tags within the `<body>`\n",
        "tag. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5oYOSrkyxuV_",
        "outputId": "60c9973d-b572-4940-ab0b-9326b951fc38"
      },
      "source": [
        "contentPTags = wikiContentSoup.body.findAll('p')\n",
        "for pTag in contentPTags[:3]:\n",
        "    print(pTag.text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Content analysis is the study of documents and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic manner.[1] One of the key advantages of using content analysis to analyse social phenomena is its non-invasive nature, in contrast to simulating social experiences or collecting survey answers.\n",
            "\n",
            "Practices and philosophies of content analysis vary between academic disciplines. They all involve systematic reading or observation of texts or artifacts which are assigned labels (sometimes called codes) to indicate the presence of interesting, meaningful pieces of content.[2][3] By systematically labeling the content of a set of texts, researchers can analyse patterns of content quantitatively using statistical methods, or use qualitative methods to analyse meanings of content within texts.\n",
            "\n",
            "Computers are increasingly used in content analysis to automate the labeling (or coding) of documents. Simple computational techniques can provide descriptive data such as word frequencies and document lengths. Machine learning classifiers can greatly increase the number of texts that can be labeled, but the scientific utility of doing so is a matter of debate. Further, numerous computer-aided text analysis (CATA) computer programs are available that analyze text for pre-determined linguistic, semantic, and psychological characteristics.[4]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K_4ehtEVEiJw",
        "outputId": "f7155c06-721f-4e9e-f0ba-1e9bf0d66881"
      },
      "source": [
        "contentPTags"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<p><b>Content analysis</b> is the study of <a href=\"/wiki/Document\" title=\"Document\">documents</a> and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic manner.<sup class=\"reference\" id=\"cite_ref-1\"><a href=\"#cite_note-1\">[1]</a></sup> One of the key advantages of using content analysis to analyse social phenomena is its non-invasive nature, in contrast to simulating social experiences or collecting survey answers.\n",
              " </p>,\n",
              " <p>Practices and philosophies of content analysis vary between academic disciplines. They all involve systematic reading or observation of <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a> or artifacts which are <a href=\"/wiki/Coding_(social_sciences)\" title=\"Coding (social sciences)\">assigned labels (sometimes called codes)</a> to indicate the presence of interesting, <a href=\"/wiki/Semantics\" title=\"Semantics\">meaningful</a> pieces of content.<sup class=\"reference\" id=\"cite_ref-2\"><a href=\"#cite_note-2\">[2]</a></sup><sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-0\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup> By systematically labeling the content of a set of <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a>, researchers can analyse patterns of content <a href=\"/wiki/Quantitative_research\" title=\"Quantitative research\">quantitatively</a> using <a href=\"/wiki/Statistics\" title=\"Statistics\">statistical methods</a>, or use <a href=\"/wiki/Qualitative_research\" title=\"Qualitative research\">qualitative</a> methods to analyse meanings of content within <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a>.\n",
              " </p>,\n",
              " <p>Computers are increasingly used in content analysis to automate the labeling (or coding) of documents. Simple computational techniques can provide descriptive data such as word frequencies and document lengths. <a href=\"/wiki/Machine_learning\" title=\"Machine learning\">Machine learning</a> classifiers can greatly increase the number of texts that can be labeled, but the scientific utility of doing so is a matter of debate. Further, numerous computer-aided text analysis (CATA) computer programs are available that analyze text for pre-determined linguistic, semantic, and psychological characteristics.<sup class=\"reference\" id=\"cite_ref-Neuendorf2016_4-0\"><a href=\"#cite_note-Neuendorf2016-4\">[4]</a></sup>\n",
              " </p>,\n",
              " <p>Content analysis is best understood as a broad family of techniques. Effective researchers choose techniques that best help them answer their substantive questions. That said, according to <a href=\"/wiki/Klaus_Krippendorff\" title=\"Klaus Krippendorff\">Klaus Krippendorff</a>, six questions must be addressed in every content analysis:<sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-0\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
              " </p>,\n",
              " <p>The simplest and most objective form of content analysis considers unambiguous characteristics of the text such as <a class=\"mw-redirect\" href=\"/wiki/Word_frequencies\" title=\"Word frequencies\">word frequencies</a>, the page area taken by a newspaper column, or the duration of a <a href=\"/wiki/Radio\" title=\"Radio\">radio</a> or <a href=\"/wiki/Television\" title=\"Television\">television</a> program. Analysis of simple word frequencies is limited because the meaning of a word depends on surrounding text.  <a href=\"/wiki/Key_Word_in_Context\" title=\"Key Word in Context\">Key Word In Context</a> (KWIC) routines address this by placing words in their textual context. This helps resolve ambiguities such as those introduced by <a href=\"/wiki/Synonym\" title=\"Synonym\">synonyms</a> and <a href=\"/wiki/Homonym\" title=\"Homonym\">homonyms</a>.\n",
              " </p>,\n",
              " <p>A further step in analysis is the distinction between dictionary-based (quantitative) approaches and qualitative approaches. Dictionary-based approaches set up a list of categories derived from the frequency list of words and control the distribution of words and their respective categories over the texts. While methods in quantitative content analysis in this way transform observations of found categories into quantitative statistical data, the qualitative content analysis focuses more on the intentionality and its implications. There are strong parallels between qualitative content analysis and <a href=\"/wiki/Thematic_analysis\" title=\"Thematic analysis\">thematic analysis</a>.<sup class=\"reference\" id=\"cite_ref-6\"><a href=\"#cite_note-6\">[6]</a></sup>\n",
              " </p>,\n",
              " <p>Quantitative content analysis highlights frequency counts and objective analysis of these coded frequencies.<sup class=\"reference\" id=\"cite_ref-:03_7-0\"><a href=\"#cite_note-:03-7\">[7]</a></sup> Additionally, quantitative content analysis begins with a framed hypothesis with coding decided on before the analysis begins. These coding categories are strictly relevant to the researcher's hypothesis. Quantitative analysis also takes a deductive approach.<sup class=\"reference\" id=\"cite_ref-:12_8-0\"><a href=\"#cite_note-:12-8\">[8]</a></sup>\n",
              " </p>,\n",
              " <p><a href=\"/wiki/Siegfried_Kracauer\" title=\"Siegfried Kracauer\">Siegfried Kracauer</a> provides a critique of quantitative analysis, asserting that it oversimplifies complex communications in order to be more reliable. On the other hand, qualitative analysis deals with the intricacies of latent interpretations, whereas quantitative has a focus on manifest meanings. He also acknowledges an \"overlap\" of qualitative and quantitative content analysis.<sup class=\"reference\" id=\"cite_ref-:03_7-1\"><a href=\"#cite_note-:03-7\">[7]</a></sup> Patterns are looked at more closely in qualitative analysis, and based on the latent meanings that the researcher may find, the course of the research could be changed. It is inductive and begins with open research questions, as opposed to a hypothesis.<sup class=\"reference\" id=\"cite_ref-:12_8-1\"><a href=\"#cite_note-:12-8\">[8]</a></sup>\n",
              " </p>,\n",
              " <p>More generally, content analysis is research using the categorization and classification of speech, written text, interviews, images, or other forms of communication.  In its beginnings, using the first newspapers at the end of the 19th century, analysis was done manually by measuring the number of columns given a subject. The approach can also be traced back to a university student studying patterns in Shakespeare's literature in 1893.<sup class=\"reference\" id=\"cite_ref-9\"><a href=\"#cite_note-9\">[9]</a></sup> With the rise of common computing facilities like PCs, computer-based methods of analysis are growing in popularity.<sup class=\"reference\" id=\"cite_ref-10\"><a href=\"#cite_note-10\">[10]</a></sup><sup class=\"reference\" id=\"cite_ref-11\"><a href=\"#cite_note-11\">[11]</a></sup><sup class=\"reference\" id=\"cite_ref-12\"><a href=\"#cite_note-12\">[12]</a></sup> Answers to open ended questions, newspaper articles, political party manifestos, medical records or systematic observations in experiments can all be subject to systematic analysis of textual data.\n",
              " </p>,\n",
              " <p>By having contents of communication available in form of machine readable texts, the input is analyzed for frequencies and coded into categories for building up inferences.\n",
              " </p>,\n",
              " <p>Computer-assisted analysis can help with large, electronic data sets by cutting out time and eliminating the need for multiple human coders to establish inter-coder reliability. However, human coders can still be employed for content analysis, as they are often more able to pick out nuanced and latent meanings in text. A study  found that human coders were able to evaluate a broader range and make inferences based on latent meanings.<sup class=\"reference\" id=\"cite_ref-13\"><a href=\"#cite_note-13\">[13]</a></sup>\n",
              " </p>,\n",
              " <p>Robert Weber notes: \"To make valid inferences from the text, it is important that the classification procedure be reliable in the sense of being consistent: Different people should code the same text in the same way\".<sup class=\"reference\" id=\"cite_ref-14\"><a href=\"#cite_note-14\">[14]</a></sup> The validity, inter-coder reliability and intra-coder reliability are subject to intense methodological research efforts over long years.<sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-1\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
              " Neuendorf suggests that when human coders are used in content analysis at least two independent coders should be used. Reliability of human coding is often measured using a statistical measure of <i>inter-coder reliability</i> or \"the amount of agreement or correspondence among two or more coders\".<sup class=\"reference\" id=\"cite_ref-Neuendorf2016_4-1\"><a href=\"#cite_note-Neuendorf2016-4\">[4]</a></sup> Lacy and Riffe identify the measurement of inter-coder reliability as a strength of quantitative content analysis, arguing that, if content analysts do not measure inter-coder reliability, their data are no more reliable than the subjective impressions of a single reader.<sup class=\"reference\" id=\"cite_ref-15\"><a href=\"#cite_note-15\">[15]</a></sup>\n",
              " </p>,\n",
              " <p>There are five types of texts in content analysis:\n",
              " </p>,\n",
              " <p>Over the years, content analysis has been applied to a variety of scopes. <a href=\"/wiki/Hermeneutics\" title=\"Hermeneutics\">Hermeneutics</a> and <a href=\"/wiki/Philology\" title=\"Philology\">philology</a> have long used content analysis to interpret sacred and profane texts and, in many cases, to attribute texts' <a class=\"mw-redirect\" href=\"/wiki/Authorship\" title=\"Authorship\">authorship</a> and <a href=\"/wiki/Authentication\" title=\"Authentication\">authenticity</a>.<sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-1\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup><sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-2\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
              " </p>,\n",
              " <p>In recent times, particularly with the advent of <a href=\"/wiki/Mass_communication\" title=\"Mass communication\">mass communication</a>, content analysis has known an increasing use to deeply analyze and understand media content and <a class=\"new\" href=\"/w/index.php?title=Media_logic&amp;action=edit&amp;redlink=1\" title=\"Media logic (page does not exist)\">media logic</a>. \n",
              " The political scientist <a href=\"/wiki/Harold_Lasswell\" title=\"Harold Lasswell\">Harold Lasswell</a> formulated the core questions of content analysis in its early-mid 20th-century mainstream version: \"Who says what, to whom, why, to what extent and with what effect?\".<sup class=\"reference\" id=\"cite_ref-16\"><a href=\"#cite_note-16\">[16]</a></sup> The strong emphasis for a quantitative approach started up by Lasswell was finally carried out by another \"father\" of content analysis, <a href=\"/wiki/Bernard_Berelson\" title=\"Bernard Berelson\">Bernard Berelson</a>, who proposed a definition of content analysis which, from this point of view, is emblematic: \"a research technique for the objective, systematic and quantitative description of the manifest content of communication\".<sup class=\"reference\" id=\"cite_ref-Berelson1952_17-0\"><a href=\"#cite_note-Berelson1952-17\">[17]</a></sup>\n",
              " </p>,\n",
              " <p>Quantitative content analysis has enjoyed a renewed popularity in recent years thanks to technological advances and fruitful application in of mass communication and personal communication research. Content analysis of textual <a href=\"/wiki/Big_data\" title=\"Big data\">big data</a> produced by <a href=\"/wiki/New_media\" title=\"New media\">new media</a>, particularly <a href=\"/wiki/Social_media\" title=\"Social media\">social media</a> and <a class=\"mw-redirect\" href=\"/wiki/Mobile_devices\" title=\"Mobile devices\">mobile devices</a> has become popular. These approaches take a simplified view of language that ignores the complexity of semiosis, the process by which meaning is formed out of language. Quantitative content analysts have been criticized for limiting the scope of content analysis to simple counting, and for applying the measurement methodologies of the natural sciences without reflecting critically on their appropriateness to social science.<sup class=\"reference\" id=\"cite_ref-:0_18-0\"><a href=\"#cite_note-:0-18\">[18]</a></sup> Conversely, qualitative content analysts have been criticized for being insufficiently systematic and too impressionistic.<sup class=\"reference\" id=\"cite_ref-:0_18-1\"><a href=\"#cite_note-:0-18\">[18]</a></sup> Krippendorff argues that quantitative and qualitative approaches to content analysis tend to overlap, and that there can be no generalisable conclusion as to which approach is superior.<sup class=\"reference\" id=\"cite_ref-:0_18-2\"><a href=\"#cite_note-:0-18\">[18]</a></sup>\n",
              " </p>,\n",
              " <p>Content analysis can also be described as studying <a href=\"/wiki/Trace_evidence\" title=\"Trace evidence\">traces</a>, which are documents from past times, and artifacts, which are non-linguistic documents. Texts are understood to be produced by communication processes in a broad sense of that phrase—often gaining mean through <a href=\"/wiki/Abductive_reasoning\" title=\"Abductive reasoning\">abduction</a>.<sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-2\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup><sup class=\"reference\" id=\"cite_ref-19\"><a href=\"#cite_note-19\">[19]</a></sup>\n",
              " </p>,\n",
              " <p>Manifest content is readily understandable at its face value. Its meaning is direct. Latent content is not as overt, and requires interpretation to uncover the meaning or implication.<sup class=\"reference\" id=\"cite_ref-20\"><a href=\"#cite_note-20\">[20]</a></sup>\n",
              " </p>,\n",
              " <p>Holsti groups fifteen uses of content analysis into three basic <a class=\"mw-redirect\" href=\"/wiki/Categorisation\" title=\"Categorisation\">categories</a>:<sup class=\"reference\" id=\"cite_ref-Holsti1969_21-0\"><a href=\"#cite_note-Holsti1969-21\">[21]</a></sup>\n",
              " </p>,\n",
              " <p>He also places these uses into the context of the basic communication <a href=\"/wiki/Paradigm\" title=\"Paradigm\">paradigm</a>.\n",
              " </p>,\n",
              " <p>The following table shows fifteen uses of content analysis in terms of their general purpose, element of the communication paradigm to which they apply, and the general question they are intended to answer.\n",
              " </p>,\n",
              " <p>The process of the initial coding scheme or approach to coding is contingent on the particular content analysis approach selected. Through a directed content analysis, the scholars draft a preliminary coding scheme from pre-existing theory or assumptions. While with the conventional content analysis approach, the initial coding scheme developed from the data.\n",
              " </p>,\n",
              " <p>With either approach above, immersing oneself into the data to obtain an overall picture is recommendable for researchers to conduct. Furthermore, identifying a consistent and clear unit of coding is vital, and researchers' choices range from a single word to several paragraphs, from texts to iconic symbols. Last, constructing the relationships between codes by sorting out them within specific categories or themes.<sup class=\"reference\" id=\"cite_ref-23\"><a href=\"#cite_note-23\">[23]</a></sup>\n",
              " </p>,\n",
              " <p><br/>\n",
              " </p>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4RJbV7MlxuV_"
      },
      "source": [
        "Another excursion for those who are not familiar with programming: for loop. For loop is used to iterate over a sequence. \"ContentPTags\" contains multiple paragraphs, each of which starts and ends with `<p>`. What the \"for pTag in contentPtags[:3]\" does here is: find each paragraph in contentPTags, which, here, we limited to the first three using contentPtags[:3], and then print each paragraph. So, we have three paragraphs. By the way, you can insert `<p>` in juputer notebook!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aYc34ySmxuWA"
      },
      "source": [
        "We now have all the text from the page, split up by paragraph. If we wanted to\n",
        "get the section headers or references as well it would require a bit more work,\n",
        "but is doable.\n",
        "\n",
        "There is one more thing we might want to do before sending this text to be\n",
        "processed, remove the references indicators (`[2]`, `[3]` , etc). To do this we\n",
        "can use a short regular expression (regex)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v8secsTtxuWA",
        "outputId": "a0a47037-6531-4e0c-f331-cd501630754e"
      },
      "source": [
        "contentParagraphs = []\n",
        "for pTag in contentPTags:\n",
        "    #strings starting with r are raw so their \\'s are not modifier characters\n",
        "    #If we didn't start with r the string would be: '\\\\[\\\\d+\\\\]'\n",
        "    contentParagraphs.append(re.sub(r'\\[\\d+\\]', '', pTag.text))\n",
        "\n",
        "#convert to a DataFrame\n",
        "contentParagraphsDF = pandas.DataFrame({'paragraph-text' : contentParagraphs})\n",
        "print(contentParagraphsDF)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "0   Content analysis is the study of documents and...\n",
            "1   Practices and philosophies of content analysis...\n",
            "2   Computers are increasingly used in content ana...\n",
            "3   Content analysis is best understood as a broad...\n",
            "4   The simplest and most objective form of conten...\n",
            "5   A further step in analysis is the distinction ...\n",
            "6   Quantitative content analysis highlights frequ...\n",
            "7   Siegfried Kracauer provides a critique of quan...\n",
            "8   More generally, content analysis is research u...\n",
            "9   By having contents of communication available ...\n",
            "10  Computer-assisted analysis can help with large...\n",
            "11  Robert Weber notes: \"To make valid inferences ...\n",
            "12  There are five types of texts in content analy...\n",
            "13  Over the years, content analysis has been appl...\n",
            "14  In recent times, particularly with the advent ...\n",
            "15  Quantitative content analysis has enjoyed a re...\n",
            "16  Content analysis can also be described as stud...\n",
            "17  Manifest content is readily understandable at ...\n",
            "18  Holsti groups fifteen uses of content analysis...\n",
            "19  He also places these uses into the context of ...\n",
            "20  The following table shows fifteen uses of cont...\n",
            "21  The process of the initial coding scheme or ap...\n",
            "22  With either approach above, immersing oneself ...\n",
            "23                                                 \\n\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9RoOYm1pxuWA"
      },
      "source": [
        "Since we learned how to do for loop, you might get what we did here: using contentParagraphs = [], we made an empty list; and then, for each paragraph in contentPTags, we substituted every [\\d+\\] with '', i.e., removed every [\\d+\\], and then appended each paragraph (now without [\\d+\\]) to the empty list. As we can see, we have a dataframe, each row of which is each paragraph of contentPTags, without reference indicators. \n",
        "\n",
        "By the way, what does [\\d+\\] mean? If you are not familiar with regex, it is a way of specifying searches in text.\n",
        "A regex engine takes in the search pattern, in the above case `'\\[\\d+\\]'` and\n",
        "some string, the paragraph texts. Then it reads the input string one character\n",
        "at a time checking if it matches the search. Here the regex `'\\d'` matches\n",
        "number characters (while `'\\['` and `'\\]'` capture the braces on either side)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZTf6xCIxuWA"
      },
      "source": [
        "Now we have a `DataFrame` containing all relevant text from the page ready to be processed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nMdCS504xuWA",
        "outputId": "30d84b0b-7907-42e9-a8d2-ae23595ffab7"
      },
      "source": [
        "findNumber = r'\\d'\n",
        "regexResults = re.search(findNumber, 'not a number, not a number, numbers 2134567890, not a number')\n",
        "regexResults"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<re.Match object; span=(36, 37), match='2'>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PlQPq7oexuWB"
      },
      "source": [
        "In Python the regex package (`re`) usually returns `Match` objects (you can have\n",
        "multiple pattern hits in a a single `Match`), to get the string that matched our\n",
        "pattern we can use the `.group()` method, and as we want the first one we will\n",
        "ask for the 0'th group."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JnQL5kK_xuWB",
        "outputId": "51ca24cf-6c37-4945-d762-c2ed8d886630"
      },
      "source": [
        "print(regexResults.group(0))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIwAuLWzxuWB"
      },
      "source": [
        "That gives us the first number, if we wanted the whole block of numbers we can\n",
        "add a wildcard `'+'` which requests 1 or more instances of the preceding\n",
        "character."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qguemHW9xuWB",
        "outputId": "66d25863-9c59-4c59-e573-9ee839ce78f8"
      },
      "source": [
        "findNumbers = r'\\d+'\n",
        "regexResults = re.search(findNumbers, 'not a number, not a number, numbers 2134567890, not a number')\n",
        "print(regexResults.group(0))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2134567890\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmsjP-N_xuWB"
      },
      "source": [
        "Now we have the whole block of numbers, there are a huge number of special\n",
        "characters in regex, for the full description of Python's implementation look at\n",
        "the [re docs](https://docs.python.org/3/library/re.html) there is also a short\n",
        "[tutorial](https://docs.python.org/3/howto/regex.html#regex-howto)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-Kbejo-xuWB"
      },
      "source": [
        "# <span style=\"color:red\">Section 1</span>\n",
        "<span style=\"color:red\">Construct cells immediately below this that describe and download webcontent relating to your anticipated final project. Use beautiful soup and at least five regular expressions to extract relevant, nontrivial *chunks* of that content (e.g., cleaned sentences, paragraphs, etc.) to a pandas `Dataframe`.</span>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYuBpl1-cI-U"
      },
      "source": [
        "## Notes\n",
        "In the cells below, I scraped a webpage written by a famous organizational psychologist. The title of the article is *“Your Ideas Are Not Your Identity”: Adam Grant on How to Get Better at Changing Your Mind*. I also used five regular expressions to extract important words and their sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nctRpa1pJssI",
        "outputId": "761e8182-6e64-4674-caef-deed6d92737a"
      },
      "source": [
        "change_mind_web = \"https://behavioralscientist.org/your-ideas-are-not-your-identity-adam-grant-on-how-to-get-better-at-changing-your-mind/\"\n",
        "change_mind_web_request = requests.get(change_mind_web)\n",
        "print(change_mind_web_request.text[:1000])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<!DOCTYPE html>\n",
            "<html lang=\"en-US\" class=\"no-js\">\n",
            "<head>\n",
            "  <meta charset=\"UTF-8\">\n",
            "  <meta name=\"viewport\" content=\"width=device-width\">\n",
            "  <link rel=\"profile\" href=\"http://gmpg.org/xfn/11\">\n",
            "  <link rel=\"pingback\" href=\"https://behavioralscientist.org/xmlrpc.php\">\n",
            "  <link rel=\"stylesheet\" href=\"https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css\">\n",
            "  <link rel=\"shortcut icon\" href=\"https://behavioralscientist.org/wp-content/themes/behavioral-scientist/favicon.ico\" />\n",
            "  <link rel=\"apple-touch-icon\" sizes=\"180x180\" href=\"https://behavioralscientist.org/wp-content/themes/behavioral-scientist/apple-touch-icon-180x180-favicon.ico\" />\n",
            "  <link rel=\"icon\" type=\"image/png\" href=\"https://behavioralscientist.org/wp-content/themes/behavioral-scientist/favicon-96x96.png\" sizes=\"96x96\">\n",
            "  <!--[if lt IE 9]>\n",
            "    <script src=\"https://behavioralscientist.org/wp-content/themes/behavioral-scientist/js/html5shiv.js\"></script>\n",
            "    <script src=\"https://behavioralscientist.org/wp-content/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zKtH7JpnL5Wz"
      },
      "source": [
        "change_mind_soup = bs4.BeautifulSoup(change_mind_web_request.text, 'html.parser')\n",
        "print(change_mind_soup.text[:200])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PPDQyIrdMVxs",
        "outputId": "339d1f7c-d1ce-478e-f12e-a266df1ed94e"
      },
      "source": [
        "change_PTags = change_mind_soup.body.findAll('p')\n",
        "change_Paragraphs = []\n",
        "for pTag in change_PTags:\n",
        "    #strings starting with r are raw so their \\'s are not modifier characters\n",
        "    #If we didn't start with r the string would be: '\\\\[\\\\d+\\\\]'\n",
        "    change_Paragraphs.append(pTag.text)\n",
        "\n",
        "#convert to a DataFrame\n",
        "change_ParagraphsDF = pandas.DataFrame({'paragraph-text' : change_Paragraphs})\n",
        "print(change_ParagraphsDF)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "0           Image: Liam Speranza/Behavioral Scientist\n",
            "1   If you purchase a book using a link below, we ...\n",
            "2   Changing our minds is hard, even in the most f...\n",
            "3   Changing your mind, more often than not, requi...\n",
            "4   And yet, how freeing it is to admit we were wr...\n",
            "5   Psychologist Adam Grant wants to make that fre...\n",
            "6   It’s the idea of flexibility and how to achiev...\n",
            "7   I wanted to learn more, so I spoke with Grant ...\n",
            "8   Our conversation has been edited for length an...\n",
            "9   Evan Nesterak: You were part of a group of col...\n",
            "10  Adam Grant: I remember being a relatively earl...\n",
            "11  As Facebook skyrocketed, I started to realize ...\n",
            "12  Well, I hate to bring this up, but in Original...\n",
            "13  [Laughs] Well, Warby Parker was not my idea. I...\n",
            "14  I’ve taken two broad lessons from both of thes...\n",
            "15  I think the other thing that jumps out to me i...\n",
            "16  If somebody sees an idea, or an opportunity, o...\n",
            "17  In the e-group case, we shut down the fall of ...\n",
            "18  What I learned from that is if somebody sees a...\n",
            "19  Daniel Kahneman taught you a lesson about the ...\n",
            "20  I gave a talk on some of my research on givers...\n",
            "21  Danny is not somebody who walks around beaming...\n",
            "22  I ended up sitting down with him and asking hi...\n",
            "23  Daniel Kahneman said something to the effect o...\n",
            "24  It was a lightbulb moment for me because it re...\n",
            "25  But what’s different about Danny is he seems t...\n",
            "26  On the how part, he said for him it’s about at...\n",
            "27  That ability to detach and say, look, your ide...\n",
            "28  This past year crushed a lot of our assumption...\n",
            "29  One thing I rethought in a major way while wri...\n",
            "30  I had a student a few years ago actually who h...\n",
            "31  I’ve completely rethought that. I now think th...\n",
            "32  I do not want to have both-sides conversations...\n",
            "33  What I now believe is that we need to complexi...\n",
            "34  That that sounded encouraging at first—like, w...\n",
            "35  So the arguments are basically mixed up and yo...\n",
            "36  That’s something I’ve rethought. I do not want...\n",
            "37  Some of your research has touched on conflict ...\n",
            "38  I actually have some new data that didn’t make...\n",
            "39  I think when we encounter people who disagree ...\n",
            "40  In the gun control condition, we said think ab...\n",
            "41  I think when we encounter people who disagree ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GMAgF0WERHSC",
        "outputId": "38fc3c80-b16b-436e-9525-3c82a3f0c75f"
      },
      "source": [
        "print(change_ParagraphsDF[change_ParagraphsDF['paragraph-text'].str.contains('[mM]inds?')])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "2   Changing our minds is hard, even in the most f...\n",
            "3   Changing your mind, more often than not, requi...\n",
            "4   And yet, how freeing it is to admit we were wr...\n",
            "24  It was a lightbulb moment for me because it re...\n",
            "28  This past year crushed a lot of our assumption...\n",
            "35  So the arguments are basically mixed up and yo...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S7WppYLma6oL",
        "outputId": "40bb7238-5863-493a-d0e4-e9a87c39b75f"
      },
      "source": [
        "print(change_ParagraphsDF[change_ParagraphsDF['paragraph-text'].str.contains('[Ii]dentity')])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "3   Changing your mind, more often than not, requi...\n",
            "7   I wanted to learn more, so I spoke with Grant ...\n",
            "26  On the how part, he said for him it’s about at...\n",
            "27  That ability to detach and say, look, your ide...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J_LZxATgcbaa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "53a7aeec-90da-4219-87d8-2ab2e53bcb65"
      },
      "source": [
        "print(change_ParagraphsDF[change_ParagraphsDF['paragraph-text'].str.contains('[Cc]hanges?')])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "41  I think when we encounter people who disagree ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bo4XY1MueZ7I",
        "outputId": "18eaa479-45d0-414c-a223-a32f57ec0f81"
      },
      "source": [
        "print(change_ParagraphsDF[change_ParagraphsDF['paragraph-text'].str.contains('[Ii]deas?')])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "5   Psychologist Adam Grant wants to make that fre...\n",
            "6   It’s the idea of flexibility and how to achiev...\n",
            "9   Evan Nesterak: You were part of a group of col...\n",
            "12  Well, I hate to bring this up, but in Original...\n",
            "13  [Laughs] Well, Warby Parker was not my idea. I...\n",
            "14  I’ve taken two broad lessons from both of thes...\n",
            "16  If somebody sees an idea, or an opportunity, o...\n",
            "18  What I learned from that is if somebody sees a...\n",
            "19  Daniel Kahneman taught you a lesson about the ...\n",
            "26  On the how part, he said for him it’s about at...\n",
            "27  That ability to detach and say, look, your ide...\n",
            "29  One thing I rethought in a major way while wri...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s4Szjh_ygrl1",
        "outputId": "9b5faf66-2b82-4fbb-f305-054cd2627580"
      },
      "source": [
        "print(change_ParagraphsDF[change_ParagraphsDF['paragraph-text'].str.contains('Danny|Daniel')])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                       paragraph-text\n",
            "7   I wanted to learn more, so I spoke with Grant ...\n",
            "19  Daniel Kahneman taught you a lesson about the ...\n",
            "20  I gave a talk on some of my research on givers...\n",
            "21  Danny is not somebody who walks around beaming...\n",
            "23  Daniel Kahneman said something to the effect o...\n",
            "25  But what’s different about Danny is he seems t...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wRb09w3txuWC"
      },
      "source": [
        "\n",
        "# Spidering\n",
        "\n",
        "What if we want to to get a bunch of different pages from wikipedia. We would\n",
        "need to get the url for each of the pages we want. Typically, we want pages that\n",
        "are linked to by other pages and so we will need to parse pages and identify the\n",
        "links. Right now we will be retrieving all links in the body of the content\n",
        "analysis page.\n",
        "\n",
        "To do this we will need to find all the `<a>` (anchor) tags with `href`s\n",
        "(hyperlink references) inside of `<p>` tags. `href` can have many\n",
        "[different](http://stackoverflow.com/questions/4855168/what-is-href-and-why-is-\n",
        "it-used) [forms](https://en.wikipedia.org/wiki/Hyperlink#Hyperlinks_in_HTML) so\n",
        "dealing with them can be tricky, but generally, you will want to extract\n",
        "absolute or relative links. An absolute link is one you can follow without\n",
        "modification, while a relative link requires a base url that you will then\n",
        "append. Wikipedia uses relative urls for its internal links: below is an example\n",
        "for dealing with them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZCr55oK3xuWC",
        "outputId": "e79f4391-7ae1-4d68-b2a3-b57801c1b0c4"
      },
      "source": [
        "wikipedia_base_url = 'https://en.wikipedia.org'\n",
        "\n",
        "otherPAgeURLS = []\n",
        "#We also want to know where the links come from so we also will get:\n",
        "#the paragraph number\n",
        "#the word the link is in\n",
        "for paragraphNum, pTag in enumerate(contentPTags):\n",
        "    #we only want hrefs that link to wiki pages\n",
        "    tagLinks = pTag.findAll('a', href=re.compile('/wiki/'), class_=False)\n",
        "    for aTag in tagLinks:\n",
        "        #We need to extract the url from the <a> tag\n",
        "        relurl = aTag.get('href')\n",
        "        linkText = aTag.text\n",
        "        #wikipedia_base_url is the base we can use the urllib joining function to merge them\n",
        "        #Giving a nice structured tupe like this means we can use tuple expansion later\n",
        "        otherPAgeURLS.append((\n",
        "            urllib.parse.urljoin(wikipedia_base_url, relurl),\n",
        "            paragraphNum,\n",
        "            linkText,\n",
        "        ))\n",
        "print(otherPAgeURLS[:10])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('https://en.wikipedia.org/wiki/Document', 0, 'documents'), ('https://en.wikipedia.org/wiki/Text_(literary_theory)', 1, 'texts'), ('https://en.wikipedia.org/wiki/Coding_(social_sciences)', 1, 'assigned labels (sometimes called codes)'), ('https://en.wikipedia.org/wiki/Semantics', 1, 'meaningful'), ('https://en.wikipedia.org/wiki/Text_(literary_theory)', 1, 'texts'), ('https://en.wikipedia.org/wiki/Quantitative_research', 1, 'quantitatively'), ('https://en.wikipedia.org/wiki/Statistics', 1, 'statistical methods'), ('https://en.wikipedia.org/wiki/Qualitative_research', 1, 'qualitative'), ('https://en.wikipedia.org/wiki/Text_(literary_theory)', 1, 'texts'), ('https://en.wikipedia.org/wiki/Machine_learning', 2, 'Machine learning')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7BbE8wOIxuWC",
        "outputId": "b16e9d77-3226-4f24-9e19-571eba0d0e20"
      },
      "source": [
        "print(contentPTags)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<p><b>Content analysis</b> is the study of <a href=\"/wiki/Document\" title=\"Document\">documents</a> and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic manner.<sup class=\"reference\" id=\"cite_ref-1\"><a href=\"#cite_note-1\">[1]</a></sup> One of the key advantages of using content analysis to analyse social phenomena is its non-invasive nature, in contrast to simulating social experiences or collecting survey answers.\n",
            "</p>, <p>Practices and philosophies of content analysis vary between academic disciplines. They all involve systematic reading or observation of <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a> or artifacts which are <a href=\"/wiki/Coding_(social_sciences)\" title=\"Coding (social sciences)\">assigned labels (sometimes called codes)</a> to indicate the presence of interesting, <a href=\"/wiki/Semantics\" title=\"Semantics\">meaningful</a> pieces of content.<sup class=\"reference\" id=\"cite_ref-2\"><a href=\"#cite_note-2\">[2]</a></sup><sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-0\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup> By systematically labeling the content of a set of <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a>, researchers can analyse patterns of content <a href=\"/wiki/Quantitative_research\" title=\"Quantitative research\">quantitatively</a> using <a href=\"/wiki/Statistics\" title=\"Statistics\">statistical methods</a>, or use <a href=\"/wiki/Qualitative_research\" title=\"Qualitative research\">qualitative</a> methods to analyse meanings of content within <a href=\"/wiki/Text_(literary_theory)\" title=\"Text (literary theory)\">texts</a>.\n",
            "</p>, <p>Computers are increasingly used in content analysis to automate the labeling (or coding) of documents. Simple computational techniques can provide descriptive data such as word frequencies and document lengths. <a href=\"/wiki/Machine_learning\" title=\"Machine learning\">Machine learning</a> classifiers can greatly increase the number of texts that can be labeled, but the scientific utility of doing so is a matter of debate. Further, numerous computer-aided text analysis (CATA) computer programs are available that analyze text for pre-determined linguistic, semantic, and psychological characteristics.<sup class=\"reference\" id=\"cite_ref-Neuendorf2016_4-0\"><a href=\"#cite_note-Neuendorf2016-4\">[4]</a></sup>\n",
            "</p>, <p>Content analysis is best understood as a broad family of techniques. Effective researchers choose techniques that best help them answer their substantive questions. That said, according to <a href=\"/wiki/Klaus_Krippendorff\" title=\"Klaus Krippendorff\">Klaus Krippendorff</a>, six questions must be addressed in every content analysis:<sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-0\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
            "</p>, <p>The simplest and most objective form of content analysis considers unambiguous characteristics of the text such as <a class=\"mw-redirect\" href=\"/wiki/Word_frequencies\" title=\"Word frequencies\">word frequencies</a>, the page area taken by a newspaper column, or the duration of a <a href=\"/wiki/Radio\" title=\"Radio\">radio</a> or <a href=\"/wiki/Television\" title=\"Television\">television</a> program. Analysis of simple word frequencies is limited because the meaning of a word depends on surrounding text.  <a href=\"/wiki/Key_Word_in_Context\" title=\"Key Word in Context\">Key Word In Context</a> (KWIC) routines address this by placing words in their textual context. This helps resolve ambiguities such as those introduced by <a href=\"/wiki/Synonym\" title=\"Synonym\">synonyms</a> and <a href=\"/wiki/Homonym\" title=\"Homonym\">homonyms</a>.\n",
            "</p>, <p>A further step in analysis is the distinction between dictionary-based (quantitative) approaches and qualitative approaches. Dictionary-based approaches set up a list of categories derived from the frequency list of words and control the distribution of words and their respective categories over the texts. While methods in quantitative content analysis in this way transform observations of found categories into quantitative statistical data, the qualitative content analysis focuses more on the intentionality and its implications. There are strong parallels between qualitative content analysis and <a href=\"/wiki/Thematic_analysis\" title=\"Thematic analysis\">thematic analysis</a>.<sup class=\"reference\" id=\"cite_ref-6\"><a href=\"#cite_note-6\">[6]</a></sup>\n",
            "</p>, <p>Quantitative content analysis highlights frequency counts and objective analysis of these coded frequencies.<sup class=\"reference\" id=\"cite_ref-:03_7-0\"><a href=\"#cite_note-:03-7\">[7]</a></sup> Additionally, quantitative content analysis begins with a framed hypothesis with coding decided on before the analysis begins. These coding categories are strictly relevant to the researcher's hypothesis. Quantitative analysis also takes a deductive approach.<sup class=\"reference\" id=\"cite_ref-:12_8-0\"><a href=\"#cite_note-:12-8\">[8]</a></sup>\n",
            "</p>, <p><a href=\"/wiki/Siegfried_Kracauer\" title=\"Siegfried Kracauer\">Siegfried Kracauer</a> provides a critique of quantitative analysis, asserting that it oversimplifies complex communications in order to be more reliable. On the other hand, qualitative analysis deals with the intricacies of latent interpretations, whereas quantitative has a focus on manifest meanings. He also acknowledges an \"overlap\" of qualitative and quantitative content analysis.<sup class=\"reference\" id=\"cite_ref-:03_7-1\"><a href=\"#cite_note-:03-7\">[7]</a></sup> Patterns are looked at more closely in qualitative analysis, and based on the latent meanings that the researcher may find, the course of the research could be changed. It is inductive and begins with open research questions, as opposed to a hypothesis.<sup class=\"reference\" id=\"cite_ref-:12_8-1\"><a href=\"#cite_note-:12-8\">[8]</a></sup>\n",
            "</p>, <p>More generally, content analysis is research using the categorization and classification of speech, written text, interviews, images, or other forms of communication.  In its beginnings, using the first newspapers at the end of the 19th century, analysis was done manually by measuring the number of columns given a subject. The approach can also be traced back to a university student studying patterns in Shakespeare's literature in 1893.<sup class=\"reference\" id=\"cite_ref-9\"><a href=\"#cite_note-9\">[9]</a></sup> With the rise of common computing facilities like PCs, computer-based methods of analysis are growing in popularity.<sup class=\"reference\" id=\"cite_ref-10\"><a href=\"#cite_note-10\">[10]</a></sup><sup class=\"reference\" id=\"cite_ref-11\"><a href=\"#cite_note-11\">[11]</a></sup><sup class=\"reference\" id=\"cite_ref-12\"><a href=\"#cite_note-12\">[12]</a></sup> Answers to open ended questions, newspaper articles, political party manifestos, medical records or systematic observations in experiments can all be subject to systematic analysis of textual data.\n",
            "</p>, <p>By having contents of communication available in form of machine readable texts, the input is analyzed for frequencies and coded into categories for building up inferences.\n",
            "</p>, <p>Computer-assisted analysis can help with large, electronic data sets by cutting out time and eliminating the need for multiple human coders to establish inter-coder reliability. However, human coders can still be employed for content analysis, as they are often more able to pick out nuanced and latent meanings in text. A study  found that human coders were able to evaluate a broader range and make inferences based on latent meanings.<sup class=\"reference\" id=\"cite_ref-13\"><a href=\"#cite_note-13\">[13]</a></sup>\n",
            "</p>, <p>Robert Weber notes: \"To make valid inferences from the text, it is important that the classification procedure be reliable in the sense of being consistent: Different people should code the same text in the same way\".<sup class=\"reference\" id=\"cite_ref-14\"><a href=\"#cite_note-14\">[14]</a></sup> The validity, inter-coder reliability and intra-coder reliability are subject to intense methodological research efforts over long years.<sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-1\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
            "Neuendorf suggests that when human coders are used in content analysis at least two independent coders should be used. Reliability of human coding is often measured using a statistical measure of <i>inter-coder reliability</i> or \"the amount of agreement or correspondence among two or more coders\".<sup class=\"reference\" id=\"cite_ref-Neuendorf2016_4-1\"><a href=\"#cite_note-Neuendorf2016-4\">[4]</a></sup> Lacy and Riffe identify the measurement of inter-coder reliability as a strength of quantitative content analysis, arguing that, if content analysts do not measure inter-coder reliability, their data are no more reliable than the subjective impressions of a single reader.<sup class=\"reference\" id=\"cite_ref-15\"><a href=\"#cite_note-15\">[15]</a></sup>\n",
            "</p>, <p>There are five types of texts in content analysis:\n",
            "</p>, <p>Over the years, content analysis has been applied to a variety of scopes. <a href=\"/wiki/Hermeneutics\" title=\"Hermeneutics\">Hermeneutics</a> and <a href=\"/wiki/Philology\" title=\"Philology\">philology</a> have long used content analysis to interpret sacred and profane texts and, in many cases, to attribute texts' <a class=\"mw-redirect\" href=\"/wiki/Authorship\" title=\"Authorship\">authorship</a> and <a href=\"/wiki/Authentication\" title=\"Authentication\">authenticity</a>.<sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-1\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup><sup class=\"reference\" id=\"cite_ref-Krippendorff2004_5-2\"><a href=\"#cite_note-Krippendorff2004-5\">[5]</a></sup>\n",
            "</p>, <p>In recent times, particularly with the advent of <a href=\"/wiki/Mass_communication\" title=\"Mass communication\">mass communication</a>, content analysis has known an increasing use to deeply analyze and understand media content and <a class=\"new\" href=\"/w/index.php?title=Media_logic&amp;action=edit&amp;redlink=1\" title=\"Media logic (page does not exist)\">media logic</a>. \n",
            "The political scientist <a href=\"/wiki/Harold_Lasswell\" title=\"Harold Lasswell\">Harold Lasswell</a> formulated the core questions of content analysis in its early-mid 20th-century mainstream version: \"Who says what, to whom, why, to what extent and with what effect?\".<sup class=\"reference\" id=\"cite_ref-16\"><a href=\"#cite_note-16\">[16]</a></sup> The strong emphasis for a quantitative approach started up by Lasswell was finally carried out by another \"father\" of content analysis, <a href=\"/wiki/Bernard_Berelson\" title=\"Bernard Berelson\">Bernard Berelson</a>, who proposed a definition of content analysis which, from this point of view, is emblematic: \"a research technique for the objective, systematic and quantitative description of the manifest content of communication\".<sup class=\"reference\" id=\"cite_ref-Berelson1952_17-0\"><a href=\"#cite_note-Berelson1952-17\">[17]</a></sup>\n",
            "</p>, <p>Quantitative content analysis has enjoyed a renewed popularity in recent years thanks to technological advances and fruitful application in of mass communication and personal communication research. Content analysis of textual <a href=\"/wiki/Big_data\" title=\"Big data\">big data</a> produced by <a href=\"/wiki/New_media\" title=\"New media\">new media</a>, particularly <a href=\"/wiki/Social_media\" title=\"Social media\">social media</a> and <a class=\"mw-redirect\" href=\"/wiki/Mobile_devices\" title=\"Mobile devices\">mobile devices</a> has become popular. These approaches take a simplified view of language that ignores the complexity of semiosis, the process by which meaning is formed out of language. Quantitative content analysts have been criticized for limiting the scope of content analysis to simple counting, and for applying the measurement methodologies of the natural sciences without reflecting critically on their appropriateness to social science.<sup class=\"reference\" id=\"cite_ref-:0_18-0\"><a href=\"#cite_note-:0-18\">[18]</a></sup> Conversely, qualitative content analysts have been criticized for being insufficiently systematic and too impressionistic.<sup class=\"reference\" id=\"cite_ref-:0_18-1\"><a href=\"#cite_note-:0-18\">[18]</a></sup> Krippendorff argues that quantitative and qualitative approaches to content analysis tend to overlap, and that there can be no generalisable conclusion as to which approach is superior.<sup class=\"reference\" id=\"cite_ref-:0_18-2\"><a href=\"#cite_note-:0-18\">[18]</a></sup>\n",
            "</p>, <p>Content analysis can also be described as studying <a href=\"/wiki/Trace_evidence\" title=\"Trace evidence\">traces</a>, which are documents from past times, and artifacts, which are non-linguistic documents. Texts are understood to be produced by communication processes in a broad sense of that phrase—often gaining mean through <a href=\"/wiki/Abductive_reasoning\" title=\"Abductive reasoning\">abduction</a>.<sup class=\"reference\" id=\"cite_ref-Tipaldo_2014_42_3-2\"><a href=\"#cite_note-Tipaldo_2014_42-3\">[3]</a></sup><sup class=\"reference\" id=\"cite_ref-19\"><a href=\"#cite_note-19\">[19]</a></sup>\n",
            "</p>, <p>Manifest content is readily understandable at its face value. Its meaning is direct. Latent content is not as overt, and requires interpretation to uncover the meaning or implication.<sup class=\"reference\" id=\"cite_ref-20\"><a href=\"#cite_note-20\">[20]</a></sup>\n",
            "</p>, <p>Holsti groups fifteen uses of content analysis into three basic <a class=\"mw-redirect\" href=\"/wiki/Categorisation\" title=\"Categorisation\">categories</a>:<sup class=\"reference\" id=\"cite_ref-Holsti1969_21-0\"><a href=\"#cite_note-Holsti1969-21\">[21]</a></sup>\n",
            "</p>, <p>He also places these uses into the context of the basic communication <a href=\"/wiki/Paradigm\" title=\"Paradigm\">paradigm</a>.\n",
            "</p>, <p>The following table shows fifteen uses of content analysis in terms of their general purpose, element of the communication paradigm to which they apply, and the general question they are intended to answer.\n",
            "</p>, <p>The process of the initial coding scheme or approach to coding is contingent on the particular content analysis approach selected. Through a directed content analysis, the scholars draft a preliminary coding scheme from pre-existing theory or assumptions. While with the conventional content analysis approach, the initial coding scheme developed from the data.\n",
            "</p>, <p>With either approach above, immersing oneself into the data to obtain an overall picture is recommendable for researchers to conduct. Furthermore, identifying a consistent and clear unit of coding is vital, and researchers' choices range from a single word to several paragraphs, from texts to iconic symbols. Last, constructing the relationships between codes by sorting out them within specific categories or themes.<sup class=\"reference\" id=\"cite_ref-23\"><a href=\"#cite_note-23\">[23]</a></sup>\n",
            "</p>, <p><br/>\n",
            "</p>]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IdZZq57AxuWC"
      },
      "source": [
        "Another excursion: Why do we use enumerate() here? enumerate() takes a collection, enumerates, and returns an enumate object with both the numbers and the collection. For example, contentPTags (the collection we used here) is comprised of paragraphs. We want the paragraph number of each paragraph. And this is what enumerate() does: it returns the paragraph number and the paragraph. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XeHf6GbSxuWC"
      },
      "source": [
        "We will be adding these new texts to our DataFrame `contentParagraphsDF` so we\n",
        "will need to add 2 more columns to keep track of paragraph numbers and sources."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 785
        },
        "id": "7ZxrUqwTxuWC",
        "outputId": "b7ab7f3f-41f1-4592-c46e-0fed457c2b07"
      },
      "source": [
        "contentParagraphsDF['source'] = [wikipedia_content_analysis] * len(contentParagraphsDF['paragraph-text'])\n",
        "contentParagraphsDF['paragraph-number'] = range(len(contentParagraphsDF['paragraph-text']))\n",
        "\n",
        "contentParagraphsDF"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paragraph-text</th>\n",
              "      <th>source</th>\n",
              "      <th>paragraph-number</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Content analysis is the study of documents and...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Practices and philosophies of content analysis...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Computers are increasingly used in content ana...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Content analysis is best understood as a broad...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The simplest and most objective form of conten...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>A further step in analysis is the distinction ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Quantitative content analysis highlights frequ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Siegfried Kracauer provides a critique of quan...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>More generally, content analysis is research u...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>By having contents of communication available ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Computer-assisted analysis can help with large...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Robert Weber notes: \"To make valid inferences ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>There are five types of texts in content analy...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Over the years, content analysis has been appl...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>In recent times, particularly with the advent ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Quantitative content analysis has enjoyed a re...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Content analysis can also be described as stud...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>Manifest content is readily understandable at ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>Holsti groups fifteen uses of content analysis...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>He also places these uses into the context of ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>The following table shows fifteen uses of cont...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>The process of the initial coding scheme or ap...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>With either approach above, immersing oneself ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>\\n</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                       paragraph-text  ... paragraph-number\n",
              "0   Content analysis is the study of documents and...  ...                0\n",
              "1   Practices and philosophies of content analysis...  ...                1\n",
              "2   Computers are increasingly used in content ana...  ...                2\n",
              "3   Content analysis is best understood as a broad...  ...                3\n",
              "4   The simplest and most objective form of conten...  ...                4\n",
              "5   A further step in analysis is the distinction ...  ...                5\n",
              "6   Quantitative content analysis highlights frequ...  ...                6\n",
              "7   Siegfried Kracauer provides a critique of quan...  ...                7\n",
              "8   More generally, content analysis is research u...  ...                8\n",
              "9   By having contents of communication available ...  ...                9\n",
              "10  Computer-assisted analysis can help with large...  ...               10\n",
              "11  Robert Weber notes: \"To make valid inferences ...  ...               11\n",
              "12  There are five types of texts in content analy...  ...               12\n",
              "13  Over the years, content analysis has been appl...  ...               13\n",
              "14  In recent times, particularly with the advent ...  ...               14\n",
              "15  Quantitative content analysis has enjoyed a re...  ...               15\n",
              "16  Content analysis can also be described as stud...  ...               16\n",
              "17  Manifest content is readily understandable at ...  ...               17\n",
              "18  Holsti groups fifteen uses of content analysis...  ...               18\n",
              "19  He also places these uses into the context of ...  ...               19\n",
              "20  The following table shows fifteen uses of cont...  ...               20\n",
              "21  The process of the initial coding scheme or ap...  ...               21\n",
              "22  With either approach above, immersing oneself ...  ...               22\n",
              "23                                                 \\n  ...               23\n",
              "\n",
              "[24 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIcbPVCWxuWD"
      },
      "source": [
        "Then we can add two more columns to our `Dataframe` and define a function to\n",
        "parse\n",
        "each linked page and add its text to our DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mHmR914vxuWD"
      },
      "source": [
        "contentParagraphsDF['source-paragraph-number'] = [None] * len(contentParagraphsDF['paragraph-text'])\n",
        "contentParagraphsDF['source-paragraph-text'] = [None] * len(contentParagraphsDF['paragraph-text'])\n",
        "\n",
        "def getTextFromWikiPage(targetURL, sourceParNum, sourceText):\n",
        "    #Make a dict to store data before adding it to the DataFrame\n",
        "    parsDict = {'source' : [], 'paragraph-number' : [], 'paragraph-text' : [], 'source-paragraph-number' : [],  'source-paragraph-text' : []}\n",
        "    #Now we get the page\n",
        "    r = requests.get(targetURL)\n",
        "    soup = bs4.BeautifulSoup(r.text, 'html.parser')\n",
        "    #enumerating gives use the paragraph number\n",
        "    for parNum, pTag in enumerate(soup.body.findAll('p')):\n",
        "        #same regex as before\n",
        "        parsDict['paragraph-text'].append(re.sub(r'\\[\\d+\\]', '', pTag.text))\n",
        "        parsDict['paragraph-number'].append(parNum)\n",
        "        parsDict['source'].append(targetURL)\n",
        "        parsDict['source-paragraph-number'].append(sourceParNum)\n",
        "        parsDict['source-paragraph-text'].append(sourceText)\n",
        "    return pandas.DataFrame(parsDict)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GsM_AuW2xuWD"
      },
      "source": [
        "And run it on our list of link tags"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3L3ZitbExuWD",
        "outputId": "b2d897f5-e710-4859-c1fc-444bbcdc6a40"
      },
      "source": [
        "for urlTuple in otherPAgeURLS[:3]:\n",
        "    #ignore_index means the indices will not be reset after each append\n",
        "    contentParagraphsDF = contentParagraphsDF.append(getTextFromWikiPage(*urlTuple),ignore_index=True)\n",
        "contentParagraphsDF"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paragraph-text</th>\n",
              "      <th>source</th>\n",
              "      <th>paragraph-number</th>\n",
              "      <th>source-paragraph-number</th>\n",
              "      <th>source-paragraph-text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Content analysis is the study of documents and...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Practices and philosophies of content analysis...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>1</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Computers are increasingly used in content ana...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>2</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Content analysis is best understood as a broad...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>3</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The simplest and most objective form of conten...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>4</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>A further step in analysis is the distinction ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>5</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Quantitative content analysis highlights frequ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>6</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Siegfried Kracauer provides a critique of quan...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>7</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>More generally, content analysis is research u...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>8</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>By having contents of communication available ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>9</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Computer-assisted analysis can help with large...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>10</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Robert Weber notes: \"To make valid inferences ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>11</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>There are five types of texts in content analy...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>12</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Over the years, content analysis has been appl...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>13</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>In recent times, particularly with the advent ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>14</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Quantitative content analysis has enjoyed a re...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>15</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Content analysis can also be described as stud...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>16</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>Manifest content is readily understandable at ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>17</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>Holsti groups fifteen uses of content analysis...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>18</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>He also places these uses into the context of ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>19</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>The following table shows fifteen uses of cont...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>20</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>The process of the initial coding scheme or ap...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>21</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>With either approach above, immersing oneself ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>22</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>\\n</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Content_analysis</td>\n",
              "      <td>23</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>A document is a written, drawn, presented, or ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>The concept of \"document\" has been defined by ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>An often-cited article concludes that \"the evo...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>\"Document\" is defined in library and informati...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>A document can be structured, like tabular doc...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Documents are used in numerous fields, e.g.:\\n</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>5</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Such standard documents can be drafted based o...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>6</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>The page layout of a document is how informati...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>7</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>Traditionally, the medium of a document was pa...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>8</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>Historically, documents were inscribed with in...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>9</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>Contemporary electronic means of memorializing...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>10</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>Digital documents usually require a specific f...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>11</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>Documents in all forms frequently serve as mat...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Document</td>\n",
              "      <td>12</td>\n",
              "      <td>0</td>\n",
              "      <td>documents</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>In literary theory, a text is any object that ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Within the field of literary criticism, \"text\"...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Since the history of writing predates the conc...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>The word text has its origins in Quintilian's ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>Relying on literary theory, the notion of text...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td></td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>This literature-related article is a stub. You...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Text_(literary_t...</td>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>texts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>In the social sciences, coding is an analytica...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>One purpose of coding is to transform the data...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>Some studies will employ multiple coders worki...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>One code should apply to only one category and...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>For quantitative analysis, data is coded usual...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>Questionnaire data can be pre-coded (process o...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>In social sciences, spreadsheets such as Excel...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>For disciplines in which a qualitative format ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Much of qualitative coding can be attributed t...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>8</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>Coding is considered a process of discovery an...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>9</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>The process can be done manually, which can be...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>10</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>After assembling codes it is time to organize ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>11</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Creating memos during the coding process is in...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Coding_(social_s...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>assigned labels (sometimes called codes)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                       paragraph-text  ...                     source-paragraph-text\n",
              "0   Content analysis is the study of documents and...  ...                                      None\n",
              "1   Practices and philosophies of content analysis...  ...                                      None\n",
              "2   Computers are increasingly used in content ana...  ...                                      None\n",
              "3   Content analysis is best understood as a broad...  ...                                      None\n",
              "4   The simplest and most objective form of conten...  ...                                      None\n",
              "5   A further step in analysis is the distinction ...  ...                                      None\n",
              "6   Quantitative content analysis highlights frequ...  ...                                      None\n",
              "7   Siegfried Kracauer provides a critique of quan...  ...                                      None\n",
              "8   More generally, content analysis is research u...  ...                                      None\n",
              "9   By having contents of communication available ...  ...                                      None\n",
              "10  Computer-assisted analysis can help with large...  ...                                      None\n",
              "11  Robert Weber notes: \"To make valid inferences ...  ...                                      None\n",
              "12  There are five types of texts in content analy...  ...                                      None\n",
              "13  Over the years, content analysis has been appl...  ...                                      None\n",
              "14  In recent times, particularly with the advent ...  ...                                      None\n",
              "15  Quantitative content analysis has enjoyed a re...  ...                                      None\n",
              "16  Content analysis can also be described as stud...  ...                                      None\n",
              "17  Manifest content is readily understandable at ...  ...                                      None\n",
              "18  Holsti groups fifteen uses of content analysis...  ...                                      None\n",
              "19  He also places these uses into the context of ...  ...                                      None\n",
              "20  The following table shows fifteen uses of cont...  ...                                      None\n",
              "21  The process of the initial coding scheme or ap...  ...                                      None\n",
              "22  With either approach above, immersing oneself ...  ...                                      None\n",
              "23                                                 \\n  ...                                      None\n",
              "24  A document is a written, drawn, presented, or ...  ...                                 documents\n",
              "25  The concept of \"document\" has been defined by ...  ...                                 documents\n",
              "26  An often-cited article concludes that \"the evo...  ...                                 documents\n",
              "27  \"Document\" is defined in library and informati...  ...                                 documents\n",
              "28  A document can be structured, like tabular doc...  ...                                 documents\n",
              "29     Documents are used in numerous fields, e.g.:\\n  ...                                 documents\n",
              "30  Such standard documents can be drafted based o...  ...                                 documents\n",
              "31  The page layout of a document is how informati...  ...                                 documents\n",
              "32  Traditionally, the medium of a document was pa...  ...                                 documents\n",
              "33  Historically, documents were inscribed with in...  ...                                 documents\n",
              "34  Contemporary electronic means of memorializing...  ...                                 documents\n",
              "35  Digital documents usually require a specific f...  ...                                 documents\n",
              "36  Documents in all forms frequently serve as mat...  ...                                 documents\n",
              "37  In literary theory, a text is any object that ...  ...                                     texts\n",
              "38  Within the field of literary criticism, \"text\"...  ...                                     texts\n",
              "39  Since the history of writing predates the conc...  ...                                     texts\n",
              "40  The word text has its origins in Quintilian's ...  ...                                     texts\n",
              "41  Relying on literary theory, the notion of text...  ...                                     texts\n",
              "42                                                     ...                                     texts\n",
              "43  This literature-related article is a stub. You...  ...                                     texts\n",
              "44  In the social sciences, coding is an analytica...  ...  assigned labels (sometimes called codes)\n",
              "45  One purpose of coding is to transform the data...  ...  assigned labels (sometimes called codes)\n",
              "46  Some studies will employ multiple coders worki...  ...  assigned labels (sometimes called codes)\n",
              "47  One code should apply to only one category and...  ...  assigned labels (sometimes called codes)\n",
              "48  For quantitative analysis, data is coded usual...  ...  assigned labels (sometimes called codes)\n",
              "49  Questionnaire data can be pre-coded (process o...  ...  assigned labels (sometimes called codes)\n",
              "50  In social sciences, spreadsheets such as Excel...  ...  assigned labels (sometimes called codes)\n",
              "51  For disciplines in which a qualitative format ...  ...  assigned labels (sometimes called codes)\n",
              "52  Much of qualitative coding can be attributed t...  ...  assigned labels (sometimes called codes)\n",
              "53  Coding is considered a process of discovery an...  ...  assigned labels (sometimes called codes)\n",
              "54  The process can be done manually, which can be...  ...  assigned labels (sometimes called codes)\n",
              "55  After assembling codes it is time to organize ...  ...  assigned labels (sometimes called codes)\n",
              "56  Creating memos during the coding process is in...  ...  assigned labels (sometimes called codes)\n",
              "\n",
              "[57 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4oIBfDjZxuWE"
      },
      "source": [
        "\n",
        "# <span style=\"color:red\">Section 2</span>\n",
        "<span style=\"color:red\">Construct cells immediately below this that spider webcontent from another site with content relating to your anticipated final project. Specifically, identify urls on a core page, then follow and extract content from them into a pandas `Dataframe`. In addition, demonstrate a *recursive* spider, which follows more than one level of links (i.e., follows links from a site, then follows links on followed sites to new sites, etc.), making sure to define a reasonable endpoint so that you do not wander the web forever :-).</span>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhtepSYOT1im"
      },
      "source": [
        "## Notes\n",
        "In the cells below, I crawled `history of China` Wikipedia page and its subpages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_KrR_dxJQIv8",
        "outputId": "994cf08a-de9d-420f-ec11-dccec4aaadaa"
      },
      "source": [
        "history = 'https://en.wikipedia.org/wiki/Population_history_of_China'\n",
        "requests.get(history)\n",
        "history_reguest = requests.get(history)\n",
        "print(history_reguest.text[:1000])\n",
        "soup = bs4.BeautifulSoup(history_reguest.text, 'html.parser')"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-03-21 05:32:55 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): en.wikipedia.org:443\n",
            "2021-03-21 05:32:55 [urllib3.connectionpool] DEBUG: https://en.wikipedia.org:443 \"GET /wiki/Population_history_of_China HTTP/1.1\" 200 43675\n",
            "2021-03-21 05:32:55 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): en.wikipedia.org:443\n",
            "2021-03-21 05:32:55 [urllib3.connectionpool] DEBUG: https://en.wikipedia.org:443 \"GET /wiki/Population_history_of_China HTTP/1.1\" 200 43675\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "<!DOCTYPE html>\n",
            "<html class=\"client-nojs\" lang=\"en\" dir=\"ltr\">\n",
            "<head>\n",
            "<meta charset=\"UTF-8\"/>\n",
            "<title>Population history of China - Wikipedia</title>\n",
            "<script>document.documentElement.className=\"client-js\";RLCONF={\"wgBreakFrames\":!1,\"wgSeparatorTransformTable\":[\"\",\"\"],\"wgDigitTransformTable\":[\"\",\"\"],\"wgDefaultDateFormat\":\"dmy\",\"wgMonthNames\":[\"\",\"January\",\"February\",\"March\",\"April\",\"May\",\"June\",\"July\",\"August\",\"September\",\"October\",\"November\",\"December\"],\"wgRequestId\":\"YFMNX1pheQHGxPiGcdwcCAAAAAE\",\"wgCSPNonce\":!1,\"wgCanonicalNamespace\":\"\",\"wgCanonicalSpecialPageName\":!1,\"wgNamespaceNumber\":0,\"wgPageName\":\"Population_history_of_China\",\"wgTitle\":\"Population history of China\",\"wgCurRevisionId\":1012644575,\"wgRevisionId\":1012644575,\"wgArticleId\":66143728,\"wgIsArticle\":!0,\"wgIsRedirect\":!1,\"wgAction\":\"view\",\"wgUserName\":null,\"wgUserGroups\":[\"*\"],\"wgCategories\":[\"Harv and Sfn no-target errors\",\"CS1 errors: missing periodical\",\"Webarchive template wayback links\",\"Demographics of China\",\"Demograp\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "96QgWkXfRQLM",
        "outputId": "95166019-0f66-4cfc-ac64-8b4923897122"
      },
      "source": [
        "save = 'wikipedia_gdp.html'\n",
        "with open(save, mode='w', encoding='utf-8') as f:\n",
        "    f.write(history_reguest.text)\n",
        "history_pTags = soup.body.findAll(r'p')\n",
        "\n",
        "otherpage_URLs = []\n",
        "\n",
        "wikipedia_base_url = history\n",
        "\n",
        "for i, pTag in enumerate(history_pTags):\n",
        "    tag_links = pTag.findAll('a', href=re.compile('/wiki/'), class_=False)\n",
        "    for aTag in tag_links:\n",
        "        # relative URL link\n",
        "        relURL = aTag.get('href')\n",
        "        link_text = aTag.text\n",
        "        otherpage_URLs.append((urllib.parse.urljoin(wikipedia_base_url,relURL),\n",
        "                               i, \n",
        "                               link_text,\n",
        "                              ))\n",
        "\n",
        "print(otherpage_URLs[:10])"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('https://en.wikipedia.org/wiki/China', 0, 'China'), ('https://en.wikipedia.org/wiki/History_of_China', 0, 'history of China'), ('https://en.wikipedia.org/wiki/Demographics_of_China', 0, 'demographics of China'), ('https://en.wikipedia.org/wiki/China#population', 0, 'China'), ('https://en.wikipedia.org/wiki/Warring_States_period', 2, 'Warring States period'), ('https://en.wikipedia.org/wiki/History_of_Chinese_currency', 2, 'a money economy'), ('https://en.wikipedia.org/wiki/Chinese_nobility', 2, 'nobility'), ('https://en.wikipedia.org/wiki/Qin_dynasty', 2, 'Qin dynasty'), ('https://en.wikipedia.org/wiki/Chang%27an', 2, \"Chang'an\"), ('https://en.wikipedia.org/wiki/Luoyang', 2, 'Luoyang')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ReP3Ei8fvj_o",
        "outputId": "cd36ba31-979d-418d-b7fe-5df36bac205e"
      },
      "source": [
        "URLs_df = pandas.DataFrame(otherpage_URLs, columns = ['URL','Paragraph_Number','Link Text'])\n",
        "URLs_df"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>URL</th>\n",
              "      <th>Paragraph_Number</th>\n",
              "      <th>Link Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://en.wikipedia.org/wiki/China</td>\n",
              "      <td>0</td>\n",
              "      <td>China</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://en.wikipedia.org/wiki/History_of_China</td>\n",
              "      <td>0</td>\n",
              "      <td>history of China</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Demographics_of_...</td>\n",
              "      <td>0</td>\n",
              "      <td>demographics of China</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://en.wikipedia.org/wiki/China#population</td>\n",
              "      <td>0</td>\n",
              "      <td>China</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Warring_States_p...</td>\n",
              "      <td>2</td>\n",
              "      <td>Warring States period</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>https://en.wikipedia.org/wiki/History_of_Chine...</td>\n",
              "      <td>2</td>\n",
              "      <td>a money economy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Chinese_nobility</td>\n",
              "      <td>2</td>\n",
              "      <td>nobility</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Qin_dynasty</td>\n",
              "      <td>2</td>\n",
              "      <td>Qin dynasty</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Chang%27an</td>\n",
              "      <td>2</td>\n",
              "      <td>Chang'an</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Chinese_city_wall</td>\n",
              "      <td>2</td>\n",
              "      <td>city walls</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Gatehouse</td>\n",
              "      <td>2</td>\n",
              "      <td>gatehouses</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Angus_Maddison</td>\n",
              "      <td>3</td>\n",
              "      <td>Angus Maddison</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Sinology</td>\n",
              "      <td>4</td>\n",
              "      <td>Sinologist historians</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Timothy_Brook</td>\n",
              "      <td>4</td>\n",
              "      <td>Timothy Brook</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>https://en.wikipedia.org/wiki/North_Zhili</td>\n",
              "      <td>4</td>\n",
              "      <td>North Zhili</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Infanticide</td>\n",
              "      <td>4</td>\n",
              "      <td>infanticide</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Hongzhi_Emperor</td>\n",
              "      <td>5</td>\n",
              "      <td>Hongzhi Emperor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Gazetteer</td>\n",
              "      <td>6</td>\n",
              "      <td>gazetteers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Chenghua_Emperor</td>\n",
              "      <td>6</td>\n",
              "      <td>Chenghua Emperor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Hongzhi_Emperor</td>\n",
              "      <td>6</td>\n",
              "      <td>Hongzhi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Zhengde_Emperor</td>\n",
              "      <td>6</td>\n",
              "      <td>Zhengde</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Jiajing_Emperor</td>\n",
              "      <td>6</td>\n",
              "      <td>Jiajing Emperor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Zhejiang</td>\n",
              "      <td>7</td>\n",
              "      <td>Zhejiang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Taiping_Rebellion</td>\n",
              "      <td>8</td>\n",
              "      <td>Taiping Rebellion</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Yunnan</td>\n",
              "      <td>9</td>\n",
              "      <td>Yunnan</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Guizhou</td>\n",
              "      <td>9</td>\n",
              "      <td>Guizhou</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Taiping_Rebellion</td>\n",
              "      <td>11</td>\n",
              "      <td>Taiping Rebellion</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Tongzhi_Restoration</td>\n",
              "      <td>12</td>\n",
              "      <td>Tongzhi Restoration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Northern_Chinese...</td>\n",
              "      <td>12</td>\n",
              "      <td>Great North China Famine</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Shanxi</td>\n",
              "      <td>12</td>\n",
              "      <td>Shanxi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Great_Chinese_Fa...</td>\n",
              "      <td>13</td>\n",
              "      <td>the 1958–1961 famine</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Great_Leap_Forward</td>\n",
              "      <td>13</td>\n",
              "      <td>Great Leap Forward</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Production_quota</td>\n",
              "      <td>13</td>\n",
              "      <td>production quotas</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Malay_Peninsula</td>\n",
              "      <td>15</td>\n",
              "      <td>Malaya</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Cultural_Revolution</td>\n",
              "      <td>17</td>\n",
              "      <td>Cultural Revolution</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>https://en.wikipedia.org/wiki/One-child_policy</td>\n",
              "      <td>18</td>\n",
              "      <td>one-child policy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Xinjiang</td>\n",
              "      <td>20</td>\n",
              "      <td>Xinjiang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Uyghurs</td>\n",
              "      <td>20</td>\n",
              "      <td>Uyghur</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>https://en.wikipedia.org/wiki/Xinjiang_conflict</td>\n",
              "      <td>20</td>\n",
              "      <td>Uyghur separatism</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>https://en.wikipedia.org/wiki/The_Economist</td>\n",
              "      <td>22</td>\n",
              "      <td>The Economist</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  URL  ...                 Link Text\n",
              "0                 https://en.wikipedia.org/wiki/China  ...                     China\n",
              "1      https://en.wikipedia.org/wiki/History_of_China  ...          history of China\n",
              "2   https://en.wikipedia.org/wiki/Demographics_of_...  ...     demographics of China\n",
              "3      https://en.wikipedia.org/wiki/China#population  ...                     China\n",
              "4   https://en.wikipedia.org/wiki/Warring_States_p...  ...     Warring States period\n",
              "5   https://en.wikipedia.org/wiki/History_of_Chine...  ...           a money economy\n",
              "6      https://en.wikipedia.org/wiki/Chinese_nobility  ...                  nobility\n",
              "7           https://en.wikipedia.org/wiki/Qin_dynasty  ...               Qin dynasty\n",
              "8            https://en.wikipedia.org/wiki/Chang%27an  ...                  Chang'an\n",
              "9               https://en.wikipedia.org/wiki/Luoyang  ...                   Luoyang\n",
              "10    https://en.wikipedia.org/wiki/Chinese_city_wall  ...                city walls\n",
              "11            https://en.wikipedia.org/wiki/Gatehouse  ...                gatehouses\n",
              "12       https://en.wikipedia.org/wiki/Angus_Maddison  ...            Angus Maddison\n",
              "13             https://en.wikipedia.org/wiki/Sinology  ...     Sinologist historians\n",
              "14        https://en.wikipedia.org/wiki/Timothy_Brook  ...             Timothy Brook\n",
              "15          https://en.wikipedia.org/wiki/North_Zhili  ...               North Zhili\n",
              "16          https://en.wikipedia.org/wiki/Infanticide  ...               infanticide\n",
              "17      https://en.wikipedia.org/wiki/Hongzhi_Emperor  ...           Hongzhi Emperor\n",
              "18            https://en.wikipedia.org/wiki/Gazetteer  ...                gazetteers\n",
              "19     https://en.wikipedia.org/wiki/Chenghua_Emperor  ...          Chenghua Emperor\n",
              "20      https://en.wikipedia.org/wiki/Hongzhi_Emperor  ...                   Hongzhi\n",
              "21      https://en.wikipedia.org/wiki/Zhengde_Emperor  ...                   Zhengde\n",
              "22      https://en.wikipedia.org/wiki/Jiajing_Emperor  ...           Jiajing Emperor\n",
              "23             https://en.wikipedia.org/wiki/Zhejiang  ...                  Zhejiang\n",
              "24    https://en.wikipedia.org/wiki/Taiping_Rebellion  ...         Taiping Rebellion\n",
              "25               https://en.wikipedia.org/wiki/Yunnan  ...                    Yunnan\n",
              "26              https://en.wikipedia.org/wiki/Guizhou  ...                   Guizhou\n",
              "27    https://en.wikipedia.org/wiki/Taiping_Rebellion  ...         Taiping Rebellion\n",
              "28  https://en.wikipedia.org/wiki/Tongzhi_Restoration  ...       Tongzhi Restoration\n",
              "29  https://en.wikipedia.org/wiki/Northern_Chinese...  ...  Great North China Famine\n",
              "30               https://en.wikipedia.org/wiki/Shanxi  ...                    Shanxi\n",
              "31  https://en.wikipedia.org/wiki/Great_Chinese_Fa...  ...      the 1958–1961 famine\n",
              "32   https://en.wikipedia.org/wiki/Great_Leap_Forward  ...        Great Leap Forward\n",
              "33     https://en.wikipedia.org/wiki/Production_quota  ...         production quotas\n",
              "34      https://en.wikipedia.org/wiki/Malay_Peninsula  ...                    Malaya\n",
              "35  https://en.wikipedia.org/wiki/Cultural_Revolution  ...       Cultural Revolution\n",
              "36     https://en.wikipedia.org/wiki/One-child_policy  ...          one-child policy\n",
              "37             https://en.wikipedia.org/wiki/Xinjiang  ...                  Xinjiang\n",
              "38              https://en.wikipedia.org/wiki/Uyghurs  ...                    Uyghur\n",
              "39    https://en.wikipedia.org/wiki/Xinjiang_conflict  ...         Uyghur separatism\n",
              "40        https://en.wikipedia.org/wiki/The_Economist  ...             The Economist\n",
              "\n",
              "[41 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "6qbv_gzfR9oJ",
        "outputId": "f554b626-caf4-41d2-b5ac-b6631e27959c"
      },
      "source": [
        "ref_regex = r'\\[\\d+\\]'\n",
        "para_regex = r'\\n'\n",
        "\n",
        "def spidering(url):\n",
        "    '''\n",
        "    Spider in wikipedia page \n",
        "\n",
        "    '''\n",
        "    request = requests.get(url)\n",
        "    soup = bs4.BeautifulSoup(request.text,'html.parser')\n",
        "    pTags = soup.body.findAll(r'p')\n",
        "\n",
        "    otherpage_URLs = []\n",
        "    base_url = history\n",
        "    \n",
        "    for i, pTag in enumerate(pTags):\n",
        "        tag_links = pTag.findAll('a',href=re.compile('/wiki/'),class_=False)\n",
        "        for aTag in tag_links:\n",
        "            relURL = aTag.get('href')\n",
        "            link_text = aTag.text\n",
        "            otherpage_URLs.append((urllib.parse.urljoin(wikipedia_base_url,relURL),\n",
        "                                   i, \n",
        "                                   link_text,\n",
        "                                  )) \n",
        "\n",
        "    paragraphs = []\n",
        "    for pTag in pTags:\n",
        "        str1 = re.sub(ref_regex,'',pTag.text)\n",
        "        str2 = re.sub(para_regex,'',str1)\n",
        "        if str2:\n",
        "            paragraphs.append(str2)\n",
        "\n",
        "    # conver to a DataFrame\n",
        "    paragraphs_df = pandas.DataFrame({'paragraph-text':paragraphs})\n",
        "    paragraphs_df['source'] = [url]*len(paragraphs_df['paragraph-text'])\n",
        "    paragraphs_df['paragraph-number'] = range(len(paragraphs_df['paragraph-text']))\n",
        "    paragraphs_df['source-paragraph-number'] = [None] * len(paragraphs_df['paragraph-text'])\n",
        "    paragraphs_df['source-paragraph-text'] = [None] * len(paragraphs_df['paragraph-text'])\n",
        "      \n",
        "\n",
        "    return paragraphs_df\n",
        "\n",
        "history_df = spidering(history)\n",
        "history_df"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-03-21 05:37:01 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): en.wikipedia.org:443\n",
            "2021-03-21 05:37:01 [urllib3.connectionpool] DEBUG: https://en.wikipedia.org:443 \"GET /wiki/Population_history_of_China HTTP/1.1\" 200 43675\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paragraph-text</th>\n",
              "      <th>source</th>\n",
              "      <th>paragraph-number</th>\n",
              "      <th>source-paragraph-number</th>\n",
              "      <th>source-paragraph-text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>The population history of China covers the lon...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>34567 b.c.</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>1</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>During the Warring States period (403–221 BC),...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>2</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Demographic historian Angus Maddison uses exte...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>3</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Sinologist historians debate the population fi...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>4</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>The number of people counted in the census of ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>5</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Historians are now turning to local gazetteers...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>6</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Even with the Jiajing reforms to document migr...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>7</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>The most significant facts of early and mid-Qi...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>8</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>The population was also remarkably mobile, per...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>9</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Chinese scholars had kept count of 1,828 insta...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>10</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>The period from 1850 to 1873 saw, as a result ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>11</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>When a stressed monarchy shifted from state ma...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>12</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>The largest famine of the 20th century, and al...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>13</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>The number of famine deaths during 1958–1961 r...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>14</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Chinese emigration first occurred thousands of...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>15</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>In 2009 there were 40-45 million overseas Chin...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>16</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>From 1980 to 2015, the government of China per...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>17</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>As China's youngest generation (born under the...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>18</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>The policy was supposedly voluntary. It was mo...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>19</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>After 2000 the policy was steadily relaxed. Ha...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>20</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>In 2016 the national policy changed to a two-c...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>21</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>In 2018, about two years after the new policy ...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>22</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>After the reform, China saw a short-lived boos...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>23</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>In China, men still have greater marital power...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>24</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Chinese couples were also polled and stated th...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>25</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>In May 2018, it was reported that Chinese auth...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>26</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>All of this is wrong</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>27</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                       paragraph-text  ... source-paragraph-text\n",
              "0   The population history of China covers the lon...  ...                  None\n",
              "1                                          34567 b.c.  ...                  None\n",
              "2   During the Warring States period (403–221 BC),...  ...                  None\n",
              "3   Demographic historian Angus Maddison uses exte...  ...                  None\n",
              "4   Sinologist historians debate the population fi...  ...                  None\n",
              "5   The number of people counted in the census of ...  ...                  None\n",
              "6   Historians are now turning to local gazetteers...  ...                  None\n",
              "7   Even with the Jiajing reforms to document migr...  ...                  None\n",
              "8   The most significant facts of early and mid-Qi...  ...                  None\n",
              "9   The population was also remarkably mobile, per...  ...                  None\n",
              "10  Chinese scholars had kept count of 1,828 insta...  ...                  None\n",
              "11  The period from 1850 to 1873 saw, as a result ...  ...                  None\n",
              "12  When a stressed monarchy shifted from state ma...  ...                  None\n",
              "13  The largest famine of the 20th century, and al...  ...                  None\n",
              "14  The number of famine deaths during 1958–1961 r...  ...                  None\n",
              "15  Chinese emigration first occurred thousands of...  ...                  None\n",
              "16  In 2009 there were 40-45 million overseas Chin...  ...                  None\n",
              "17  From 1980 to 2015, the government of China per...  ...                  None\n",
              "18  As China's youngest generation (born under the...  ...                  None\n",
              "19  The policy was supposedly voluntary. It was mo...  ...                  None\n",
              "20  After 2000 the policy was steadily relaxed. Ha...  ...                  None\n",
              "21  In 2016 the national policy changed to a two-c...  ...                  None\n",
              "22  In 2018, about two years after the new policy ...  ...                  None\n",
              "23  After the reform, China saw a short-lived boos...  ...                  None\n",
              "24  In China, men still have greater marital power...  ...                  None\n",
              "25  Chinese couples were also polled and stated th...  ...                  None\n",
              "26  In May 2018, it was reported that Chinese auth...  ...                  None\n",
              "27                               All of this is wrong  ...                  None\n",
              "\n",
              "[28 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5KD7R-YCSnXx"
      },
      "source": [
        "# recursive spidering\n",
        "def getTextFromWikiPage(targetURL, sourceParNum, sourceText):\n",
        "    #Make a dict to store data before adding it to the DataFrame\n",
        "    parsDict = {'source' : [], 'paragraph-number' : [], 'paragraph-text' : [], 'source-paragraph-number' : [],  'source-paragraph-text' : []}\n",
        "    #Now we get the page\n",
        "    r = requests.get(targetURL)\n",
        "    soup = bs4.BeautifulSoup(r.text, 'html.parser')\n",
        "    #enumerating gives use the paragraph number\n",
        "    for parNum, pTag in enumerate(soup.body.findAll('p')):\n",
        "        #same regex as before\n",
        "        parsDict['paragraph-text'].append(re.sub(r'\\[\\d+\\]', '', pTag.text))\n",
        "        parsDict['paragraph-number'].append(parNum)\n",
        "        parsDict['source'].append(targetURL)\n",
        "        parsDict['source-paragraph-number'].append(sourceParNum)\n",
        "        parsDict['source-paragraph-text'].append(sourceText)    \n",
        "    # Recursion\n",
        "    tag_links = pTag.findAll('a',href=re.compile('/wiki/'),class_=False)\n",
        "    for aTag in tag_links:\n",
        "        relURL = aTag.get('href')\n",
        "        link_text = aTag.text\n",
        "        otherpage_URLs.append((urllib.parse.urljoin(wikipedia_base_url,relURL),\n",
        "                               i, \n",
        "                               link_text,\n",
        "                              )) \n",
        "    return pd.DataFrame(parsDict)"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UI4Z48h_SuKh"
      },
      "source": [
        "level_of_recursion = 3\n",
        "urls_per_level = 10\n",
        "\n",
        "for i in range(level_of_recursion):\n",
        "    newURLs = []\n",
        "    for url_tuple in otherpage_URLs[:urls_per_level]:\n",
        "        history_df = history_df.append(getTextFromWikiPage(*url_tuple),ignore_index=True)\n",
        "    otherpage_URLs = newURLs"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 589
        },
        "id": "mM6JH9AITsOb",
        "outputId": "5def2681-fb29-4d3c-b3d9-fc0e46fccdb3"
      },
      "source": [
        "history_df"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paragraph-text</th>\n",
              "      <th>source</th>\n",
              "      <th>paragraph-number</th>\n",
              "      <th>source-paragraph-number</th>\n",
              "      <th>source-paragraph-text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>The population history of China covers the lon...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>34567 b.c.</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>1</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>During the Warring States period (403–221 BC),...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>2</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Demographic historian Angus Maddison uses exte...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>3</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Sinologist historians debate the population fi...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Population_histo...</td>\n",
              "      <td>4</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>855</th>\n",
              "      <td>The city can be reached by highways, trains or...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>36</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>856</th>\n",
              "      <td>The main station for conventional rail service...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>37</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>857</th>\n",
              "      <td>Luoyang is served by Luoyang Beijiao Airport.\\n</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>38</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>858</th>\n",
              "      <td>Luoyang is twinned with:\\n</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>39</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>859</th>\n",
              "      <td>aDirect-administered Municipalities.\\nbSub-pro...</td>\n",
              "      <td>https://en.wikipedia.org/wiki/Luoyang</td>\n",
              "      <td>40</td>\n",
              "      <td>2</td>\n",
              "      <td>Luoyang</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>860 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        paragraph-text  ... source-paragraph-text\n",
              "0    The population history of China covers the lon...  ...                  None\n",
              "1                                           34567 b.c.  ...                  None\n",
              "2    During the Warring States period (403–221 BC),...  ...                  None\n",
              "3    Demographic historian Angus Maddison uses exte...  ...                  None\n",
              "4    Sinologist historians debate the population fi...  ...                  None\n",
              "..                                                 ...  ...                   ...\n",
              "855  The city can be reached by highways, trains or...  ...               Luoyang\n",
              "856  The main station for conventional rail service...  ...               Luoyang\n",
              "857    Luoyang is served by Luoyang Beijiao Airport.\\n  ...               Luoyang\n",
              "858                         Luoyang is twinned with:\\n  ...               Luoyang\n",
              "859  aDirect-administered Municipalities.\\nbSub-pro...  ...               Luoyang\n",
              "\n",
              "[860 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YohImMuxuWE"
      },
      "source": [
        "## API (Tumblr)\n",
        "\n",
        "Generally website owners do not like you scraping their sites. If done badly,\n",
        "scarping can act like a DOS attack so you should be careful how often you make\n",
        "calls to a site. Some sites want automated tools to access their data, so they\n",
        "create [application programming interface\n",
        "(APIs)](https://en.wikipedia.org/wiki/Application_programming_interface). An API\n",
        "specifies a procedure for an application (or script) to access their data. Often\n",
        "this is though a [representational state transfer\n",
        "(REST)](https://en.wikipedia.org/wiki/Representational_state_transfer) web\n",
        "service, which just means if you make correctly formatted HTTP requests they\n",
        "will return nicely formatted data.\n",
        "\n",
        "A nice example for us to study is [Tumblr](https://www.tumblr.com), they have a\n",
        "[simple RESTful API](https://www.tumblr.com/docs/en/api/v1) that allows you to\n",
        "read posts without any complicated html parsing.\n",
        "\n",
        "We can get the first 20 posts from a blog by making an http GET request to\n",
        "`'http://{blog}.tumblr.com/api/read/json'`, were `{blog}` is the name of the\n",
        "target blog. Lets try and get the posts from [http://lolcats-lol-\n",
        "cat.tumblr.com/](http://lolcats-lol-cat.tumblr.com/) (Note the blog says at the\n",
        "top 'One hour one pic lolcats', but the canonical name that Tumblr uses is in\n",
        "the URL 'lolcats-lol-cat')."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4DVA6F8UxuWE",
        "outputId": "dd1bf926-ad3a-4bb2-e311-422d719cf553"
      },
      "source": [
        "tumblrAPItarget = 'http://{}.tumblr.com/api/read/json'\n",
        "\n",
        "r = requests.get(tumblrAPItarget.format('lolcats-lol-cat'))\n",
        "\n",
        "print(r.text[:1000])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "var tumblr_api_read = {\"tumblelog\":{\"title\":\"One hour one pic lolcats\",\"description\":\"\",\"name\":\"lolcats-lol-cat\",\"timezone\":\"Europe\\/Paris\",\"cname\":false,\"feeds\":[]},\"posts-start\":0,\"posts-total\":3751,\"posts-type\":false,\"posts\":[{\"id\":\"645711888306585600\",\"url\":\"https:\\/\\/lolcats-lol-cat.tumblr.com\\/post\\/645711888306585600\",\"url-with-slug\":\"https:\\/\\/lolcats-lol-cat.tumblr.com\\/post\\/645711888306585600\\/dreaming-of-playing-her-favorite-game\",\"type\":\"photo\",\"date-gmt\":\"2021-03-15 09:00:43 GMT\",\"date\":\"Mon, 15 Mar 2021 10:00:43\",\"bookmarklet\":0,\"mobile\":0,\"feed-item\":\"\",\"from-feed-id\":0,\"unix-timestamp\":1615798843,\"format\":\"html\",\"reblog-key\":\"pPfv34HT\",\"slug\":\"dreaming-of-playing-her-favorite-game\",\"is-submission\":false,\"like-button\":\"<div class=\\\"like_button\\\" data-post-id=\\\"645711888306585600\\\" data-blog-name=\\\"lolcats-lol-cat\\\" id=\\\"like_button_645711888306585600\\\"><iframe id=\\\"like_iframe_645711888306585600\\\" src=\\\"https:\\/\\/assets.tumblr.com\\/assets\\/html\\/like_iframe.html?_v=66c2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CxW26dGExuWE"
      },
      "source": [
        "This might not look very good on first inspection, but it has far fewer angle\n",
        "braces than html, which makes it easier to parse. What we have is\n",
        "[JSON](https://en.wikipedia.org/wiki/JSON) a 'human readable' text based data\n",
        "transmission format based on javascript. Luckily, we can readily convert it to a\n",
        "python `dict`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g21QQb8XxuWE",
        "outputId": "81b41cc9-211f-49ef-f251-40b550c43503"
      },
      "source": [
        "#We need to load only the stuff between the curly braces\n",
        "d = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
        "print(d.keys())\n",
        "print(len(d['posts']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dict_keys(['tumblelog', 'posts-start', 'posts-total', 'posts-type', 'posts'])\n",
            "20\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2RZwc4FxuWE"
      },
      "source": [
        "If we read the [API specification](https://www.tumblr.com/docs/en/api/v1), we\n",
        "will see there are a lot of things we can get if we add things to our GET\n",
        "request. First we can retrieve posts by their id number. Let's first get post\n",
        "`146020177084`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MMsYp_20xuWE"
      },
      "source": [
        "r = requests.get(tumblrAPItarget.format('lolcats-lol-cat'), params = {'id' : 146020177084})\n",
        "d = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
        "d['posts'][0].keys()\n",
        "d['posts'][0]['photo-url-1280']\n",
        "\n",
        "with open('lolcat.gif', 'wb') as f:\n",
        "    gifRequest = requests.get(d['posts'][0]['photo-url-1280'], stream = True)\n",
        "    f.write(gifRequest.content)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXOh22bmxuWF"
      },
      "source": [
        "<img src='lolcat.gif'>\n",
        "\n",
        "Such beauty; such vigor (If you can't see it you have to refresh the page). Now\n",
        "we could retrieve the text from all posts as well\n",
        "as related metadata, like the post date, caption or tags. We could also get\n",
        "links to all the images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "rBIYSc7lxuWF",
        "outputId": "61877e31-f001-4e54-e660-724fedb50cb6"
      },
      "source": [
        "#Putting a max in case the blog has millions of images\n",
        "#The given max will be rounded up to the nearest multiple of 50\n",
        "def tumblrImageScrape(blogName, maxImages = 200):\n",
        "    #Restating this here so the function isn't dependent on any external variables\n",
        "    tumblrAPItarget = 'http://{}.tumblr.com/api/read/json'\n",
        "\n",
        "    #There are a bunch of possible locations for the photo url\n",
        "    possiblePhotoSuffixes = [1280, 500, 400, 250, 100]\n",
        "\n",
        "    #These are the pieces of information we will be gathering,\n",
        "    #at the end we will convert this to a DataFrame.\n",
        "    #There are a few other datums we could gather like the captions\n",
        "    #you can read the Tumblr documentation to learn how to get them\n",
        "    #https://www.tumblr.com/docs/en/api/v1\n",
        "    postsData = {\n",
        "        'id' : [],\n",
        "        'photo-url' : [],\n",
        "        'date' : [],\n",
        "        'tags' : [],\n",
        "        'photo-type' : []\n",
        "    }\n",
        "\n",
        "    #Tumblr limits us to a max of 50 posts per request\n",
        "    for requestNum in range(maxImages // 50):\n",
        "        requestParams = {\n",
        "            'start' : requestNum * 50,\n",
        "            'num' : 50,\n",
        "            'type' : 'photo'\n",
        "        }\n",
        "        r = requests.get(tumblrAPItarget.format(blogName), params = requestParams)\n",
        "        requestDict = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
        "        for postDict in requestDict['posts']:\n",
        "            #We are dealing with uncleaned data, we can't trust it.\n",
        "            #Specifically, not all posts are guaranteed to have the fields we want\n",
        "            try:\n",
        "                postsData['id'].append(postDict['id'])\n",
        "                postsData['date'].append(postDict['date'])\n",
        "                postsData['tags'].append(postDict['tags'])\n",
        "            except KeyError as e:\n",
        "                raise KeyError(\"Post {} from {} is missing: {}\".format(postDict['id'], blogName, e))\n",
        "\n",
        "            foundSuffix = False\n",
        "            for suffix in possiblePhotoSuffixes:\n",
        "                try:\n",
        "                    photoURL = postDict['photo-url-{}'.format(suffix)]\n",
        "                    postsData['photo-url'].append(photoURL)\n",
        "                    postsData['photo-type'].append(photoURL.split('.')[-1])\n",
        "                    foundSuffix = True\n",
        "                    break\n",
        "                except KeyError:\n",
        "                    pass\n",
        "            if not foundSuffix:\n",
        "                #Make sure your error messages are useful\n",
        "                #You will be one of the users\n",
        "                raise KeyError(\"Post {} from {} is missing a photo url\".format(postDict['id'], blogName))\n",
        "\n",
        "    return pandas.DataFrame(postsData)\n",
        "tumblrImageScrape('lolcats-lol-cat', 50)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>photo-url</th>\n",
              "      <th>date</th>\n",
              "      <th>tags</th>\n",
              "      <th>photo-type</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>645711888306585600</td>\n",
              "      <td>https://64.media.tumblr.com/2aa57c4414f11ad8eb...</td>\n",
              "      <td>Mon, 15 Mar 2021 10:00:43</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>645704327082704896</td>\n",
              "      <td>https://64.media.tumblr.com/b7b271a894a0f835d8...</td>\n",
              "      <td>Mon, 15 Mar 2021 08:00:32</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>645681731762077696</td>\n",
              "      <td>https://64.media.tumblr.com/ddc0d52827a01ce892...</td>\n",
              "      <td>Mon, 15 Mar 2021 02:01:24</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>645500515139174400</td>\n",
              "      <td>https://64.media.tumblr.com/23bfe328dd71cc7a29...</td>\n",
              "      <td>Sat, 13 Mar 2021 02:01:02</td>\n",
              "      <td>[gif, lolcat, lolcats, cat, funny]</td>\n",
              "      <td>gif</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>644896541426008064</td>\n",
              "      <td>https://64.media.tumblr.com/a3f93012b1e3ab6110...</td>\n",
              "      <td>Sat, 06 Mar 2021 10:01:08</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>644737970778374145</td>\n",
              "      <td>https://64.media.tumblr.com/8d2a4976aa44077db2...</td>\n",
              "      <td>Thu, 04 Mar 2021 16:00:43</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>png</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>644730428596584448</td>\n",
              "      <td>https://64.media.tumblr.com/ecab6a19f2cf1679ad...</td>\n",
              "      <td>Thu, 04 Mar 2021 14:00:50</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>644662504074723328</td>\n",
              "      <td>https://64.media.tumblr.com/3699ee4bb8dffc707a...</td>\n",
              "      <td>Wed, 03 Mar 2021 20:01:12</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>644428418420719616</td>\n",
              "      <td>https://64.media.tumblr.com/b221ae8e39af4d076a...</td>\n",
              "      <td>Mon, 01 Mar 2021 06:00:31</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>644420865050673153</td>\n",
              "      <td>https://64.media.tumblr.com/d7d4fb891e023fdba8...</td>\n",
              "      <td>Mon, 01 Mar 2021 04:00:27</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>644390682111475712</td>\n",
              "      <td>https://64.media.tumblr.com/9e3f0fdb4e4c8cd0cc...</td>\n",
              "      <td>Sun, 28 Feb 2021 20:00:43</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>png</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>644322771711754240</td>\n",
              "      <td>https://64.media.tumblr.com/4c83c0d20ae46771c2...</td>\n",
              "      <td>Sun, 28 Feb 2021 02:01:18</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>644201974528704512</td>\n",
              "      <td>https://64.media.tumblr.com/6cd0283b0fe794a4e1...</td>\n",
              "      <td>Fri, 26 Feb 2021 18:01:17</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>644194375680835585</td>\n",
              "      <td>https://64.media.tumblr.com/ea77ec683571b6c6f7...</td>\n",
              "      <td>Fri, 26 Feb 2021 16:00:30</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>644179310864957440</td>\n",
              "      <td>https://64.media.tumblr.com/68657b5d3c111d3ec3...</td>\n",
              "      <td>Fri, 26 Feb 2021 12:01:03</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>644149104115679232</td>\n",
              "      <td>https://64.media.tumblr.com/b79d7e30b855205ea1...</td>\n",
              "      <td>Fri, 26 Feb 2021 04:00:56</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>644141551244771328</td>\n",
              "      <td>https://64.media.tumblr.com/d1700fa7a7fb4b2539...</td>\n",
              "      <td>Fri, 26 Feb 2021 02:00:53</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>644126436384505856</td>\n",
              "      <td>https://64.media.tumblr.com/bb918150b7be051e6a...</td>\n",
              "      <td>Thu, 25 Feb 2021 22:00:38</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>644058500866228224</td>\n",
              "      <td>https://64.media.tumblr.com/750face916a25914ec...</td>\n",
              "      <td>Thu, 25 Feb 2021 04:00:50</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>644050965032517632</td>\n",
              "      <td>https://64.media.tumblr.com/368996d2dbda5ae973...</td>\n",
              "      <td>Thu, 25 Feb 2021 02:01:03</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>644035836859547648</td>\n",
              "      <td>https://64.media.tumblr.com/7a3fcc870c5ba74673...</td>\n",
              "      <td>Wed, 24 Feb 2021 22:00:36</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>644020737916715009</td>\n",
              "      <td>https://64.media.tumblr.com/d04b91a493c237a98a...</td>\n",
              "      <td>Wed, 24 Feb 2021 18:00:36</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>643597946497138688</td>\n",
              "      <td>https://64.media.tumblr.com/332d01eefd3b9dd333...</td>\n",
              "      <td>Sat, 20 Feb 2021 02:00:31</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>643582865514397696</td>\n",
              "      <td>https://64.media.tumblr.com/81866535ed006a7849...</td>\n",
              "      <td>Fri, 19 Feb 2021 22:00:49</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>643575374076428288</td>\n",
              "      <td>https://64.media.tumblr.com/e7237034ecb90cae33...</td>\n",
              "      <td>Fri, 19 Feb 2021 20:01:44</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>643567764652638209</td>\n",
              "      <td>https://64.media.tumblr.com/5de5da8137bb200f0f...</td>\n",
              "      <td>Fri, 19 Feb 2021 18:00:47</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>643560245250113536</td>\n",
              "      <td>https://64.media.tumblr.com/58dcc9f1962b6e86f8...</td>\n",
              "      <td>Fri, 19 Feb 2021 16:01:16</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>643507365074878465</td>\n",
              "      <td>https://64.media.tumblr.com/c08584191e8b323f2f...</td>\n",
              "      <td>Fri, 19 Feb 2021 02:00:46</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>643431872444203008</td>\n",
              "      <td>https://64.media.tumblr.com/34c7335c7d827ed879...</td>\n",
              "      <td>Thu, 18 Feb 2021 06:00:51</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>643401669834670080</td>\n",
              "      <td>https://64.media.tumblr.com/449c1f99908b61385e...</td>\n",
              "      <td>Wed, 17 Feb 2021 22:00:47</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>643386573795295233</td>\n",
              "      <td>https://64.media.tumblr.com/3362f9ec7ad6247aed...</td>\n",
              "      <td>Wed, 17 Feb 2021 18:00:50</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>643326154022043648</td>\n",
              "      <td>https://64.media.tumblr.com/ec0e74f9e710be40e0...</td>\n",
              "      <td>Wed, 17 Feb 2021 02:00:30</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>643243109266915328</td>\n",
              "      <td>https://64.media.tumblr.com/60480331867380096e...</td>\n",
              "      <td>Tue, 16 Feb 2021 04:00:32</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>643235558959841280</td>\n",
              "      <td>https://64.media.tumblr.com/422f87ceb0e3012a3e...</td>\n",
              "      <td>Tue, 16 Feb 2021 02:00:31</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>643220504533155840</td>\n",
              "      <td>https://64.media.tumblr.com/d17596fcd66e688b5e...</td>\n",
              "      <td>Mon, 15 Feb 2021 22:01:14</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>643144953475809280</td>\n",
              "      <td>https://64.media.tumblr.com/6dae560a43db2d85db...</td>\n",
              "      <td>Mon, 15 Feb 2021 02:00:23</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>643129875889307648</td>\n",
              "      <td>https://64.media.tumblr.com/f368cd642649837ef2...</td>\n",
              "      <td>Sun, 14 Feb 2021 22:00:44</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>643024167275495424</td>\n",
              "      <td>https://64.media.tumblr.com/6f21ad1c50c4d9a522...</td>\n",
              "      <td>Sat, 13 Feb 2021 18:00:33</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>642963775101845504</td>\n",
              "      <td>https://64.media.tumblr.com/dc8de324d687bcb87c...</td>\n",
              "      <td>Sat, 13 Feb 2021 02:00:38</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>642918492134408192</td>\n",
              "      <td>https://64.media.tumblr.com/c687f0e116da99764d...</td>\n",
              "      <td>Fri, 12 Feb 2021 14:00:53</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>642910941028204545</td>\n",
              "      <td>https://64.media.tumblr.com/5c5d6493d77ef22afd...</td>\n",
              "      <td>Fri, 12 Feb 2021 12:00:52</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>642903390348820480</td>\n",
              "      <td>https://64.media.tumblr.com/e9c8d6a398ba122aa5...</td>\n",
              "      <td>Fri, 12 Feb 2021 10:00:50</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>641967214574125056</td>\n",
              "      <td>https://64.media.tumblr.com/2fd5041f358f2855de...</td>\n",
              "      <td>Tue, 02 Feb 2021 02:00:44</td>\n",
              "      <td>[gif, lolcat, lolcats, cat, funny]</td>\n",
              "      <td>gif</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>641612367628730368</td>\n",
              "      <td>https://64.media.tumblr.com/9170a88d3e52a51033...</td>\n",
              "      <td>Fri, 29 Jan 2021 04:00:36</td>\n",
              "      <td>[gif, lolcat, lolcats, cat, funny]</td>\n",
              "      <td>gif</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>639732477227892736</td>\n",
              "      <td>https://64.media.tumblr.com/31e8882b2fd934f7b9...</td>\n",
              "      <td>Fri, 08 Jan 2021 10:00:32</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>639724919901585408</td>\n",
              "      <td>https://64.media.tumblr.com/fb868dea41853ad734...</td>\n",
              "      <td>Fri, 08 Jan 2021 08:00:25</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>639717378220015616</td>\n",
              "      <td>https://64.media.tumblr.com/61e3d59b36423880a9...</td>\n",
              "      <td>Fri, 08 Jan 2021 06:00:33</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>639709877361688576</td>\n",
              "      <td>https://64.media.tumblr.com/12f20ceaaa85f5a91e...</td>\n",
              "      <td>Fri, 08 Jan 2021 04:01:19</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>639702313021440000</td>\n",
              "      <td>https://64.media.tumblr.com/2386aff8793f75dcd7...</td>\n",
              "      <td>Fri, 08 Jan 2021 02:01:06</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>639687179569889280</td>\n",
              "      <td>https://64.media.tumblr.com/61e3d59b36423880a9...</td>\n",
              "      <td>Thu, 07 Jan 2021 22:00:33</td>\n",
              "      <td>[cat, cats, lol, lolcat, lolcats]</td>\n",
              "      <td>jpg</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                    id  ... photo-type\n",
              "0   645711888306585600  ...        jpg\n",
              "1   645704327082704896  ...        jpg\n",
              "2   645681731762077696  ...        jpg\n",
              "3   645500515139174400  ...        gif\n",
              "4   644896541426008064  ...        jpg\n",
              "5   644737970778374145  ...        png\n",
              "6   644730428596584448  ...        jpg\n",
              "7   644662504074723328  ...        jpg\n",
              "8   644428418420719616  ...        jpg\n",
              "9   644420865050673153  ...        jpg\n",
              "10  644390682111475712  ...        png\n",
              "11  644322771711754240  ...        jpg\n",
              "12  644201974528704512  ...        jpg\n",
              "13  644194375680835585  ...        jpg\n",
              "14  644179310864957440  ...        jpg\n",
              "15  644149104115679232  ...        jpg\n",
              "16  644141551244771328  ...        jpg\n",
              "17  644126436384505856  ...        jpg\n",
              "18  644058500866228224  ...        jpg\n",
              "19  644050965032517632  ...        jpg\n",
              "20  644035836859547648  ...        jpg\n",
              "21  644020737916715009  ...        jpg\n",
              "22  643597946497138688  ...        jpg\n",
              "23  643582865514397696  ...        jpg\n",
              "24  643575374076428288  ...        jpg\n",
              "25  643567764652638209  ...        jpg\n",
              "26  643560245250113536  ...        jpg\n",
              "27  643507365074878465  ...        jpg\n",
              "28  643431872444203008  ...        jpg\n",
              "29  643401669834670080  ...        jpg\n",
              "30  643386573795295233  ...        jpg\n",
              "31  643326154022043648  ...        jpg\n",
              "32  643243109266915328  ...        jpg\n",
              "33  643235558959841280  ...        jpg\n",
              "34  643220504533155840  ...        jpg\n",
              "35  643144953475809280  ...        jpg\n",
              "36  643129875889307648  ...        jpg\n",
              "37  643024167275495424  ...        jpg\n",
              "38  642963775101845504  ...        jpg\n",
              "39  642918492134408192  ...        jpg\n",
              "40  642910941028204545  ...        jpg\n",
              "41  642903390348820480  ...        jpg\n",
              "42  641967214574125056  ...        gif\n",
              "43  641612367628730368  ...        gif\n",
              "44  639732477227892736  ...        jpg\n",
              "45  639724919901585408  ...        jpg\n",
              "46  639717378220015616  ...        jpg\n",
              "47  639709877361688576  ...        jpg\n",
              "48  639702313021440000  ...        jpg\n",
              "49  639687179569889280  ...        jpg\n",
              "\n",
              "[50 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWJmPygsxuWF"
      },
      "source": [
        "Now we have the urls of a bunch of images and can run OCR on them to gather\n",
        "compelling meme narratives, accompanied by cats.\n",
        "\n",
        "# Files\n",
        "\n",
        "What if the text we want isn't on a webpage? There are a many other sources of\n",
        "text available, typically organized into *files*.\n",
        "\n",
        "## Raw text (and encoding)\n",
        "\n",
        "The most basic form of storing text is as a _raw text_ document. Source code\n",
        "(`.py`, `.r`, etc) is usually raw text as are text files (`.txt`) and those with\n",
        "many other extension (e.g., .csv, .dat, etc.). Opening an unknown file with a\n",
        "text editor is often a great way of learning what the file is.\n",
        "\n",
        "We can create a text file in python with the `open()` function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MqMdjoo2xuWF"
      },
      "source": [
        "#example_text_file = 'sometextfile.txt'\n",
        "#stringToWrite = 'A line\\nAnother line\\nA line with a few unusual symbols \\u2421 \\u241B \\u20A0 \\u20A1 \\u20A2 \\u20A3 \\u0D60\\n'\n",
        "stringToWrite = 'A line\\nAnother line\\nA line with a few unusual symbols ␡ ␛ ₠ ₡ ₢ ₣ ൠ\\n'\n",
        "\n",
        "with open(example_text_file, mode = 'w', encoding='utf-8') as f:\n",
        "    f.write(stringToWrite)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5UYBM0RxuWG"
      },
      "source": [
        "Notice the `encoding='utf-8'` argument, which specifies how we map the bits from\n",
        "the file to the glyphs (and whitespace characters like tab (`'\\t'`) or newline\n",
        "(`'\\n'`)) on the screen. When dealing only with latin letters, arabic numerals\n",
        "and the other symbols on America keyboards you usually do not have to worry\n",
        "about encodings as the ones used today are backwards compatible with\n",
        "[ASCII](https://en.wikipedia.org/wiki/ASCII), which gives the binary\n",
        "representation of 128 characters.\n",
        "\n",
        "Some of you, however, will want to use other characters (e.g., Chinese\n",
        "characters). To solve this there is\n",
        "[Unicode](https://en.wikipedia.org/wiki/Unicode) which assigns numbers to\n",
        "symbols, e.g., 041 is `'A'` and 03A3 is `'Σ'` (numbers starting with 0 are\n",
        "hexadecimal). Often non/beyond-ASCII characters are called Unicode characters.\n",
        "Unicode contains 1,114,112 characters, about 10\\% of which have been assigned.\n",
        "Unfortunately there are many ways used to map combinations of bits to Unicode\n",
        "symbols. The ones you are likely to encounter are called by Python _utf-8_,\n",
        "_utf-16_ and _latin-1_. _utf-8_ is the standard for Linux and Mac OS while both\n",
        "_utf-16_ and _latin-1_ are used by windows. If you use the wrong encoding,\n",
        "characters can appear wrong, sometimes change in number or Python could raise an\n",
        "exception. Lets see what happens when we open the file we just created with\n",
        "different encodings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8JrKKu3oxuWG",
        "outputId": "48a7903d-aaa0-4240-ffe3-08ad7a8d887c"
      },
      "source": [
        "with open(example_text_file, encoding='utf-8') as f:\n",
        "    print(\"This is with the correct encoding:\")\n",
        "    print(f.read())\n",
        "\n",
        "with open(example_text_file, encoding='latin-1') as f:\n",
        "    print(\"This is with the wrong encoding:\")\n",
        "    print(f.read())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "This is with the correct encoding:\n",
            "A line\n",
            "Another line\n",
            "A line with a few unusual symbols ␡ ␛ ₠ ₡ ₢ ₣ ൠ\n",
            "\n",
            "This is with the wrong encoding:\n",
            "A line\n",
            "Another line\n",
            "A line with a few unusual symbols â¡ â â  â¡ â¢ â£ àµ \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDPs8oaoxuWG"
      },
      "source": [
        "Notice that with _latin-1_ the unicode characters are mixed up and there are too\n",
        "many of them. You need to keep in mind encoding when obtaining text files.\n",
        "Determining the encoding can sometime involve substantial work."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "er3tw2l-xuWG"
      },
      "source": [
        "We can also load many text files at once. Lets start by looking at the Shakespeare files in the `data` directory "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W1yuAem8xuWG",
        "outputId": "e2dfb12e-cfea-48c7-9d42-e0d6bbc6e8db"
      },
      "source": [
        "with open('/content/Homework-Notebooks/data/Shakespeare/midsummer_nights_dream.txt') as f:\n",
        "    midsummer = f.read()\n",
        "print(midsummer[-700:])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ", and Train.]\n",
            "\n",
            "PUCK\n",
            "  If we shadows have offended,\n",
            "  Think but this,--and all is mended,--\n",
            "  That you have but slumber'd here\n",
            "  While these visions did appear.\n",
            "  And this weak and idle theme,\n",
            "  No more yielding but a dream,\n",
            "  Gentles, do not reprehend;\n",
            "  If you pardon, we will mend.\n",
            "  And, as I am an honest Puck,\n",
            "  If we have unearned luck\n",
            "  Now to 'scape the serpent's tongue,\n",
            "  We will make amends ere long;\n",
            "  Else the Puck a liar call:\n",
            "  So, good night unto you all.\n",
            "  Give me your hands, if we be friends,\n",
            "  And Robin shall restore amends.\n",
            "\n",
            "[Exit.]\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "End of Project Gutenberg Etext of A Midsummer Night's Dream by Shakespeare\n",
            "PG has multiple editions of William Shakespeare's Complete Works\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLy0UNFOxuWH"
      },
      "source": [
        "By the way, depending on your working directory, you might get errors such as: [Errno 2] No such file or directory: '../data/Shakespeare/midsummer_nights_dream.txt.' Don't panic, it's nothing, just check your working directory. \n",
        "\n",
        "Then to load all the files in `./data/Shakespeare` we can use a for loop with `scandir`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEFjb0azxuWH"
      },
      "source": [
        "targetDir = '/content/Homework-Notebooks/data/Shakespeare' #Change this to your own directory of texts\n",
        "shakespearText = []\n",
        "shakespearFileName = []\n",
        "\n",
        "for file in (file for file in os.scandir(targetDir) if file.is_file() and not file.name.startswith('.')):\n",
        "    with open(file.path, encoding=\"utf-8\") as f:\n",
        "        shakespearText.append(f.read())\n",
        "    shakespearFileName.append(file.name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylDHYMZ8xuWH"
      },
      "source": [
        "Then we can put them all in pandas DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "m5T13RgRxuWH",
        "outputId": "4ebfb283-99e4-4592-86be-355f2ba00e3d"
      },
      "source": [
        "shakespear_df = pandas.DataFrame({'text' : shakespearText}, index = shakespearFileName)\n",
        "shakespear_df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>king_henry_5.txt</th>\n",
              "      <td>THE LIFE OF KING HENRY THE FIFTH\\n\\nby William...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>merchant_of_venice.txt</th>\n",
              "      <td>The Merchant of Venice\\n\\nActus primus.\\n\\nEnt...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>pericles_prince_of_tyre.txt</th>\n",
              "      <td>PERICLES PRINCE OF TYRE\\n\\nby William Shakespe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>coriolanus.txt</th>\n",
              "      <td>THE TRAGEDY OF CORIOLANUS\\n\\nby William Shakes...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tempest.txt</th>\n",
              "      <td>The Tempest\\n\\nActus primus, Scena prima.\\n\\nA...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>timon_of_athens.txt</th>\n",
              "      <td>THE LIFE OF TIMON OF ATHENS\\n\\nby William Shak...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sonnets.txt</th>\n",
              "      <td>THE SONNETS\\n\\nby William Shakespeare\\n\\n\\n\\n ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_8.txt</th>\n",
              "      <td>KING HENRY THE EIGHTH\\n\\nby William Shakespear...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_john.txt</th>\n",
              "      <td>The life and death of King John\\n\\nActus Primu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>comedy_of_errors.txt</th>\n",
              "      <td>DRAMATIS PERSONAE\\n\\nSOLINUS, Duke of Ephesus\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>twelth_night.txt</th>\n",
              "      <td>TWELFTH NIGHT;\\n ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>taming_of_the_shrew.txt</th>\n",
              "      <td>THE TAMING OF THE SHREW\\n\\nby William Shakespe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>phoenix_and_the_turtle.txt</th>\n",
              "      <td>THE PHOENIX AND THE TURTLE\\n\\nby William Shake...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>alls_well_that_ends_well.txt</th>\n",
              "      <td>All's Well, that Ends Well\\n\\nActus primus. Sc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_richard_2.txt</th>\n",
              "      <td>DRAMATIS PERSONAE\\n\\n  KING RICHARD THE SECOND...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>merry_wives_of_windsor.txt</th>\n",
              "      <td>THE MERRY WIVES OF WINDSOR\\n\\nby William Shake...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>venus_and_adonis.txt</th>\n",
              "      <td>VENUS AND ADONIS\\n\\nby William Shakespeare\\n\\n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_6_p3.txt</th>\n",
              "      <td>The third Part of Henry the Sixt\\n\\nwith the d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>loves_labors_lost.txt</th>\n",
              "      <td>LOVE'S LABOUR'S LOST\\n\\nby William Shakespeare...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_6_p2.txt</th>\n",
              "      <td>The second Part of Henry the Sixt\\n\\nwith the ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lovers_complaint.txt</th>\n",
              "      <td>A LOVER'S COMPLAINT\\n\\nby William Shakespeare\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>romeo_and_juliet.txt</th>\n",
              "      <td>ROMEO AND JULIET\\n\\nby William Shakespeare\\n\\n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>as_you_like_it.txt</th>\n",
              "      <td>AS YOU LIKE IT\\n\\nby William Shakespeare\\n\\n\\n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hamlet.txt</th>\n",
              "      <td>The Tragedie of Hamlet\\n\\nActus Primus. Scoena...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rape_of_lucrece.txt</th>\n",
              "      <td>THE RAPE OF LUCRECE\\n\\nby William Shakespeare\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>anthonie_and_cleopatra.txt</th>\n",
              "      <td>The Tragedie of Anthonie, and Cleopatra\\n\\nAct...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>macbeth.txt</th>\n",
              "      <td>MACBETH\\n\\nby William Shakespeare\\n\\n\\n\\n\\nPer...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>othello.txt</th>\n",
              "      <td>THE TRAGEDY OF OTHELLO, MOOR OF VENICE\\n\\nby W...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>much_ado_about_nothing.txt</th>\n",
              "      <td>MUCH ADO ABOUT NOTHING\\n\\nby William Shakspere...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>titus_andronicus.txt</th>\n",
              "      <td>The Tragedie of Titus Andronicus\\n\\nActus Prim...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_4_p2.txt</th>\n",
              "      <td>KING HENRY IV, SECOND PART\\n\\nby William Shake...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>cymbeline.txt</th>\n",
              "      <td>The Tragedie of Cymbeline\\n\\nActus Primus. Sco...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_6_p1.txt</th>\n",
              "      <td>Dramatis Personae\\n\\nKING HENRY the Sixth\\nDUK...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>julius_caesar.txt</th>\n",
              "      <td>Dramatis Personae\\n\\n  JULIUS CAESAR, Roman st...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>midsummer_nights_dream.txt</th>\n",
              "      <td>A MIDSUMMER NIGHT'S DREAM\\n\\nby William Shakes...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>passionate_pilgrim.txt</th>\n",
              "      <td>THE PASSIONATE PILGRIM\\n\\nby William Shakespea...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>troilus_and_cressida.txt</th>\n",
              "      <td>THE HISTORY OF TROILUS AND CRESSIDA\\n\\nby Will...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_richard_3.txt</th>\n",
              "      <td>KING RICHARD III\\n\\nby William Shakespeare\\n\\n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_lear.txt</th>\n",
              "      <td>The Tragedie of King Lear\\n\\n\\nActus Primus. S...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>measure_for_measure.txt</th>\n",
              "      <td>MEASURE FOR MEASURE\\n\\nby William Shakespeare\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>two_gentlemen_of_verona.txt</th>\n",
              "      <td>THE TWO GENTLEMEN OF VERONA\\n\\nby William Shak...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>king_henry_4_p1.txt</th>\n",
              "      <td>The First Part of Henry the Fourth\\n\\nwith the...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>winters_tale.txt</th>\n",
              "      <td>THE WINTER'S TALE\\n\\nby William Shakespeare\\n\\...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                           text\n",
              "king_henry_5.txt              THE LIFE OF KING HENRY THE FIFTH\\n\\nby William...\n",
              "merchant_of_venice.txt        The Merchant of Venice\\n\\nActus primus.\\n\\nEnt...\n",
              "pericles_prince_of_tyre.txt   PERICLES PRINCE OF TYRE\\n\\nby William Shakespe...\n",
              "coriolanus.txt                THE TRAGEDY OF CORIOLANUS\\n\\nby William Shakes...\n",
              "tempest.txt                   The Tempest\\n\\nActus primus, Scena prima.\\n\\nA...\n",
              "timon_of_athens.txt           THE LIFE OF TIMON OF ATHENS\\n\\nby William Shak...\n",
              "sonnets.txt                   THE SONNETS\\n\\nby William Shakespeare\\n\\n\\n\\n ...\n",
              "king_henry_8.txt              KING HENRY THE EIGHTH\\n\\nby William Shakespear...\n",
              "king_john.txt                 The life and death of King John\\n\\nActus Primu...\n",
              "comedy_of_errors.txt          DRAMATIS PERSONAE\\n\\nSOLINUS, Duke of Ephesus\\...\n",
              "twelth_night.txt                                           TWELFTH NIGHT;\\n ...\n",
              "taming_of_the_shrew.txt       THE TAMING OF THE SHREW\\n\\nby William Shakespe...\n",
              "phoenix_and_the_turtle.txt    THE PHOENIX AND THE TURTLE\\n\\nby William Shake...\n",
              "alls_well_that_ends_well.txt  All's Well, that Ends Well\\n\\nActus primus. Sc...\n",
              "king_richard_2.txt            DRAMATIS PERSONAE\\n\\n  KING RICHARD THE SECOND...\n",
              "merry_wives_of_windsor.txt    THE MERRY WIVES OF WINDSOR\\n\\nby William Shake...\n",
              "venus_and_adonis.txt          VENUS AND ADONIS\\n\\nby William Shakespeare\\n\\n...\n",
              "king_henry_6_p3.txt           The third Part of Henry the Sixt\\n\\nwith the d...\n",
              "loves_labors_lost.txt         LOVE'S LABOUR'S LOST\\n\\nby William Shakespeare...\n",
              "king_henry_6_p2.txt           The second Part of Henry the Sixt\\n\\nwith the ...\n",
              "lovers_complaint.txt          A LOVER'S COMPLAINT\\n\\nby William Shakespeare\\...\n",
              "romeo_and_juliet.txt          ROMEO AND JULIET\\n\\nby William Shakespeare\\n\\n...\n",
              "as_you_like_it.txt            AS YOU LIKE IT\\n\\nby William Shakespeare\\n\\n\\n...\n",
              "hamlet.txt                    The Tragedie of Hamlet\\n\\nActus Primus. Scoena...\n",
              "rape_of_lucrece.txt           THE RAPE OF LUCRECE\\n\\nby William Shakespeare\\...\n",
              "anthonie_and_cleopatra.txt    The Tragedie of Anthonie, and Cleopatra\\n\\nAct...\n",
              "macbeth.txt                   MACBETH\\n\\nby William Shakespeare\\n\\n\\n\\n\\nPer...\n",
              "othello.txt                   THE TRAGEDY OF OTHELLO, MOOR OF VENICE\\n\\nby W...\n",
              "much_ado_about_nothing.txt    MUCH ADO ABOUT NOTHING\\n\\nby William Shakspere...\n",
              "titus_andronicus.txt          The Tragedie of Titus Andronicus\\n\\nActus Prim...\n",
              "king_henry_4_p2.txt           KING HENRY IV, SECOND PART\\n\\nby William Shake...\n",
              "cymbeline.txt                 The Tragedie of Cymbeline\\n\\nActus Primus. Sco...\n",
              "king_henry_6_p1.txt           Dramatis Personae\\n\\nKING HENRY the Sixth\\nDUK...\n",
              "julius_caesar.txt             Dramatis Personae\\n\\n  JULIUS CAESAR, Roman st...\n",
              "midsummer_nights_dream.txt    A MIDSUMMER NIGHT'S DREAM\\n\\nby William Shakes...\n",
              "passionate_pilgrim.txt        THE PASSIONATE PILGRIM\\n\\nby William Shakespea...\n",
              "troilus_and_cressida.txt      THE HISTORY OF TROILUS AND CRESSIDA\\n\\nby Will...\n",
              "king_richard_3.txt            KING RICHARD III\\n\\nby William Shakespeare\\n\\n...\n",
              "king_lear.txt                 The Tragedie of King Lear\\n\\n\\nActus Primus. S...\n",
              "measure_for_measure.txt       MEASURE FOR MEASURE\\n\\nby William Shakespeare\\...\n",
              "two_gentlemen_of_verona.txt   THE TWO GENTLEMEN OF VERONA\\n\\nby William Shak...\n",
              "king_henry_4_p1.txt           The First Part of Henry the Fourth\\n\\nwith the...\n",
              "winters_tale.txt              THE WINTER'S TALE\\n\\nby William Shakespeare\\n\\..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bW27Wt8RxuWH"
      },
      "source": [
        "Getting your text in a format like this is the first step of most analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xYeHo4JuxuWH"
      },
      "source": [
        "## PDF\n",
        "\n",
        "Another common way text will be stored is in a PDF file. First we will download\n",
        "a pdf in Python. To do that lets grab a chapter from\n",
        "_Speech and Language Processing_, chapter 21 is on Information Extraction which\n",
        "seems apt. It is stored as a pdf at [https://web.stanford.edu/~jurafsky/slp3/21.\n",
        "pdf](https://web.stanford.edu/~jurafsky/slp3/21.pdf) although we are downloading\n",
        "from a copy just in case Jurafsky changes their website."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UJu7qYnOxuWH",
        "outputId": "647d5086-820e-470c-fbfb-c46e092bf3bb"
      },
      "source": [
        "information_extraction_pdf = 'https://github.com/KnowledgeLab/content_analysis/raw/data/21.pdf'\n",
        "\n",
        "infoExtractionRequest = requests.get(information_extraction_pdf, stream=True)\n",
        "print(infoExtractionRequest.text[:1000])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "%PDF-1.5\n",
            "%����\n",
            "97 0 obj\n",
            "<<\n",
            "/Length 2988      \n",
            "/Filter /FlateDecode\n",
            ">>\n",
            "stream\n",
            "xڝYY��\u0011~�_��T,U�\u0018\u0012����f<{����z\\[)o\u001e(\n",
            "���!�������n��4�J�/b��\u0000\u001a}7�9\u0007�s������o���I�,�|�a�d�\u001b\u0002J��M��y�9?�~<i]\u001cכHE���\t�}�\u001c���e��k\u000b��esp�\u001bߋ�շyS�Jf�\u001b�|�?>��\u0006�׺\u0017�+��y7����==w��8����\u0013\u0004�M<�x�\n",
            "����`?���Q�\u001f*gs�����!���Wn\u001c\u0005\u0010\u0002\u0013)Ϗ�����\u000f�\u0004a�*?p6�\u001bƑ\u00102?��N����wӍq�A�ڽ|h��z�;\u0019%���pƍg�๑���s����\u0014�9��fQ��\u0006�����$����pw��Շ���W[�\u0016��f��{\u001e�M��M�(�M=gã�����l�?��r��s|Ǐ}׏B��o�:E}�\u0013���N\u0006>b�t����{:�M2�\u001d\u0015�I�����\r\u0002߉\u0003�M�Q�k�k�tW�C�6덂\n",
            "�\u001d����\u0005�拍���Y\u001c��o乤�8s\u0019���Fy-��\u0005���g��vGvM ����3���}���[+o�y�\u001b\u0018�\u000fשnٰ|7N\u0012\u00076\u0003U\u001aŽ�f��\u0019��l����D�\u000eD��|[i�n��MY��f�3��W�n`�>���X!~lڧ��>��K�7�\n",
            "�_�d��el\u0007}��g�S�.4:���ve�WƟ�0�.�y���V�\u0010j��\u0018�\u0016���bh��i�)�uW������[\u0007�jGnG�\u0002T,?9�u]\u0013\u000f��-\u00041�s��7e\u0005�\u001d�.��XU%�ޗ5��[��²�؅\u000b���4w�W\u0012��N��淼)��R�MT,�,F4��o�\b>n\r��\u000fP�z\u0013�\u0018��\u0007���s;\n",
            "*��\\��o^=���\u001c���\u0010\u001ad�8��P��P����\u0018@p��^(�&N�[L�d)�H�\u0012����a����\u001awf��\u0011!\u001d�\b�톁�$(dI\u0011f��h�4���\u0015;J,O���X\\�u�$\u000ex\u0006ʗ\r�zx����\u0005�t��|ӎ�\u000e��n\u001d��\u0010�V{�A\u0011v@�Wt:�!bc!*�\u0013��>�\u0014\u000e���΋�@R\u001a�\bQ`��_�A���Aл�y\u001dG+J���z[��\\[G�\u00041-\u0019r��U�m�A�|��'�2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VqTPStwPxuWH"
      },
      "source": [
        "It says `'pdf'`, so thats a good sign. The rest though looks like we are having\n",
        "issues with an encoding. The random characters are not caused by our encoding\n",
        "being wrong, however. They are cause by there not being an encoding for those\n",
        "parts at all. PDFs are nominally binary files, meaning there are sections of\n",
        "binary that are specific to pdf and nothing else so you need something that\n",
        "knows about pdf to read them. To do that we will be using\n",
        "[`PyPDF2`](https://github.com/mstamy2/PyPDF2), a PDF processing library for\n",
        "Python 3.\n",
        "\n",
        "\n",
        "Because PDFs are a very complicated file format pdfminer requires a large amount\n",
        "of boilerplate code to extract text, we have written a function that takes in an\n",
        "open PDF file and returns the text so you don't have to."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybeKUMI0xuWI"
      },
      "source": [
        "def readPDF(pdfFile):\n",
        "    #Based on code from http://stackoverflow.com/a/20905381/4955164\n",
        "    #Using utf-8, if there are a bunch of random symbols try changing this\n",
        "    codec = 'utf-8'\n",
        "    rsrcmgr = pdfminer.pdfinterp.PDFResourceManager()\n",
        "    retstr = io.StringIO()\n",
        "    layoutParams = pdfminer.layout.LAParams()\n",
        "    device = pdfminer.converter.TextConverter(rsrcmgr, retstr, laparams = layoutParams, codec = codec)\n",
        "    #We need a device and an interpreter\n",
        "    interpreter = pdfminer.pdfinterp.PDFPageInterpreter(rsrcmgr, device)\n",
        "    password = ''\n",
        "    maxpages = 0\n",
        "    caching = True\n",
        "    pagenos=set()\n",
        "    for page in pdfminer.pdfpage.PDFPage.get_pages(pdfFile, pagenos, maxpages=maxpages, password=password,caching=caching, check_extractable=True):\n",
        "        interpreter.process_page(page)\n",
        "    device.close()\n",
        "    returnedString = retstr.getvalue()\n",
        "    retstr.close()\n",
        "    return returnedString"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qXJexRfGxuWI"
      },
      "source": [
        "First we need to take the response object and convert it into a 'file like'\n",
        "object so that pdfminer can read it. To do this we will use `io`'s `BytesIO`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xzxmqGR4xuWI"
      },
      "source": [
        "infoExtractionBytes = io.BytesIO(infoExtractionRequest.content)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nya-hSkXxuWJ"
      },
      "source": [
        "Now we can give it to pdfminer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YBKDAGQzxuWJ",
        "outputId": "bbaeb2c8-d7e9-4427-e5b2-4c494085cb2b"
      },
      "source": [
        "print(readPDF(infoExtractionBytes)[:550])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Speech and Language Processing. Daniel Jurafsky & James H. Martin.\n",
            "rights reserved.\n",
            "\n",
            "Draft of November 7, 2016.\n",
            "\n",
            "Copyright c(cid:13) 2016.\n",
            "\n",
            "All\n",
            "\n",
            "CHAPTER\n",
            "\n",
            "21 Information Extraction\n",
            "\n",
            "I am the very model of a modern Major-General,\n",
            "I’ve information vegetable, animal, and mineral,\n",
            "I know the kings of England, and I quote the ﬁghts historical\n",
            "From Marathon to Waterloo, in order categorical...\n",
            "Gilbert and Sullivan, Pirates of Penzance\n",
            "\n",
            "Imagine that you are an analyst with an investment ﬁrm that tracks airline stocks.\n",
            "You’re given the task of determini\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87Q8LR8lxuWJ"
      },
      "source": [
        "From here we can either look at the full text or fiddle with our PDF reader and\n",
        "get more information about individual blocks of text.\n",
        "\n",
        "## Word Docs\n",
        "\n",
        "The other type of document you are likely to encounter is the `.docx`, these are\n",
        "actually a version of [XML](https://en.wikipedia.org/wiki/Office_Open_XML), just\n",
        "like HTML, and like HTML we will use a specialized parser.\n",
        "\n",
        "For this class we will use [`python-docx`](https://python-\n",
        "docx.readthedocs.io/en/latest/) which provides a nice simple interface for\n",
        "reading `.docx` files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G-oEZGmTxuWJ"
      },
      "source": [
        "#example_docx = 'https://github.com/KnowledgeLab/content_analysis/raw/data/example_doc.docx'\n",
        "\n",
        "r = requests.get(example_docx, stream=True)\n",
        "d = docx.Document(io.BytesIO(r.content))\n",
        "for paragraph in d.paragraphs[:7]:\n",
        "    print(paragraph.text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZdpkIsUMxuWJ"
      },
      "source": [
        "This procedure uses the `io.BytesIO` class again, since `docx.Document` expects\n",
        "a file. Another way to do it is to save the document to a file and then read it\n",
        "like any other file. If we do this we can either delete the file afterwords, or\n",
        "save it and avoid downloading the following time.\n",
        "\n",
        "This function is useful as a part of many different tasks so it and others like it will be added to the helper package `lucem_illud_2020` so we can use it later without having to retype it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GmXNtIBZxuWJ"
      },
      "source": [
        "def downloadIfNeeded(targetURL, outputFile, **openkwargs):\n",
        "    if not os.path.isfile(outputFile):\n",
        "        outputDir = os.path.dirname(outputFile)\n",
        "        #This function is a more general os.mkdir()\n",
        "        if len(outputDir) > 0:\n",
        "            os.makedirs(outputDir, exist_ok = True)\n",
        "        r = requests.get(targetURL, stream=True)\n",
        "        #Using a closure like this is generally better than having to\n",
        "        #remember to close the file. There are ways to make this function\n",
        "        #work as a closure too\n",
        "        with open(outputFile, 'wb') as f:\n",
        "            f.write(r.content)\n",
        "    return open(outputFile, **openkwargs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvdCwMYQxuWK"
      },
      "source": [
        "This function will download, save and open `outputFile` as `outputFile` or just\n",
        "open it if `outputFile` exists. By default `open()` will open the file as read\n",
        "only text with the local encoding, which may cause issues if its not a text\n",
        "file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aTiolY8pxuWK"
      },
      "source": [
        "try:\n",
        "    d = docx.Document(downloadIfNeeded(example_docx, example_docx_save))\n",
        "except Exception as e:\n",
        "    print(e)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UJFguRRvxuWK"
      },
      "source": [
        "We need to tell `open()` to read in binary mode (`'rb'`), this is why we added\n",
        "`**openkwargs`, this allows us to pass any keyword arguments (kwargs) from\n",
        "`downloadIfNeeded` to `open()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fzUZVLNqxuWK"
      },
      "source": [
        "d = docx.Document(downloadIfNeeded(example_docx, example_docx_save, mode = 'rb'))\n",
        "for paragraph in d.paragraphs[:7]:\n",
        "    print(paragraph.text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vOfWkafRxuWK"
      },
      "source": [
        "Now we can read the file with `docx.Document` and not have to wait for it to be\n",
        "downloaded every time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rD6x5DdMxuWK"
      },
      "source": [
        "\n",
        "# <span style=\"color:red\">Section 3</span>\n",
        "<span style=\"color:red\">Construct cells immediately below this that extract and organize textual content from text, PDF or Word into a pandas dataframe.</span>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GwoMwNGaxLrU"
      },
      "source": [
        "## Notes\n",
        "In the cells below, I extracted the content of an article titled ***Should social science be more solution-oriented?*** from an online PDF file. I also put the words that I care about and their corresponding sentences into a dataframe. Interestingly, the word `solution` only appears 9 times in the article, although it is in the title.\n",
        "\n",
        "Compared to the traditional `ctrl + F` in a PDF reader, Python seems to be a much more powerful tool for reading PDFs. It can feed us content with important words/jargons and do much more than that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zjhnl22mxuWK",
        "outputId": "2c2333fe-20c3-4aad-e8c1-c0170d0ce364"
      },
      "source": [
        "pdf_3 = 'https://faculty.washington.edu/jwilker/559/shouldss.pdf'\n",
        "infoExtractionRequest_3 = requests.get(pdf_3, stream=True)\n",
        "infoExtractionBytes_3 = io.BytesIO(infoExtractionRequest_3.content)\n",
        "text_content = readPDF(infoExtractionBytes_3)\n",
        "print(text_content[:500])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTICLE NUMBER: 0015\n",
            "\n",
            "PERSPECTIVE\n",
            "\n",
            "Should social science be more solution-oriented?\n",
            "\n",
            "Duncan J. Watts\n",
            "\n",
            "Over the past 100 years, social science has generated a tremendous number of theories on the topics of individual and collec-\n",
            "tive human behaviour. However, it has been much less successful at reconciling the innumerable inconsistencies and contradic-\n",
            "tions among these competing explanations, a situation that has not been resolved by recent advances in ‘c\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AS7J88GQm1hn"
      },
      "source": [
        "# set up some words that are important to me\n",
        "matter_words=['solution', 'machine', 'data', 'social science', 'problem']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m8ovgIt0ovA4"
      },
      "source": [
        "# Using matter words to extract sentences\n",
        "WordSentence = {}\n",
        "for word in matter_words:\n",
        "    WordSentence[word] = []\n",
        "\n",
        "l_matter_words = []\n",
        "l_sentences = []\n",
        "for line in re.findall(r'[^.]{20,}', text_content):\n",
        "    line = re.sub(r'\\n','',line)\n",
        "    if re.search(r'[p]\\s*[p]', line) == None and re.search(r':[\\d]', line) == None:\n",
        "        for word in matter_words:\n",
        "            if word in line:\n",
        "                sentence = re.sub(r'\\n\\d*','',line.strip())\n",
        "                l_matter_words.append(word)\n",
        "                l_sentences.append(sentence)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 415
        },
        "id": "djPU7nnzo7re",
        "outputId": "8170ab04-a052-4064-e1f0-933073785062"
      },
      "source": [
        "text_df = pandas.DataFrame({'Matter Words' : l_keywords, 'Sentence' : l_sentences})\n",
        "text_df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Matter Words</th>\n",
              "      <th>Sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>solution</td>\n",
              "      <td>PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTIC...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>social science</td>\n",
              "      <td>PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTIC...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>social science</td>\n",
              "      <td>WattsOver the past 100 years, social science h...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>social science</td>\n",
              "      <td>However, it has been much less successful at r...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>solution</td>\n",
              "      <td>In this Perspective, I argue that this ‘incohe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>74</th>\n",
              "      <td>problem</td>\n",
              "      <td>If one  could  identify  even  a  handful  of ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>social science</td>\n",
              "      <td>Computational social science</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>76</th>\n",
              "      <td>social science</td>\n",
              "      <td>Can social science matter? Items http://items</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>solution</td>\n",
              "      <td>Should social science be more solution-oriente...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>social science</td>\n",
              "      <td>Should social science be more solution-oriente...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>79 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Matter Words                                           Sentence\n",
              "0         solution  PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTIC...\n",
              "1   social science  PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTIC...\n",
              "2   social science  WattsOver the past 100 years, social science h...\n",
              "3   social science  However, it has been much less successful at r...\n",
              "4         solution  In this Perspective, I argue that this ‘incohe...\n",
              "..             ...                                                ...\n",
              "74         problem  If one  could  identify  even  a  handful  of ...\n",
              "75  social science                       Computational social science\n",
              "76  social science      Can social science matter? Items http://items\n",
              "77        solution  Should social science be more solution-oriente...\n",
              "78  social science  Should social science be more solution-oriente...\n",
              "\n",
              "[79 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gPTi8RrryGdH",
        "outputId": "5dd1a555-e6c3-4263-c302-3e1a79fd519d"
      },
      "source": [
        "text_df[\"Matter Words\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "social science    32\n",
              "problem           30\n",
              "solution           9\n",
              "data               7\n",
              "machine            1\n",
              "Name: Matter Words, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjIiAyPFxuWK"
      },
      "source": [
        "### Other sources:\n",
        "\n",
        "Other popular sources for internet data:\n",
        "\n",
        "[reddit](https://www.reddit.com/) - https://praw.readthedocs.io/en/v2.1.21/\n",
        "\n",
        "[twitter](https://twitter.com/) - https://pypi.org/project/python-twitter/\n",
        "\n",
        "[project gutenburg](https://www.gutenberg.org/) - https://github.com/ageitgey/Gutenberg \n",
        "\n"
      ]
    }
  ]
}