spacy_coreference_resolver_neuralcoref.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ddb45f7f-64bd-4abe-9ae2-de49a93b77b8",
   "metadata": {},
   "source": [
    "### Eploring and testing NeuralCoref\n",
    "\n",
    "[20 juin 2023] \n",
    "\n",
    "I tested the library with a specifc conda environment although it seems not to be developed actively\n",
    "\n",
    "\"NeuralCoref is a pipeline extension for spaCy 2.1+ which annotates and resolves coreference clusters using a neural network. NeuralCoref is production-ready, integrated in spaCy's NLP pipeline and extensible to new training datasets.\"\n",
    "\n",
    "* https://spacy.io/universe/project/neuralcoref \n",
    "* [NeuralCoref 4.0: Coreference Resolution in spaCy with Neural Networks](https://github.com/huggingface/neuralcoref)\n",
    "* https://medium.com/huggingface/state-of-the-art-neural-coreference-resolution-for-chatbots-3302365dcf30\n",
    "* Online Test Service : https://huggingface.co/coref/\n",
    "\n",
    "I also tested changing the parameters.\n",
    "* greedyness=0.55 — default = 0.5 : the new one produces wrong results\n",
    "\n",
    "Results:\n",
    "* It is an older library, not maintained ?\n",
    "* There are some issues, not solved or wrongly solved coreferences\n",
    "* I decided to use _coreferee_\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9357850b-c222-4107-873c-6dacd0e63631",
   "metadata": {},
   "source": [
    "Installation instructions\n",
    "\n",
    "* https://anaconda.org/conda-forge/neuralcoref\n",
    "* The last version appears to be NeuralCoref 4.0\n",
    "* Package neuralcoref-4.0-py37hc9558a2_0 requires python >=3.7,<3.8.0a0, but none of the providers can be installed\n",
    "* Do not install spacy before neuralcoref but let conda choose the suitable dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "085c691a-656d-4f16-a42e-b17bbf357e13",
   "metadata": {},
   "source": [
    "```\n",
    "mamba create -n py3_7_spacy_neuralcoref python=3.7 pandas ipykernel neuralcoref psycopg2 matplotlib itables\n",
    "\n",
    "conda activate py3_7_spacy_neuralcoref\n",
    "jupyter kernelspec list\n",
    "python -m ipykernel install --user --name py3_7_spacy_neuralcoref\n",
    "\n",
    "python -m spacy download en_core_web_lg\n",
    "\n",
    "\n",
    "conda remove --name py3_7_spacy_neuralcoref --all\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "712e116e-3e86-4aaa-8121-7d2c172078c4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "49ed20c8-4d3f-4cb3-9669-1370d9d06acd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import neuralcoref"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4c769851-89ba-487a-a07d-057103403829",
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib import reload\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "import psycopg2\n",
    "import settings as stt\n",
    "\n",
    "from itables import init_notebook_mode, show\n",
    "import re\n",
    "\n",
    "from time import strftime, gmtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7a57bdf-defc-418c-910a-58d465781ca4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3dbd57c8-850d-4697-90b5-a941dde522c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import postgresql_functions as pgf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d9c09e9c-beac-4366-84f7-b1005e1e236d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reload(pgf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f7bea6-a3e9-4c30-92a1-55770292abcb",
   "metadata": {},
   "source": [
    "## Get specimens from database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "555ae1ee-608a-46dd-b72a-cbba4becb720",
   "metadata": {},
   "outputs": [],
   "source": [
    "### connect to the local database\n",
    "conn = psycopg2.connect(host=\"localhost\", port = 5432, database=\"espace_intellectuel\", \n",
    "                        user=\"postgres\", password=stt.dbw)\n",
    "#conn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "80bf535d-cbe0-4997-8955-e0419c393e8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "q1 = \"\"\"\n",
    "select pk_mathshistory, \"name\", url, dates, length(biography) as eff, biography \n",
    "from astronomers.mathshistory m \n",
    "where pk_mathshistory in (103, 117, 133, 159, 186);\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "36cac17f-ae81-471b-909b-1172c96838b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = pgf.sql_explore(q1, conn)\n",
    "# print(f'Lines count: {len(result[0])}, errors count: {len(result[1])}, \\nFirst lines: {result[0][:5]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6ff46ad1-b1f7-47cc-8302-04a6b6cb2ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "textes = pd.DataFrame(result[0])\n",
    "textes.columns = ['id', 'name', 'url', 'dates', 'length_bio', 'texte']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cbfead55-0dd5-4393-8fac-16b6e6671f74",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "      <th>dates</th>\n",
       "      <th>length_bio</th>\n",
       "      <th>texte</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>103</td>\n",
       "      <td>Christopher Clavius</td>\n",
       "      <td>https://mathshistory.st-andrews.ac.uk/Biograph...</td>\n",
       "      <td>1538-1612</td>\n",
       "      <td>10616</td>\n",
       "      <td>Christopher Clavius was born in a German regio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>117</td>\n",
       "      <td>Michael Mästlin</td>\n",
       "      <td>https://mathshistory.st-andrews.ac.uk/Biograph...</td>\n",
       "      <td>1550-1631</td>\n",
       "      <td>10122</td>\n",
       "      <td>Michael Mästlin was born in Göppingen which wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>133</td>\n",
       "      <td>Giuseppe Biancani</td>\n",
       "      <td>https://mathshistory.st-andrews.ac.uk/Biograph...</td>\n",
       "      <td>1566-1624</td>\n",
       "      <td>10035</td>\n",
       "      <td>Giuseppe Biancani's name also appears in its L...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>159</td>\n",
       "      <td>Wilhelm Schickard</td>\n",
       "      <td>https://mathshistory.st-andrews.ac.uk/Biograph...</td>\n",
       "      <td>1592-1635</td>\n",
       "      <td>10272</td>\n",
       "      <td>Wilhelm Schickard's name is sometimes written ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>186</td>\n",
       "      <td>Johannes Hevelius</td>\n",
       "      <td>https://mathshistory.st-andrews.ac.uk/Biograph...</td>\n",
       "      <td>1611-1687</td>\n",
       "      <td>10856</td>\n",
       "      <td>The first problem that we have to address is t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                 name  \\\n",
       "0  103  Christopher Clavius   \n",
       "1  117      Michael Mästlin   \n",
       "2  133    Giuseppe Biancani   \n",
       "3  159    Wilhelm Schickard   \n",
       "4  186    Johannes Hevelius   \n",
       "\n",
       "                                                 url      dates  length_bio  \\\n",
       "0  https://mathshistory.st-andrews.ac.uk/Biograph...  1538-1612       10616   \n",
       "1  https://mathshistory.st-andrews.ac.uk/Biograph...  1550-1631       10122   \n",
       "2  https://mathshistory.st-andrews.ac.uk/Biograph...  1566-1624       10035   \n",
       "3  https://mathshistory.st-andrews.ac.uk/Biograph...  1592-1635       10272   \n",
       "4  https://mathshistory.st-andrews.ac.uk/Biograph...  1611-1687       10856   \n",
       "\n",
       "                                               texte  \n",
       "0  Christopher Clavius was born in a German regio...  \n",
       "1  Michael Mästlin was born in Göppingen which wa...  \n",
       "2  Giuseppe Biancani's name also appears in its L...  \n",
       "3  Wilhelm Schickard's name is sometimes written ...  \n",
       "4  The first problem that we have to address is t...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cc57c251-1549-42c2-a7ed-f4b725586764",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Michael Mästlin was born in Göppingen which was a village about 50 km east of Tübingen. His father, Jakob Mästlin, and his mother, Dorothea Simon, were both devout Lutherans and Michael was brought up in that faith and remained strongly committed to it throughout his life. He was the middle child of the family, having an older sister and a younger brother. He attended the monastic school in Königsbronn then, after his studies there, entered Tübingen University in 1568. [3]:-\n",
      "As was the case with many young scholars including Kepler, his most famous student, [Mästlin] did his undergraduate studies at a preparatory school and came to the university to take his final exams and pick up his baccalaureate degree.At Tübingen University he studied mathematics and astronomy for a Master's degree under Philipp Apian who was Peter Apian's son. In 1570, while a student, he purchased a copy of Copernicus's De revolutionibus from the widow of Victorin Strigel, who had been professor of theology at Leipzig and the author of an astronomy text. Mästlin was awarded a Master's degree with distinction in 1571. While studying for this degree he had edited a new edition of the Prutenis Tables which was published in 1571. The Prutenis Tables had been originally compiled by Erasmus Reinhold who had based them on Copernicus's version of the solar system. After graduating, Mästlin spent the following years teaching in Tübingen, and at the same time took [3]:-\n",
      "... the theological programme, because Tübingen University's primary mission was to prepare young men for the Lutheran ministry.Then, in 1576, he was sent to be a deacon at the Lutheran Church in Backnang, a town about 30 km northwest of Göppingen. He married Margarete Gruuninger (1551-1588) in April 1577 not long after he took up the position in Backnang; they had three sons and three daughters but Margarete died due to complications with the birth of their sixth child. Mästlin then married Margarete Burkhardt, daughter of a professor at Tübingen; they had eight children.\n",
      "\n",
      "Mästlin is famous for his excellent, very accurate, observations of the comet of 1577, observed while he was in Backnang, and published in Tübingen in 1578 as Observatio et demonstratio cometae aetherae qui anno 1577 et 1578 constitutus in sphaera Veneris apparuit cum admirandius eius passionibus varietate scilicet motus loco orbe distantia a terro centro etc. adhibitis demonstrationibus geometricis et calculo arithmetico cuius modi de alio quoquam cometa nunquam visa est (Observations and demonstrations of the ethereal comets of the years 1577 and 1578). We discuss this achievement in more detail below. He remained in Backnang for four years, then was appointed as professor of mathematics at the University of Heidelberg in 1580. There he published the first edition of his famous astronomy textbook Epitome astronomiae (1582) - he published six further editions of this popular work during his lifetime. Despite his commitment to the views of Copernicus (which we state below in his own words) this teaching textbook was written purely as a description of astronomy based on the geocentric model of Ptolemy.\n",
      "\n",
      "After four years in Heidelberg he returned to his position in Tübingen where he spent the rest of his career [2]:-\n",
      "At Tübingen, Mästlin was elected dean of the arts faculty several times. He was well liked by both his colleagues and his students. Mästlin was very generous both to his family and to others. He was a religious man; he followed the Lutheran line in opposing the Gregorian calendar reform partly because it was initiated by the Pope. Mästlin had several students who became noted mathematicians, the most famous being Kepler. Mästlin also maintained interests in Biblical chronology and geography.Perhaps his greatest achievement (other than being Kepler's teacher) is that he was the first to compute the orbit of a comet, although his method was not sound. He found, however, a sun centred orbit for the comet of 1577 which he claimed supported Copernicus's heliocentric system. He did show that the comet was further away than the moon, which contradicted the accepted teachings of Aristotle. Although clearly believing in the system as proposed by Copernicus, he taught astronomy using his own textbook which was based on Ptolemy's system. However for the more advanced lectures he adopted the heliocentric approach - Kepler credited Mästlin with introducing him to Copernican ideas while he was a student at Tübingen (1589-94). In [6] Methuen looks at these two different world systems in Mästlin's teaching. A G Molland reviews Methuen's paper:-\n",
      "Michael Mästlin is regularly ascribed a firm if minor place in accounts of the Astronomical Revolution, as the teacher of Johannes Kepler and as an early believer in the physical reality of the Copernican system. What has been less clear is how this related to his regular teaching at the University of Tübingen, with some having argued that he only taught Copernicanism cautiously and in private. Methuen examines this question in the light of Maestlin's published writings, including five sets of theses over whose disputations (two at Heidelberg, one at Tübingen) he presided. She shows that he had no compunction about firmly asserting the supralunar position of the New Star of 1572 and of the comets of 1577 - 1578 and 1580. In general, his writings, especially his textbook 'Epitome astronomiae' of 1582, show no firm Copernican commitment, although there are signs of a leaning in that direction, and he was concerned with emphasizing the interconvertibility of hypotheses. His elementary teaching was certainly based on traditional astronomy, but Methuen concludes that he taught newer material to his more advanced students. She claims that he made a clear distinction between \"spheres\" and \"orbs\", with the former possessing physical reality and the latter being merely useful mathematical constructions ...Mästlin was both a great expert on spherical trigonometry and also a fine observer producing accurate data - the quality of his eyesight is seen from the fact that he saw, and sketched the positions of, 11 stars in the Pleiades cluster. Of course there were not any bright city lights around then, but we challenge any reader to equal this achievement however dark a site they find. He seems to have been the first to claim that the dark part of the moon shone through sunlight reflected from the earth but Leonardo da Vinci has also been credited with this idea. Another first for Mästlin is an accurate calculation of the golden ratio as \"approximately 0.6180340\" stated in a letter he wrote to Kepler in 1597.\n",
      "\n",
      "He was an innovative thinker who was quite prepared to challenge conventional views. For example he attempted to measure the parallax of a supernova and, having failed to find any, deduced that it was as far away as the \"fixed stars\". This of course contradicted the view, held since Aristotle, that all changes in the heavens occurred closer to earth than the realm of the stars which was unchanging.\n",
      "\n",
      "Mästlin lived to see the invention of the telescope for astronomical observations by Galileo. He had two, rather poor, telescopes with which he was able to observe sunspots and the moons of Jupiter. He was still making accurate observations at the age of 78 when he observed the lunar eclipse of 1628.\n",
      "\n",
      "As we mentioned above Mästlin acquired a copy of Copernicus's De revolutionibus in 1570 and he wrote extensive notes near the beginning of the book. These give much insight into his views on Copernicus and we quote the notes using Gingerich's translations. Mästlin wrote that [3]:-\n",
      "... the arrangement presented in this book is the sort of structure in which all the sidereal motions and phenomena are explained very exactly. Therefore this hypothesis recommends itself to the intellect.Mästlin continued:-\n",
      "The heavenly motions were at the point of collapse, and so [Copernicus] concluded that appropriate hypotheses were needed to explain these motions. When he noticed that the common hypotheses were insufficient, he eventually accepted the idea of the Earth's mobility, since indeed, it not only satisfied the phenomena very well but it didn't lead to anything absurd.\n",
      "\n",
      "In fact, if anyone would straighten out the common hypotheses so that they would agree with the phenomena and allow no inconsistencies, then I would gratefully trust him; clearly he would bring very many to his views. But I see that some, even very outstanding mathematicians, have laboured on this yet, in the end, without results. Therefore, I think that unless the common hypotheses are reformed (a task that I am not up to because of my inadequate abilities), I will accept the hypotheses and opinion of Copernicus - after Ptolemy, the prince of all Astronomers.There are further annotations written by Mästlin [3] which are very interesting. On the back of the title page of De revolutionibus is the infamous notice which states that \"... these hypotheses need not be true nor even probable; it is sufficient if the calculations agree with the observations.\" Mästlin adds a note to his copy stating:- \n",
      "This preface was added by someone, whoever its author may be, (for indeed, its weakness of style and choice of words reveal that it is not by Copernicus).He later added a further note:-\n",
      "I found the following words written somewhere among the books of Philipp Apian (which I bought from his widow); although no author was given I could recognise Apian's hand:\n",
      "On account of this letter Georg Joachim Rheticus, the Leipzig professor and disciple of Copernicus, became involved in a very bitter wrangle with the printer, who asserted that it had been turned over to him with the rest of the work. Rheticus, however, suspected that Osiander [the proof-reader of Copernicus's book] had prefaced it to the work. If he knew this for certain, he declared, he would handle that fellow so that in future he would mind his own business and not slander astronomers any more. Nevertheless, [Peter] Apian told me that Osiander had openly admitted to him that he had added this all by himself.\n"
     ]
    }
   ],
   "source": [
    "### Choose one document\n",
    "tx = textes.iloc[1].texte \n",
    "print(tx)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abc3f370-a920-43eb-8584-2abac04ac111",
   "metadata": {},
   "source": [
    "## NLP pipeline with NeuralCoref"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3c25d137-52f2-4617-932b-0bfec1988db0",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_lg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "f71f22f5-1c0a-4db3-ac64-894ab06e9228",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<spacy.lang.en.English at 0x7f58f6d3a950>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neuralcoref.add_to_pipe(nlp, max_dist= 300, max_dist_match = 4000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aff4de92-4add-475f-b18e-5e803a0a9a76",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "7df67835-c90f-4125-b110-ea913e4684c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(tx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "861f792f-26bc-49a4-98a5-dabe5d1d31aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc._.has_coref"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "19aead54-6f5f-4e45-a412-caad1b25093c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Michael Mästlin was born in Göppingen which was a village about 50 km east of Tübingen. Michael Mästlin father, Jakob Mästlin, and Michael Mästlin mother, Dorothea Simon, were both devout Lutherans and Michael was brought up in that faith and remained strongly committed to it throughout Michael Mästlin life. Michael Mästlin was the middle child of the family, having an older sister and a younger brother. Michael Mästlin attended the monastic school in Königsbronn then, after Michael Mästlin studies there, entered Tübingen University in 1568. [3]:-\\nAs was the case with many young scholars inclu'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_text = doc._.coref_resolved\n",
    "new_text[:600]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3cc74680-c4d9-4e96-99ac-e5f64e837081",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Michael Mästlin: [Michael Mästlin, His, his, his, He, He, his, his, his, his, his, he, Mästlin, he, Mästlin, he, He, he, Mästlin, Mästlin, his, he, Mästlin, Mästlin, his, He, he, Mästlin, Mästlin, his, he, his, He, he, He, he, his, he, Mästlin, him, he, Mästlin, Mästlin, Mästlin, Mästlin, He, he, He, he, Mästlin, he, his, Mästlin, he, he, him, he, his, Mästlin, his],\n",
       " Göppingen: [Göppingen, Göppingen],\n",
       " Tübingen: [Tübingen, Tübingen, Tübingen, Tübingen, Tübingen, Tübingen, Tübingen, Tübingen],\n",
       " Tübingen University: [Tübingen University, the university, Tübingen University, Tübingen University, Tübingen University's],\n",
       " Copernicus: [Copernicus, Copernicus],\n",
       " Backnang: [Backnang, Backnang],\n",
       " three sons and three daughters: [three sons and three daughters, their],\n",
       " Margarete Burkhardt, daughter of a professor at Tübingen: [Margarete Burkhardt, daughter of a professor at Tübingen, they],\n",
       " He: [He, he, his, he, his, his, his, he, his, he, his, He, his, his, his, he],\n",
       " Heidelberg: [Heidelberg, Heidelberg, Heidelberg],\n",
       " the Gregorian calendar reform: [the Gregorian calendar reform, it],\n",
       " a comet: [a comet, the comet],\n",
       " Copernicus's heliocentric system: [Copernicus's heliocentric system, the system, the Copernican system]]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc._.coref_clusters[:13]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "17535f92-7e89-428a-82e3-ce421f686975",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'; Margarete Burkhardt, daughter of a professor at Tübingen had eight children.\\n\\nMichael Mästlin is famous for Michael Mästlin excellent, very accurate, observations of the comet of 1577, observed while Michael Mästlin was in Backnang, and published in Tübingen in 1578 as Observatio et demonstratio cometae aetherae qui anno 1577 et 1578 constitutus in sphaera Veneris apparuit cum admirandius eius passionibus varietate scilicet motus loco orbe distantia a terro centro etc. adhibitis demonstrationibus geometricis et calculo arithmetico cuius modi de alio quoquam cometa nunquam visa est (Observations and demonstrations of the ethereal comets of the years 1577 and 1578). We discuss this achievement in more detail below. He remained in Backnang for four years, then was appointed as professor of mathematics at the University of Heidelberg in 1580. There He published the first edition of He famous astronomy textbook Epitome astronomiae (1582) - He published six further editions of this popular work during He lifetime. Despite He commitment to the views of Copernicus (which we state below in He own words) this teaching textbook was written purely as a description of astronomy based on the geocentric model of Ptolemy.\\n\\nAfter four years in Heidelberg He returned to He position in Tübingen where He spent the rest of He career [2]:-\\nAt Tübingen, Michael Mästlin was elected dean of the arts faculty several times. He was well liked by both He colleagues and He students. Michael Mästlin was very generous both to Michael Mästlin family and to others. Michael Mästlin was a religious man; Michael Mästlin followed the Lutheran line in opposing the Gregorian calendar reform partly because the Gregorian calendar reform was initiated by the Pope. Michael Mästlin had several students who became noted mathematicians, the most famous being Kepler. Michael Mästlin also maintained interests in Biblical chronology and geography.Perhaps Michael Mästlin greatest achievement (other than being Kepler\\'s teacher) is that Michael Mästlin was the first to compute the orbit of a comet, although Michael Mästlin method was not sound. Michael Mästlin found, however, a sun centred orbit for the comet of 1577 which Michael Mästlin claimed supported Copernicus\\'s heliocentric system. Michael Mästlin did show that a comet was further away than the moon, which contradicted the accepted teachings of Aristotle. Although clearly believing in Copernicus\\'s heliocentric system as proposed by Copernicus, Michael Mästlin taught astronomy using Michael Mästlin own textbook which was based on Ptolemy\\'s system. However for the more advanced lectures Michael Mästlin adopted the heliocentric approach - Kepler credited Michael Mästlin with introducing Michael Mästlin to Copernican ideas while Michael Mästlin was a student at Tübingen (1589-94). In [6] Methuen looks at these two different world systems in Michael Mästlin\\'s teaching. A G Molland reviews Methuen\\'s paper:-\\nMichael Mästlin is regularly ascribed a firm if minor place in accounts of the Astronomical Revolution, as the teacher of Johannes Kepler and as an early believer in the physical reality of Copernicus\\'s heliocentric system. What has been less clear is how this related to He regular teaching at the University of Tübingen, with some having argued that He only taught Copernicanism cautiously and in private. Methuen examines this question in the light of Maestlin\\'s published writings, including five sets of theses over whose disputations (two at Heidelberg, one at Tübingen) Methuen presided. Methuen shows that Methuen had no compunction about firmly asserting the supralunar position of the New Star of 1572 and of the comets of 1577 - 1578 and 1580. In general, Methuen writings, especially Methuen textbook \\'Epitome astronomiae\\' of 1582, show no firm Copernican commitment, although there are signs of a leaning in that direction, and Methuen was concerned with emphasizing the interconvertibility of hypotheses. Methuen elementary teaching was certainly based on traditional astronomy, but Methuen concludes that Methuen taught newer material to Methuen more advanced students. Methuen claims that Methuen made a clear distinction between \"spheres\" and \"orbs\", with the former possessing physical reality and the latter being merely useful mathematical constructions ...Michael Mästlin was both a great expert on spherical trigonometry and also a fine observer producing accurate data - the quality of Methuen eyesight is seen from the fact that Methuen saw, and sketched the positions of, 11 stars in the Pleiades cluster. Of course there were not any bright city lights around then, but we challenge any reader to equal this achievement however dark a site they find. Methuen seems to have been the first to claim that the dark part of the moon shone through sunlight reflected from the Earth but Leonardo da Vinci has also been credited with this idea. Another first for Michael Mästlin is an accurate calculation of the golden ratio as \"approximately 0.6180340\" stated in a letter Methuen wrote to Kepler in 1597.\\n\\nMethuen was an innovative thinker who was quite prepared to challenge conventional views. For example Methuen attempted to measure the parallax of a supernova and, having failed to find any, deduced that it was as far away as the \"fixed stars\". This of course contradicted the view, held since Aristotle, that all changes in the heavens occurred closer to earth than the realm of the stars which was unchanging.\\n\\nMichael Mästlin lived to see the invention of the telescope for astronomical observations by Galileo. Michael Mästlin had two, rather poor, telescopes with which Michael Mästlin was able to observe sunspots and the moons of Jupiter. Michael Mästlin was still making accurate observations at the age of 78 when Michael Mästlin observed the lunar eclipse of 1628.\\n\\nAs we mentioned above Michael Mästlin acquired a copy of Copernicus\\'s De revolutionibus in 1570 and Michael Mästlin wrote extensive notes near the beginning of the book. These give much insight into Michael Mästlin views on Copernicus and we quote the notes using Gingerich\\'s translations. Michael Mästlin wrote that [3]:-\\n... the arrangement presented in the book is the sort of structure in which all the sidereal motions and phenomena are explained very exactly. Therefore this hypothesis recommends this hypothesis to the intellect.Mästlin continued:-\\nThe heavenly motions were at the point of collapse, and so CopernicusCopernicus] concluded that appropriate hypotheses were needed to explain The heavenly motions. When Michael Mästlin noticed that the common hypotheses were insufficient, Michael Mästlin eventually accepted the idea of the Earth\\'s mobility, since indeed, it not only satisfied the phenomena very well but it didn\\'t lead to anything absurd.\\n\\nIn fact, if anyone would straighten out the common hypotheses so that anyone would agree with the phenomena and allow no inconsistencies, then I would gratefully trust Michael Mästlin; clearly Michael Mästlin would bring very many to his views on Copernicus. But I see that some, even very outstanding mathematicians, have laboured on this yet, in the end, without results. Therefore, I think that unless the common hypotheses are reformed (a task that I am not up to because of my inadequate abilities), I will accept the hypotheses and opinion of Copernicus - after Ptolemy, the prince of all Astronomers.There are further annotations written by Mästlin [3] which are very interesting. On the back of the title page of De revolutionibus is the infamous notice which states that \"... these hypotheses need not be true nor even probable; it is sufficient if the calculations agree with the observations.\" Michael Mästlin adds a note to Michael Mästlin copy stating:- \\nThis preface was added by someone, whoever This preface author may be, (for indeed, This preface weakness of style and choice of words reveal that it is not by Copernicus).He later added a further note:-\\nI found the following words written somewhere among the books of Philipp Apian (which I bought from his widow); although no author was given I could recognise Apian\\'s hand:\\nOn account of this letter Georg Joachim Rheticus, the Leipzig professor and disciple of Copernicus, became involved in a very bitter wrangle with the printer, who asserted that it had been turned over to him with the rest of the work. Rheticus, however, suspected that Osiander [the proof-reader of Copernicus\\'s book] had prefaced it to the work. If him knew this for certain, him declared, him would handle that fellow so that in future him would mind him own business and not slander astronomers any more. Nevertheless, [Peter] Apian told me that Osiander had openly admitted to Peter] that Peter] had added this all by Peter].'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_text[2250:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9280bb4-2a98-4659-bcbb-3ab0ea2cf411",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "726cb091-cbbc-4e00-ace2-53c8e2df0885",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Michael Mästlin, {Michael Mästlin: -0.00941610336303711}],\n",
       " [Göppingen,\n",
       "  {Göppingen: 1.6938384771347046, Michael Mästlin: -1.7341481447219849}],\n",
       " [a village about 50 km east of Tübingen,\n",
       "  {a village about 50 km east of Tübingen: 1.791876196861267,\n",
       "   Michael Mästlin: -1.529648780822754,\n",
       "   Göppingen: -1.5075124502182007}],\n",
       " [50 km,\n",
       "  {50 km: 1.8876539468765259,\n",
       "   Michael Mästlin: -1.6026262044906616,\n",
       "   Göppingen: -1.5367844104766846,\n",
       "   a village about 50 km east of Tübingen: -1.5255197286605835}],\n",
       " [Tübingen,\n",
       "  {Tübingen: 1.6877461671829224,\n",
       "   Michael Mästlin: -1.7620389461517334,\n",
       "   Göppingen: -1.5420584678649902,\n",
       "   a village about 50 km east of Tübingen: -1.5022647380828857,\n",
       "   50 km: -1.520662546157837}]]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cor_sco = [[i,v] for i,v in doc._.coref_scores.items()]\n",
    "cor_sco[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac6906c0-85d5-4672-8eb7-d2ff11112568",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8b9f3a23-e589-42c9-b58b-f68bee033856",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "82"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Produce a list of the sentences in the document and add an index to each sentence \n",
    "sents = [[i, s, [nc for nc in s.noun_chunks]]  for i, s in enumerate(doc.sents)]\n",
    "len(sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "aead1dc5-9e9b-4864-afff-82b61575161a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----\n",
      " [0, Michael Mästlin was born in Göppingen which was a village about 50 km east of Tübingen., [Michael Mästlin, Göppingen, a village, Tübingen]]\n",
      "----\n",
      " [1, His father, Jakob Mästlin, and his mother, Dorothea Simon, were both devout Lutherans and Michael was brought up in that faith and remained strongly committed to it throughout his life., [His father, Jakob Mästlin, his mother, Dorothea Simon, devout Lutherans, Michael, that faith, it, his life]]\n",
      "----\n",
      " [2, He was the middle child of the family, having an older sister and a younger brother., [He, the middle child, the family, an older sister, a younger brother]]\n",
      "----\n",
      " [3, He attended the monastic school in Königsbronn then, after his studies there, entered Tübingen University in 1568., [He, the monastic school, Königsbronn, his studies, Tübingen University]]\n",
      "----\n",
      " [4, [3]:-\n",
      "As was the case with many young scholars including Kepler, his most famous student, [Mästlin] did his undergraduate studies at a preparatory school and came to the university to take his final exams and pick up his baccalaureate degree., [the case, many young scholars, Kepler, his most famous student, [Mästlin, his undergraduate studies, a preparatory school, the university, his final exams, his baccalaureate degree]]\n"
     ]
    }
   ],
   "source": [
    "for sent in sents[:5]:\n",
    "    print('----\\n',sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191f0add-ea29-4826-88af-37faae5e9bc7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "418782e4-c999-4260-82c4-74771f48e26d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "667ad16b-8ee7-4f63-82ad-cb28416580cb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py3_7_spacy_neuralcoref",
   "language": "python",
   "name": "py3_7_spacy_neuralcoref"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}