W4D2-Exercise_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data formats and open data\n",
    "**Exercises for week 4B** in Digital Methods, University of Copenhagen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be working with two main types of data formats: tabular and non-tabular. In 99% of practical scenarios\n",
    "that translates to files in either CSV format (tabular) or JSON format (non-tabular)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 1**: Download [this file](https://www.dropbox.com/s/8ntns0td580be3i/output_got.csv?dl=0) and load it as a list\n",
    "of lists. Print the first 10 lines, as well as the total number of lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T12:58:39.496624Z",
     "start_time": "2020-02-29T12:58:39.492916Z"
    }
   },
   "outputs": [],
   "source": [
    "lines = []\n",
    "with open('output_got.csv') as fp:\n",
    "    for line in fp:\n",
    "        lines.append(line.split(\",\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T12:58:40.283664Z",
     "start_time": "2020-02-29T12:58:40.278042Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['date',\n",
       "  'username',\n",
       "  'to',\n",
       "  'replies',\n",
       "  'retweets',\n",
       "  'favorites',\n",
       "  'text',\n",
       "  'geo',\n",
       "  'mentions',\n",
       "  'hashtags',\n",
       "  'id',\n",
       "  'permalink\\n'],\n",
       " ['2020-02-28 08:15:19',\n",
       "  'raebengala',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"Per tranquillizzarci ci dicono che morti #coronavirus sono quasi tutti 80enni. Ci chiedono di tornare normalità',\n",
       "  ' come se questi #anziani non contassero nulla. Pochi giorni fa parlavamo di come si viva più a lungo. Oggi ci sembra una fortuna che muoiano \"\"solo\"\" anziani #COVID2019\"',\n",
       "  '',\n",
       "  '',\n",
       "  '#coronavirus #anziani #COVID2019',\n",
       "  '1233304662671069184',\n",
       "  'https://twitter.com/raebengala/status/1233304662671069184\\n'],\n",
       " ['2020-02-28 08:15:19',\n",
       "  'martin_lainez',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"El sevillano con coronavirus: «Mando un mensaje de tranquilidad',\n",
       "  ' tenemos una sanidad preparada» https://sevilla.abc.es/sevilla/sevi-sevillano-coronavirus-mando-mensaje-tranquilidad-tenemos-sanidad-preparada-202002272347_noticia.html#vca=rrss-inducido&vmc=abcdesevilla-es&vso=tw&vli=noticia-entrevista … vía @abcdesevilla @RamonRomanR\"',\n",
       "  '',\n",
       "  '@abcdesevilla @RamonRomanR',\n",
       "  '#vca',\n",
       "  '1233304662092152832',\n",
       "  'https://twitter.com/martin_lainez/status/1233304662092152832\\n'],\n",
       " ['2020-02-28 08:15:19',\n",
       "  'Davenews5',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"#HDPROS Frontières/coronavirus : contrôler les frontières c\\'est prévenir et tenter d\\'endiguer la propagation en amont. Les autorités ont renoncé à cela et tentent seulement de réagir localement sans empêcher de nouveaux cas potentiels de venir en France. #Minable #Irresponsable\"',\n",
       "  '',\n",
       "  '',\n",
       "  '#HDPROS #Minable #Irresponsable',\n",
       "  '1233304660473151488',\n",
       "  'https://twitter.com/Davenews5/status/1233304660473151488\\n'],\n",
       " ['2020-02-28 08:15:19',\n",
       "  'nnoeliaperez',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"No quiero ser alarmista pero VAMOS A MORIR ALFODFK jaja jk jk dudes! Este año he tenido gripe y resfriado',\n",
       "  ' el coronavirus no me preocupa xD\"',\n",
       "  '',\n",
       "  '',\n",
       "  '',\n",
       "  '1233304660188041216',\n",
       "  'https://twitter.com/nnoeliaperez/status/1233304660188041216\\n'],\n",
       " ['2020-02-28 08:15:19',\n",
       "  'paulstpancras',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"Why Trump is secretly terrified of coronavirus • https://www.independent.co.uk/voices/coronavirus-trump-vaccine-ebola-stock-market-pandemic-economy-election-november-a9359301.html … •\"',\n",
       "  '',\n",
       "  '',\n",
       "  '',\n",
       "  '1233304659156180994',\n",
       "  'https://twitter.com/paulstpancras/status/1233304659156180994\\n'],\n",
       " ['2020-02-28 08:15:18',\n",
       "  'Nova153',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"L’Italie enquête sur les prix élevés des masques et des gels antibactériens contre le coronavirus https://www.nouvelordremondial.cc/2020/02/28/litalie-enquete-sur-les-prix-eleves-des-masques-et-des-gels-antibacteriens-contre-le-coronavirus/ …\"',\n",
       "  '',\n",
       "  '',\n",
       "  '',\n",
       "  '1233304658237628417',\n",
       "  'https://twitter.com/Nova153/status/1233304658237628417\\n'],\n",
       " ['2020-02-28 08:15:18',\n",
       "  '74_CortoMaltese',\n",
       "  'francescatotolo',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"@francescatotolo ti vorrei ricordare che Burioni il 2 Febbraio diceva che in Italia il rischio di coronavirus era Zero. La gismondo ha ragione',\n",
       "  ' parla sencondo dati',\n",
       "  ' i morti avevano già patologie importanti il coronavirus è stata un aggravante.\"',\n",
       "  '',\n",
       "  '@francescatotolo',\n",
       "  '',\n",
       "  '1233304658191491072',\n",
       "  'https://twitter.com/74_CortoMaltese/status/1233304658191491072\\n'],\n",
       " ['2020-02-28 08:15:18',\n",
       "  'Official_Jo_El',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"the corona virus dier if ereach here paaa ego regret o nbs taabea then rooter mixture we go take kill am\"',\n",
       "  '',\n",
       "  '',\n",
       "  '',\n",
       "  '1233304657591664640',\n",
       "  'https://twitter.com/Official_Jo_El/status/1233304657591664640\\n'],\n",
       " ['2020-02-28 08:15:18',\n",
       "  'mmusa5053',\n",
       "  '',\n",
       "  '0',\n",
       "  '0',\n",
       "  '0',\n",
       "  '\"Coronavirus in Africa: These are the 3 countries with the highest risk https://www.pulselive.co.ke/bi/lifestyle/coronavirus-in-africa-these-are-the-3-countries-with-the-highest-risk/rzveer1 … via @Twitter\"',\n",
       "  '',\n",
       "  '@Twitter',\n",
       "  '',\n",
       "  '1233304656715096064',\n",
       "  'https://twitter.com/mmusa5053/status/1233304656715096064\\n']]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 2**: Load the same file into a `pandas.DataFrame`. What do you think it describes, and can you characterize\n",
    "it just from a glance? What would be some things we could investigate about this data?\n",
    ">\n",
    "> *Hint: To load a csv as a pandas dataframe you can `import pandas as pd` (if you do not have it installed, run `pip install pandas`\n",
    "in your terminal/command prompt) and load it using [`pd.read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T12:59:03.236724Z",
     "start_time": "2020-02-29T12:59:02.837492Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('output_got.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T12:59:24.587760Z",
     "start_time": "2020-02-29T12:59:24.575268Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>username</th>\n",
       "      <th>to</th>\n",
       "      <th>replies</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "      <th>text</th>\n",
       "      <th>geo</th>\n",
       "      <th>mentions</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>id</th>\n",
       "      <th>permalink</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-02-28 08:15:19</td>\n",
       "      <td>raebengala</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Per tranquillizzarci ci dicono che morti #coro...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>#coronavirus #anziani #COVID2019</td>\n",
       "      <td>1233304662671069184</td>\n",
       "      <td>https://twitter.com/raebengala/status/12333046...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-02-28 08:15:19</td>\n",
       "      <td>martin_lainez</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>El sevillano con coronavirus: «Mando un mensaj...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>@abcdesevilla @RamonRomanR</td>\n",
       "      <td>#vca</td>\n",
       "      <td>1233304662092152832</td>\n",
       "      <td>https://twitter.com/martin_lainez/status/12333...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-02-28 08:15:19</td>\n",
       "      <td>Davenews5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>#HDPROS Frontières/coronavirus : contrôler les...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>#HDPROS #Minable #Irresponsable</td>\n",
       "      <td>1233304660473151488</td>\n",
       "      <td>https://twitter.com/Davenews5/status/123330466...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-02-28 08:15:19</td>\n",
       "      <td>nnoeliaperez</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>No quiero ser alarmista pero VAMOS A MORIR ALF...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1233304660188041216</td>\n",
       "      <td>https://twitter.com/nnoeliaperez/status/123330...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-02-28 08:15:19</td>\n",
       "      <td>paulstpancras</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Why Trump is secretly terrified of coronavirus...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1233304659156180994</td>\n",
       "      <td>https://twitter.com/paulstpancras/status/12333...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  date       username   to  replies  retweets  favorites  \\\n",
       "0  2020-02-28 08:15:19     raebengala  NaN        0         0          0   \n",
       "1  2020-02-28 08:15:19  martin_lainez  NaN        0         0          0   \n",
       "2  2020-02-28 08:15:19      Davenews5  NaN        0         0          0   \n",
       "3  2020-02-28 08:15:19   nnoeliaperez  NaN        0         0          0   \n",
       "4  2020-02-28 08:15:19  paulstpancras  NaN        0         0          0   \n",
       "\n",
       "                                                text  geo  \\\n",
       "0  Per tranquillizzarci ci dicono che morti #coro...  NaN   \n",
       "1  El sevillano con coronavirus: «Mando un mensaj...  NaN   \n",
       "2  #HDPROS Frontières/coronavirus : contrôler les...  NaN   \n",
       "3  No quiero ser alarmista pero VAMOS A MORIR ALF...  NaN   \n",
       "4  Why Trump is secretly terrified of coronavirus...  NaN   \n",
       "\n",
       "                     mentions                          hashtags  \\\n",
       "0                         NaN  #coronavirus #anziani #COVID2019   \n",
       "1  @abcdesevilla @RamonRomanR                              #vca   \n",
       "2                         NaN   #HDPROS #Minable #Irresponsable   \n",
       "3                         NaN                               NaN   \n",
       "4                         NaN                               NaN   \n",
       "\n",
       "                    id                                          permalink  \n",
       "0  1233304662671069184  https://twitter.com/raebengala/status/12333046...  \n",
       "1  1233304662092152832  https://twitter.com/martin_lainez/status/12333...  \n",
       "2  1233304660473151488  https://twitter.com/Davenews5/status/123330466...  \n",
       "3  1233304660188041216  https://twitter.com/nnoeliaperez/status/123330...  \n",
       "4  1233304659156180994  https://twitter.com/paulstpancras/status/12333...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It describes tweets on the COVID-19 virus. One could\n",
    "* go ahead and analyse languages to get an idea of what demographics were\n",
    "talking about it. \n",
    "* Reconstruct a network of users based on replies and/or retweets (if more data\n",
    "were available)\n",
    "* Analyse the language in the tweets. Is it fear driven or do people seem more\n",
    "rational?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 3**: Are there missing values in the data? How does `pandas` handle these? Can we remedy missing values in\n",
    "any way?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are lots of missing values. If we inspect the raw data, we see that a \n",
    "missing value occurs every time we have two consecutive commas. Pandas represents\n",
    "these as 'NaN' values, i.e. \"Not a Number\" (as everything in a computer is\n",
    "ultimately a number)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ***If you are motivated***: Install [GetOldTweets3](https://pypi.org/project/GetOldTweets3/). Use it to download\n",
    "a dataset of tweets about Corona virus originating from Denmark (or in Danish). Print a handful of tweets and summarize the discourse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:07:38.900358Z",
     "start_time": "2020-02-29T13:07:33.941053Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading tweets...\n",
      "Saved 100\n",
      "Done. Output file generated \"coronadk.csv\".\n"
     ]
    }
   ],
   "source": [
    "!GetOldTweets3 --querysearch \"corona\" --near \"56.035014, 10.392624\" --within 150km --maxtweets 100 --output \"coronadk.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:08:07.282856Z",
     "start_time": "2020-02-29T13:08:07.275653Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"coronadk.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:09:03.556682Z",
     "start_time": "2020-02-29T13:09:03.549882Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vent fr, jeg skal vidst ud og hamstre corona til sommeren\n",
      "\n",
      "Jeg giver det 48 timer, så er hele det danske mediebilledes Corona-selvsving skiftet ud med FLYGTNINGENE KOMMER!!!« https://twitter.com/piphotosagency/status/1233410354023149569 …\n",
      "\n",
      "»Granddad… was that before the Corona virus apocalypse?«\n",
      "\n",
      "I forhold til hvordan man straks har sat et beredskab igang i lande som f.eks. England når det gælder Corona-virus. Og hvor man i Irland straks frarådede rejser til Italien efter udbruddet der. Så virker det danske myndigheds håndtering total amatør-agtig. #dkpol #covid2019\n",
      "\n",
      "Aldi i Aalborgs Vestby udnytter lige hypen og navnegenkendelsen til at introducere en nyhed i sortimentet. Ps. Bare rolig. Man kan forholdsvist risikofrit drikke Corona, blot man husker at bruge maske. pic.twitter.com/upMnL4LJTr\n",
      "\n",
      "har corona med skal du smage nilaash\n",
      "\n",
      "Corona virussen er en oplagt mulighed til at starte kønsdebatten\n",
      "\n",
      "Og hey: IKKE i orden, at SARS og Corona parrer sig.\n",
      "\n",
      "corona makes u stronger. so its a drug.. wtf\n",
      "\n",
      "Looks like a horror Movie Since Corona has spread in the World. Pigs in US and England has been tested and they found hepatitis which there are No vaccine against so bon appetit Carnivores Maybe Karma hits again https://twitter.com/jamieallingham1/status/1233542582992539649 …\n",
      "\n",
      "Op zich raar hoe de verslaggeving over het corona virus verloopt. In China geven ze het zo aan en kun je zien dat, Hubei 66337 besmettingen had, maar nu nog 34680. Bijna 29000!!! Zijn gezond verklaard. Succesverhalen zijn ook wel fijn zo nu en dan te horen! pic.twitter.com/254DCTz90Q\n",
      "\n",
      "This might be ‘interesting’ - corona virus has now arrived at the city where I live. Hospital employee caught virus from an Italian while attending a conference in Munich - then returned to work for 3 days (!) before testing positive. This is 3rd confirmed positive in Denmark. https://twitter.com/drnyheder/status/1233669871755378689 …\n",
      "\n",
      "38% af amerikanerne vil ikke købe Corona øl fordi....Coronavirus #dkpolhttps://twitter.com/cnn/status/1233406525491814400 …\n",
      "\n",
      "Ansat smittet med corona - Styrelsen for Patientsikkerhed kontakter alle der kan være i risikogruppen. Hvis du ikke bliver kontaktet er du ikke i risikogruppen. https://bit.ly/2T9URs0\n",
      "\n",
      "Closed the borders? I can book a flight from Europe to NY any second of the day. No deaths? Well, maybe that is because you don't keep track of it and don't distinguish between regular flu deaths and corona virus deaths.... Just saying....\n",
      "\n",
      "Please don't get Corona.\n",
      "\n",
      "Guess corona virus spread risk is limited to one source at these rallies since Trump supporters don't know about a world outside home state yet alone visited such unknown world....\n",
      "\n",
      "stiftendk: Aarhus Havn om corona-virus: - Vi regner med at blive påvirket snart https://ift.tt/3a5m5FU\n",
      "\n",
      "stiftendk: Århusianere higer efter mundbind mod corona: Apotek håber på snarlig levering https://ift.tt/2I2wE0p\n",
      "\n",
      "Dihadapanmu saya komunis, dihadapan tuhan saya muslim, dihadapan virus corona kita semua sama saja, asal jangan... *dilanjutkan chants suporter\n",
      "\n",
      "@politiken @btdk @jyllandsposten @EkstraBladet journalistik er ikke kun at berette om corona,men i virkeligheden om langt vigtigere ting som optager den danske befolkning, nemlig borgerforslaget om fjernelse af gensidig forsørgerpligt/ægtefælleafhængighed B34, er det for svært?\n",
      "\n",
      "Forlod Shanghai på grund af corona-virus: - Der var ingen grund til at blive, for vi kunne ingenting https://ift.tt/2wf4UDf (Dette er en plus-artikel) #sydfyn #fynsamtsavis\n",
      "\n",
      "Berkuasa dan Lalim, Seluruh Jajaran Pemerintah RI tertular Virus Corona *judul cerita Hidayah bulan ini\n",
      "\n",
      "Id rather get the corona virus and mb die than not being able to win money/play tournaments\n",
      "\n",
      "but why is no one talking about the fact that the media is hyping up the anxiety of corona and not actually mentioning the biggest problems that it gives like the economy #COVID2019\n",
      "\n",
      "a minha pack da outra conta ai gente cuidado com o corona pic.twitter.com/NbkdeQ5EEX\n",
      "\n",
      "my mom’s all worried about me getting corona in Spain not knowing that i’m already dead after #FALLINGMUSICVIDEO today like wtf\n",
      "\n",
      "If the corona was planned, the world has not come so far in its evolution. The good thing about this is that, with Blockchain the world needs to work together. https://twitter.com/looP_rM311_7211/status/1220519864928034817 …\n",
      "\n",
      "̶E̶b̶o̶l̶a̶ ̶ Corona (Na Na) ~ Rucka Rucka Ali feat DJ Not Nice https://youtu.be/RlxrccYb7XM via @YouTube is a banger\n",
      "\n",
      "If you have met any men you would know that at least that gender would buy more Corona just as a joke.\n",
      "\n",
      "Ye and she wont die of corona. Lets all calm down a bit Just a flu.\n",
      "\n",
      "Also specialist of viral disease like Corona etc pic.twitter.com/cVxxsMT4W3\n",
      "\n",
      "The corona virus is not a bacterial infection, but infact a virus as the name implicates. So antibiotics would be useless, sadly :)\n",
      "\n",
      "Just had the best laugh of the day watching @alexdowsett on instagram while bring in lockdown at #UAEtour. And now I can't wait for Corona League to happen.\n",
      "\n",
      "Fandt denne. #denergo #true #fact #corona #covid19pic.twitter.com/ZrqjgbhQ4S\n",
      "\n",
      "Roughly speaking, and from what we know, if all of your Twitter followers got seasonal flu, 8 of them would die. If they got the new Corona virus potentially 200 of them would...\n",
      "\n",
      "Mængden af grædende Alternativister har nu oversteget det globale antal corona-smittede. Fock, noget piv.\n",
      "\n",
      "Ok, fair nok. Det havde jeg faktisk misset, men det understreger semi min pointe. Lige nu slå medierne plat på corona, det er ok at tappe ind i, men mit æstetiske sans skriger. Det handler så om myrer, men køn er den forside ikke. Myrer og en murermaske? Hvor er koblingen?\n",
      "\n",
      "Jeg er heller ikke sådan super fan af den forside, men myrerne har jo ikke noget med Corona at gøre - de henviser til den ærøske myresovs.\n",
      "\n",
      "Sådan forholder vi os til corona-virus i Kalundborg Kommune https://ift.tt/3acfppj\n",
      "\n",
      "Nå, de turde alligevel vise en lille corona-virus til det rød-gule setup #xfactordk\n",
      "\n",
      "Sådan forholder vi os til corona-virus i Kalundborg Kommune https://ift.tt/398bWIh\n",
      "\n",
      "Nissens-produktion lagt ned af corona-virus: Vi er taknemmelige for ingen ansatte er smittet https://ift.tt/2uAAvP4\n",
      "\n",
      "The cure to the corona virus pic.twitter.com/IAOpRkJh7N\n",
      "\n",
      "jeg gider simpelthen ikke høre mere om det corona-virus! Jeg slår nu væk eller slukker når der er noget. Eneste vigtige for mig at vide er om der skulle være tilfælde i nærheden af Kalundborg og eller på de strækninger jeg bevæger mig. Men det kan man jo ikke vide alligevel\n",
      "\n",
      "Jeg har tre asiater med sars i kælderen og bytter dem gerne til tre med corona\n",
      "\n",
      "Hvis dine børn spørger om Corona-virus, så lyt her sammen med dem Coronavirus https://www.podbean.com/eu/pb-wg2k2-d40543#.XllXy-9rCb4.twitter …\n",
      "\n",
      "Stakkels Corona-øl... Har lige lanceret $40 mio. dyr kampagne - i et marked, hvor folk ikke tør bestille en Corona i offentligt rum pga. navnebrors-virus https://twitter.com/cnni/status/1233393636672446464 …\n",
      "\n",
      "Jeg har købt aktier i en verden delvist lammet af corona virus. Jeg er ikke smart. https://www.zetland.dk/historie/sowGXLEp-mOMVdnXp-29f10 …\n",
      "\n",
      "Følelsen når en af dem man er ret tætte med, skal til Skejby og testes for corona...\n",
      "\n",
      "Yup! Endnu én person har influenza. Det tager vi helt cool. Den der Corona Virus er jo ikke engang så slem som en ordentlig Mande-Influenza, som er værre end fødsler og Verdens undergang tilsammen! Så du interview med den \"ramte\" person fra TV2 i går? Han var klar til at feste\n",
      "\n",
      "Don't confuse stupid with scared and uninformed. It's a stupid act, to not buy Corona beer because of the virus. But most likely, the fear comes from all the media attention and people aren't willing to \"risk it\". Totally uninformed, but utterly human.\n",
      "\n",
      "De to sidste er dog dyre på nøgletal. Og hvis det her ikke bare er ren Corona-fear, men noget mere grundlæggende, så tror jeg at high growth er mere risikabelt end value. SAP trader eks på PE på 30. Adobe har PE på 30, vækster 25% om året og har højt ARR. Den ska købes. 2/2\n",
      "\n",
      "Tror du de går godt sammen med en Corona!?\n",
      "\n",
      "Racer-talent fra Hedensted i Corona-karantæne: Går glip af F2-prøver https://ift.tt/39buuHU\n",
      "\n",
      "it's comforting to know that if this music video doesn't kill me, the corona virus will\n",
      "\n",
      "Enig, normal influenza har dræbt sådan 10 gange så meget som corona har i 2020\n",
      "\n",
      "SHUT UP THE CORONA VIRUS IS IN AARHUS MY DADS BOSS HAS IT BYE\n",
      "\n",
      "Emotions play a role here,all we need it for you to suspend all flights from China until this Corona Virus issue is sorted,bring our Kenyans back home, simple and clear!!!!!!\n",
      "\n",
      "Jeg er ikke specielt bange for den, som det er nu. Men smittekæder er så dumt og går ud over små børn, der ikke selv har valgt det, og vi ved ikke ret meget om corona endnu.\n",
      "\n",
      "Jeg forstår simpelthen ikke hvorfor hele verden er så bange for den corona virus?\n",
      "\n",
      "Only 20 out of 53 european countries got corona virus cases. and of those only 5 got more than 10. YOu can hardly call 33 out of 53 \"a few\"\n",
      "\n",
      "Aarhus Havn om corona-virus: - Vi regner med at blive påvirket snart https://ift.tt/3a5m5FU\n",
      "\n",
      "Elle hvornår maskeforbuddet skal hæves. #corona\n",
      "\n",
      "Surely not because of the corona virus?\n",
      "\n",
      "Århusianere higer efter mundbind mod corona: Apotek håber på snarlig levering https://ift.tt/2I2wE0p\n",
      "\n",
      "Ja, så han kan smitte dig, så du bliver one shotted af Corona\n",
      "\n",
      "Jeg troede de blev forbudt fordi det er fascistisk at give hånd? Corona-frygt: Newcastle forbyder håndtryk https://www.bold.dk/fodbold/nyheder/corona-frygt-newcastle-forbyder-haandtryk/ …\n",
      "\n",
      "Good thing you left the EU..now the corona virus cant reach you\n",
      "\n",
      "Hvis du ikke har opdaget det, så er planen fra de danske eksperter at lade stå til og håbe på at Corona dør ud med forårets komme En udemærket strategi hvis det virker... og det er et stort hvis DR lavede en hel times udsendelse om at man skal rejse og gøre som man plejer\n",
      "\n",
      "#Corona #Coronavid19 #coronavirusus Hvis nu der dør en milliard eller to af C19, så kunne vi sikkert nå klimamålene\n",
      "\n",
      "Corona kontra influenza .. Jeg er helt enig i at medierne malker det og skræmmer folk unødigt.. Men for at svare på det du skriver - Levetiden for influenza kontra Corona er nat og dag.. - Corona har eksisteret i en måned? Og tallene fordobler sig hastigt..\n",
      "\n",
      "Som du selv har pointeret, ville den bedste plan da være IKKE at få Corona ind i DK Nu er det så at begrænse skaden, Diamond princess lærte os at hjemme isolation er ineffektiv, så \"gør som vi altid gør\"-princippet er ubrugeligt Eksperternes \"plan\" er at håbe på tidligt forår\n",
      "\n",
      "Corona: Blodbank beder donorer om at blive hjemme http://dlvr.it/RQvt9V pic.twitter.com/LQsKFH7IiH\n",
      "\n",
      "Når man bor i Mørkøv, og man i sit nyhedsfeed ser den her overskrift fra det lokale mediehus #corona https://sn.dk/Sport/Moerkoev-er-isoleret-grundet-coronafrygt-ved-bane-VM/artikel/918570 …\n",
      "\n",
      "Jeg er mere spændt på hvad de gør på Corona Camping?\n",
      "\n",
      "Corona har spredt sig til de finansielle markeder. Jeg tror nu de fleste skal slå koldt vand i blodet og holde fast i deres langsigtede strategi. Det er tid i markedet og ikke market timing, der betyder noget for dit afkast. #aktier #investering #corona https://www.youtube.com/watch?v=bA_60FjFxfA …\n",
      "\n",
      "Hvor er jeg træt af at læse om folk der gør grin med Corona virus og ikke forstår omfanget. Nej det er ikke en sundhedskrise, det er en potentiel global økonomisk krise.\n",
      "\n",
      "https://nyheder.tv2.dk/2020-02-28-hund-testet-svagt-positiv-for-coronavirus … A dog in China is suspected of having a weak corona virus and imo bad and Good. How many PETs must die now by the hands of owners in panic . Good it could end Dog Cat Meat Trade forever if the dog is infected\n",
      "\n",
      "Corona-effekt: Mundbind er totalt udsolgt på apotekerne i Horsens https://ift.tt/3a7qmZf\n",
      "\n",
      "#sundpol Hoste er kardinalsymptomet på corona. pic.twitter.com/pY9lCz34uP\n",
      "\n",
      "Feliz viernes Romi! Aquí en Dinamarca hay que bailar para entrar en. Aloe hoy Todo brilla por la helada de anoche. Hoy anunciaron que los primeros conciertos de @BTS_twt en Soul están cancelados a causa del virus corona #TrasnocheLos40 Feliz finde!\n",
      "\n",
      "Vi skal “minimere muligheden” for corona virus på arbejdet. MULIGHEDEN. ikke risikoen, men muligheden. Gg\n",
      "\n",
      "The good thing about struggling with crippling social anxiety... I'll probably get through this Corona virus pandemic, without a hitch #coronavirus #itscoronatime\n",
      "\n",
      "Hvilken myndighed ringer man til, når man kommer ind i depotet og opdager, at alle ølkasserne er fyldt med Corona?\n",
      "\n",
      "Hi Corona v he pic.twitter.com/fkZb04DQEF\n",
      "\n",
      "Hvis Ståle har lidt humor, så køber han sejrsøl til #fck spillerne i aften og fortæller medierne, at mindst halvdelen af truppen har fået corona. #CELFCK #eldk\n",
      "\n",
      "Corona dit corona dat ughh\n",
      "\n",
      "2006 was my first year at Boom. This year will be my 3 time. I pray for no more corona!\n",
      "\n",
      "The news. 1 guy comes home from skiing holidays with Corona symptoms. Pharmacies all over Denmark has sold out facemasks and disinfectants. People react so fast to this shit with panic. News already talking about fear of going to work spaces. What a selfish man to go skiing lol\n",
      "\n",
      "Corona-chan adalah teman yang akan menginfeksi siapapun mereka yang berani menggangguku. -- Tsubasa Ozora dalam bukunya yang berjudul: Nankatsu's Secret\n",
      "\n",
      "Løbet er blevet aflyst på grund af corona-virussen.\n",
      "\n",
      "hi hi i just got home from work to the news that corona is in my country now eyem\n",
      "\n",
      "Jep nu når hysteriet nye højder nu har vi både coronavirus og klimaet at gå i fuldstændig selvsving over, lad os lukke grænserne hermetisk, så tror nogen at vi klare klima og nu også corona, med lidt får vi stoppet så meget andet også. Jow jow\n",
      "\n",
      "#dkmedier #dkpol er jeg den eneste der syntes TV2 er gået fuldstændig i selvsving over Corona virus\n",
      "\n",
      "Vel kun et spørgsmål om tid før Janni & Co breaker i Corona terror tilstand........\n",
      "\n",
      "I am soon going to Mexico. Isn’t that the country of origin of Corona?\n",
      "\n",
      "@mbCARMAC You're doing a wonderful job at Katowice. We can't control everything, especially not something as big as the Corona virus. I have no doubt you all are doing your best to deliver quality Counter-Strike entertainment for the fans despite the circumstances!\n",
      "\n",
      "Super weird watching Inter play Europa League 1/8 finals at home in front of ZERO spectators due to corona virus outbreak in Italy pic.twitter.com/CNP0HjGRAU\n",
      "\n",
      "LOL hier is net de aller eerste Corona patiënt gevonden. Heerlijk....gelijk het hele land in rep en roer. En de patiënt zelf is niet ziek ziek. Maar zoiets tussenin.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for text in df['text']:\n",
    "    print(text, end=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My summary: Lots of tweets in other languages (primarily english) talking about\n",
    "the situaion in DK (atm 3 infected). Most of these are heavy on fear. At the same\n",
    "time lots of tweets in danish are criticizing the Danish media for being hysterical,\n",
    "and drawing attention away from more important matters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Non-tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 4** In Python, JSON is rendered as a `dict`-type object. `dict`s are key-value stores, much like \n",
    "dictionaries that map words between e.g. Danish and English: input a word in Danish (the key) and get the word\n",
    "in English (the value). In Python this operation is achieved by \"keying\" into a dictionary with code like\n",
    "`my_dict[the_key]`. JSON objects are, however, a step more complicated. How? Because values can themselves be\n",
    "dictionaries (or lists), thus nesting the data in a tree structure. In this exercise, you should \"key\" into `my_json_obj`\n",
    "below to access the list of useless cats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:14:09.013780Z",
     "start_time": "2020-02-29T13:14:09.010420Z"
    }
   },
   "outputs": [],
   "source": [
    "my_json_obj = {\n",
    "    'cats': {\n",
    "        'awesome': ['Missy'],\n",
    "        'useless': ['Kim', 'Frank', 'Sandy']\n",
    "    },\n",
    "    'dogs': {\n",
    "        'awesome': ['Finn', 'Dolores', 'Fido', 'Casper'],\n",
    "        'useless': []\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:14:16.094688Z",
     "start_time": "2020-02-29T13:14:16.090668Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Kim', 'Frank', 'Sandy']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_json_obj['cats']['useless']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 5**: Run the code snippet below to download frontpage of posts from the *[r/coronavirus](https://www.reddit.com/r/Coronavirus/)* subreddit in\n",
    "[json format](https://www.reddit.com/r/coronavirus.json). With pen and paper (or another illustration tool) draw a sketch that outlines the data\n",
    "as a tree (similar to example in lecture)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:14:20.829490Z",
     "start_time": "2020-02-29T13:14:20.017659Z"
    }
   },
   "outputs": [],
   "source": [
    "import requests as rq\n",
    "data = rq.get(\n",
    "    \"https://www.reddit.com/r/coronavirus.json\",    # link to the data\n",
    "    headers={'User-agent': 'digital_methods_2020'}  # a user agent that tells reddit who we are (good measure)\n",
    ").json()                                            # render the response from reddits server as JSON data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to my sketch](https://www.dropbox.com/s/2et2bs5d1xu07ud/ex5_sketch.png?dl=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 6** Similar to how you returned the list of useless cats in Ex. 4, now \"key\" into `data`\n",
    "to access a value that informs you this data is from the 'Coronavirus' subreddit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-29T13:34:41.441984Z",
     "start_time": "2020-02-29T13:34:41.437774Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coronavirus'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['data']['children'][0]['data']['subreddit']"
   ]
  }
 ],
 "metadata": {
  "display_name": "Python 3",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}