Start_here.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GDELT Demo project showcase\n",
    "\n",
    "This is the simple presentation of my findings and a bit about the skills involved in getting there. The [README.md](https://github.com/reed9999/gdelt-demo/blob/master/README.md) file is helpful (along with its children) for:\n",
    "\n",
    "* Setup instructions\n",
    "* A summary of skills demonstrated in this project.\n",
    "\n",
    "At present all findings are exploratory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Findings and visualizations\n",
    "\n",
    "This project is still in early stages, but I'll post small scale findings incrementally. So far I'm just running a few simple descriptives and regressions to get a feel for various aspects of the data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: To allow for execution of these demos without placing multiple gigabytes in the repo, I am engineering them to run with sample data when bigger datasets (either my local copy, still a sample but bigger, or on the full dataset) are unavailable. Thus specific metrics may not agree with what I report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "import analysis.goldsteinscale_avgtone as ga\n",
    "importlib.reload(ga)\n",
    "# This seemed to make things very slow but perhaps it's just browser slowness.\n",
    "# %reload_ext autoreload\n",
    "# %autoreload 1\n",
    "\n",
    "regr = ga.GARegression()\n",
    "regr.prepare_data()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis (EDA)\n",
    "Truthfully I haven't spent enough time here (especially with visualizations) since I rushed to get to other topics. When I get my classifiers to a \"minimum viable\" status I will come back to this, because I don't want to contribute to the undervaluing of EDA! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        goldsteinscale     avgtone\n",
      "count       710.000000  710.000000\n",
      "unique             NaN         NaN\n",
      "top                NaN         NaN\n",
      "freq               NaN         NaN\n",
      "mean          0.300282   -0.385485\n",
      "std           4.642020    4.965812\n",
      "min         -10.000000  -22.206776\n",
      "25%          -2.000000   -3.936718\n",
      "50%           1.000000   -0.201449\n",
      "75%           3.000000    3.589744\n",
      "max          10.000000   10.074627\n",
      "\n",
      "For column nummentions:\n",
      "mean    18.239437\n",
      "std     91.829665\n",
      "Name: nummentions, dtype: float64\n",
      "\n",
      "For column numsources:\n",
      "mean     3.412676\n",
      "std     17.513166\n",
      "Name: numsources, dtype: float64\n",
      "\n",
      "For column numarticles:\n",
      "mean    17.994366\n",
      "std     91.046137\n",
      "Name: numarticles, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "importlib.reload(ga)\n",
    "regr.report_descriptives()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### International stability and media tone\n",
    "\n",
    "Goldstein is a measure of propensity of each *type* of event to promote stability; avgtone is the average tone across all items (mostly media stories, I think) referencing the event. My intuition is that they should be positively correlated, because the media reflects the public's interest in stability. \n",
    "\n",
    "#### Descriptives and simple plots\n",
    "A good place to start, before I do inferential stats, is to get to know the data. I jumped ahead a bit but now am coming back to this.\n",
    "\n",
    "**COMING NEXT**: Means, stdev, plots -- maybe box plots with interquartile range, maybe histograms "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regression analysis\n",
    "\n",
    "The most interesting story I've found so far is that the tone of coverage trends downwards for time while controlling for the Goldstein score of each kind of event:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "*** REGRESSION RESULTS ***\n",
      "Dependent variable is avgtone (average tone of documents)\n",
      "Coefficients on the regression:\n",
      "    fractiondate: -0.22944612354667906\n",
      "    goldsteinscale: 0.21230102615077256\n",
      "MSRE: 10.733810479252504\n",
      "r2: 0.5180875389014963\n"
     ]
    }
   ],
   "source": [
    "regr.go(plot=False)\n",
    "regr.print_output()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Results from running on my local \"mini\" database (different results from the sample data in this repo):\n",
    "\n",
    "```\n",
    "Dependent variable is avgtone (average tone of documents)\n",
    "\n",
    "Coefficients on the regression:\n",
    "\n",
    "    fractiondate: -0.20230581030052106\n",
    "    goldsteinscale: 0.07914705296303393\n",
    "MSRE: 7.249942180177441\n",
    "r2: 0.46995787329613636\n",
    "```\n",
    "\n",
    "#### Interpretation\n",
    "Each year is associated with a -0.20 change in average tone, holding constant the theoretical propensity of the events to promote stability (i.e. holding constant the Goldstein score). Each 1.0 point increase in the Goldstein store is associated with a +0.08 change, holding constant the date.\n",
    "\n",
    "In isolation, all we can tell from the MSRE is that a mean absolute value of the std error would be its square root. In principle avgtone can run from -100 to 100, but in practice it seems to usually be between 0 and 10. So, missing estimates by something around 2 or 3 seems not-great; however, it's a more useful metric to compare to other analyses than in isolation. The r-squared looks pretty nice for just my first attempt at specifying a regression with two independent variables, but I should be cautious about this metric as I run more regressions and risk overfitting.\n",
    "\n",
    "I have not yet run this on all the data (e.g., with Spark on AWS Elastic MapReduce), just a subset of six data files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Research questions\n",
    "To get interesting answers, I need good questions. This is what makes this project more than just an extended SQL or PySpark tutorial. In this kind of research there's going to be some iteration in RQs, because things you think *a priori* might be interesting turn out not to be, and vice versa. So this is largely a running list of ideas, .\n",
    "\n",
    "1. **Media tone and stability** - The Rosenstein score is a measure of how each kind of action promotes or erodes stability. The tone score is something about the tone of the media coverage of the event. See above.\n",
    "\n",
    "1. **Actor affinity by dyad** - Do some actor dyads consistently produce higher scores (Rosenstein or media tone) than others? I'd think this would be a trivial \"Yes\" because relationships between allies should produce more positive news than those between adversaries. So this is exploring the obvious, but a good sanity check to make sure I'm understanding the nature of this data.\n",
    "\n",
    "  1. **Asymmetrical relationships** - Just brainstorming. If X threatens Y more than Y threatens X that's also significant and interesting.\n",
    "  \n",
    "1. **Longitudinal time series questions** - This dataset is so rich that it could support a lot of this.\n",
    "\n",
    "1. **Clustering countries or other actors** - If I want to polish up those *k*-means clustering skills, there's probably all kinds of variables I could derive (\"number of threats made\", \"number of overtures to negotiation\", \"general tone of media coverage\", etc.).\n",
    "\n",
    "1. **Network graph analysis** - I don't actually know much about this topic, but clearly this data would support a lot of inferences about how actors build their networks. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A note on data engineering and AWS Elastic MapReduce\n",
    "It's a bit of a gray area whether learning how to move files around to AWS EMR, get Spark to run my Python scripts without hanging, etc. is properly part of data science or whether it's actually data engineering. I find it valuable to learn these skills even though they don't lead directly to findings above. See the README.md for the skills developed in this subset of the project.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gdelt-demo",
   "language": "python",
   "name": "gdelt-demo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}