recap_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Going deeper with Tensorflow\n",
    "\n",
    "In this seminar, we're going to play with [Tensorflow](https://www.tensorflow.org/) and see how it helps us build deep learning models.\n",
    "\n",
    "If you're running this notebook outside the course environment, you'll need to install tensorflow:\n",
    "* `pip install tensorflow` should install cpu-only TF on Linux & Mac OS\n",
    "* If you want GPU support from offset, see [TF install page](https://www.tensorflow.org/install/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "gpu_options = tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=0.1)\n",
    "s = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warming up\n",
    "For starters, let's implement a python function that computes the sum of squares of numbers from 0 to N-1.\n",
    "* Use numpy or python\n",
    "* An array of numbers 0 to N - numpy.arange(N)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "def sum_squares(N):\n",
    "    return ((np.arange(0, N))**2).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 381 ms, sys: 695 ms, total: 1.08 s\n",
      "Wall time: 2.19 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "662921401752298880"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "sum_squares(10**8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow teaser\n",
    "\n",
    "Doing the very same thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#I gonna be your function parameter\n",
    "N = tf.placeholder('int64', name=\"input_to_your_function\")\n",
    "\n",
    "#i am a recipe on how to produce sum of squares of arange of N given N\n",
    "result = tf.reduce_sum((tf.range(N)**2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "662921401752298880\n",
      "CPU times: user 545 ms, sys: 346 ms, total: 890 ms\n",
      "Wall time: 1.81 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#example of computing the same as sum_squares\n",
    "print(result.eval({N:10**8}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How does it work?\n",
    "1. define placeholders where you'll send inputs;\n",
    "2. make symbolic graph: a recipe for mathematical transformation of those placeholders;\n",
    "3. compute outputs of your graph with particular values for each placeholder\n",
    "  * output.eval({placeholder:value}) \n",
    "  * s.run(output, {placeholder:value})\n",
    "\n",
    "* So far there are two main entities: \"placeholder\" and \"transformation\"\n",
    "* Both can be numbers, vectors, matrices, tensors, etc.\n",
    "* Both can be int32/64, floats of booleans (uint8) of various size.\n",
    "\n",
    "* You can define new transformations as an arbitrary operation on placeholders and other transformations\n",
    " * tf.reduce_sum(tf.arange(N)\\**2) are 3 sequential transformations of placeholder N\n",
    " * There's a tensorflow symbolic version for every numpy function\n",
    "   * `a+b, a/b, a**b, ...` behave just like in numpy\n",
    "   * np.mean -> tf.reduce_mean\n",
    "   * np.arange -> tf.range\n",
    "   * np.cumsum -> tf.cumsum\n",
    "   * If if you can't find the op you need, see the [docs](https://www.tensorflow.org/api_docs/python).\n",
    " \n",
    " \n",
    "Still confused? We gonna fix that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Default placeholder that can be arbitrary float32 scalar, vector, matrix, etc.\n",
    "arbitrary_input = tf.placeholder('float32')\n",
    "\n",
    "#Input vector of arbitrary length\n",
    "input_vector = tf.placeholder('float32',shape=(None,))\n",
    "\n",
    "#Input vector that _must_ have 10 elements and integer type\n",
    "fixed_vector = tf.placeholder('int32',shape=(10,))\n",
    "\n",
    "#Matrix of arbitrary n_rows and 15 columns (e.g. a minibatch your data table)\n",
    "input_matrix = tf.placeholder('float32',shape=(None,15))\n",
    "\n",
    "#You can generally use None whenever you don't need a specific shape\n",
    "input1 = tf.placeholder('float64',shape=(None,100,None))\n",
    "input2 = tf.placeholder('int32',shape=(None,None,3,224,224))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#elementwise multiplication\n",
    "double_the_vector = input_vector*2\n",
    "\n",
    "#elementwise cosine\n",
    "elementwise_cosine = tf.cos(input_vector)\n",
    "\n",
    "#difference between squared vector and vector itself\n",
    "vector_squares = input_vector**2 - input_vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Practice time: create two vectors of type float32\n",
    "my_vector = tf.placeholder('float32', shape=(None,))\n",
    "my_vector2 = tf.placeholder('float32', shape=(None,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Write a transformation(recipe):\n",
    "#(vec1)*(vec2) / (sin(vec1) +1)\n",
    "my_transformation = (my_vector*my_vector2)/(tf.sin(my_vector) + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"truediv:0\", shape=(?,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(my_transformation)\n",
    "#it's okay, it's a symbolic graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 1. 2. 3. 4.]\n",
      "[4. 3. 2. 1. 0.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0.       , 1.6291324, 2.0950115, 2.6289961, 0.       ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\n",
    "dummy = np.arange(5).astype('float32')\n",
    "print(dummy)\n",
    "print(dummy[::-1])\n",
    "my_transformation.eval({my_vector:dummy,my_vector2:dummy[::-1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing graphs\n",
    "\n",
    "It's often useful to visualize the computation graph when debugging or optimizing. \n",
    "Interactive visualization is where tensorflow really shines as compared to other frameworks. \n",
    "\n",
    "There's a special instrument for that, called Tensorboard. You can launch it from console:\n",
    "\n",
    "```tensorboard --logdir=/tmp/tboard --port=7007```\n",
    "\n",
    "If you're pathologically afraid of consoles, try this:\n",
    "\n",
    "```os.system(\"tensorboard --logdir=/tmp/tboard --port=7007 &\"```\n",
    "\n",
    "_(but don't tell anyone we taught you that)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Port: 7000\n"
     ]
    }
   ],
   "source": [
    "# launch tensorflow the ugly way, uncomment if you need that\n",
    "import os\n",
    "port = 6000 + os.getuid()\n",
    "print(\"Port: %d\" % port)\n",
    "#!killall tensorboard\n",
    "os.system(\"tensorboard --logdir=./tboard --port=%d &\" % port)\n",
    "\n",
    "# show graph to tensorboard\n",
    "writer = tf.summary.FileWriter(\"./tboard\", graph=tf.get_default_graph())\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One basic functionality of tensorboard is drawing graphs. Once you've run the cell above, go to `localhost:7007` in your browser and switch to _graphs_ tab in the topbar. \n",
    "\n",
    "Here's what you should see:\n",
    "\n",
    "<img src=\"https://s12.postimg.org/a374bmffx/tensorboard.png\" width=480>\n",
    "\n",
    "Tensorboard also allows you to draw graphs (e.g. learning curves), record images & audio ~~and play flash games~~. This is useful when monitoring learning progress and catching some training issues.\n",
    "\n",
    "One researcher said:\n",
    "```\n",
    "If you spent last four hours of your worktime watching as your algorithm prints numbers and draws figures, you're probably doing deep learning wrong.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can read more on tensorboard usage [here](https://www.tensorflow.org/get_started/graph_viz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do It Yourself\n",
    "\n",
    "__[2 points max]__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Quest #1 - implement a function that computes a mean squared error of two input vectors\n",
    "# Your function has to take 2 vectors and return a single number\n",
    "\n",
    "v1 = tf.placeholder('float32', shape=(None,))\n",
    "v2 = tf.placeholder('float32', shape=(None,))\n",
    "\n",
    "mse =tf.reduce_sum((v1 - v2)**2)/tf.cast(tf.shape(v1)[0], tf.float32)\n",
    "\n",
    "compute_mse = lambda vector1, vector2: mse.eval({v1:vector1, v2:vector2})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Tests\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "for n in [1,5,10,10**3]:\n",
    "    \n",
    "    elems = [np.arange(n),np.arange(n,0,-1), np.zeros(n),\n",
    "             np.ones(n),np.random.random(n),np.random.randint(100,size=n)]\n",
    "    \n",
    "    for el in elems:\n",
    "        for el_2 in elems:\n",
    "            true_mse = np.array(mean_squared_error(el,el_2))\n",
    "            my_mse = compute_mse(el,el_2)\n",
    "            if not np.allclose(true_mse,my_mse):\n",
    "                print('Wrong result:')\n",
    "                print('mse(%s,%s)' % (el,el_2))\n",
    "                print(\"should be: %f, but your function returned %f\" % (true_mse,my_mse))\n",
    "                raise ValueError(\"Что-то не так\")\n",
    "\n",
    "print(\"All tests passed\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# variables\n",
    "\n",
    "The inputs and transformations have no value outside function call. This isn't too comfortable if you want your model to have parameters (e.g. network weights) that are always present, but can change their value over time.\n",
    "\n",
    "Tensorflow solves this with `tf.Variable` objects.\n",
    "* You can assign variable a value at any time in your graph\n",
    "* Unlike placeholders, there's no need to explicitly pass values to variables when `s.run(...)`-ing\n",
    "* You can use variables the same way you use transformations \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "#creating shared variable\n",
    "shared_vector_1 = tf.Variable(initial_value=np.ones(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial value [1. 1. 1. 1. 1.]\n"
     ]
    }
   ],
   "source": [
    "#initialize variable(s) with initial values\n",
    "s.run(tf.global_variables_initializer())\n",
    "\n",
    "#evaluating shared variable (outside symbolic graph)\n",
    "print(\"initial value\", s.run(shared_vector_1))\n",
    "\n",
    "# within symbolic graph you use them just as any other inout or transformation, not \"get value\" needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new value [0. 1. 2. 3. 4.]\n"
     ]
    }
   ],
   "source": [
    "#setting new value\n",
    "s.run(shared_vector_1.assign(np.arange(5)))\n",
    "\n",
    "#getting that new value\n",
    "print(\"new value\", s.run(shared_vector_1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tf.gradients - why graphs matter\n",
    "* Tensorflow can compute derivatives and gradients automatically using the computation graph\n",
    "* Gradients are computed as a product of elementary derivatives via chain rule:\n",
    "\n",
    "$$ {\\partial f(g(x)) \\over \\partial x} = {\\partial f(g(x)) \\over \\partial g(x)}\\cdot {\\partial g(x) \\over \\partial x} $$\n",
    "\n",
    "It can get you the derivative of any graph as long as it knows how to differentiate elementary operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_scalar = tf.placeholder('float32')\n",
    "\n",
    "scalar_squared = my_scalar**2\n",
    "\n",
    "#a derivative of scalar_squared by my_scalar\n",
    "derivative = tf.gradients(scalar_squared, my_scalar)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd4FFXbx/HvSe+BQKgh9F4CIaGD\ngCCoCIIiRUFUROARCyoqWB5FRMWKWABRRBCpFlQE6U1KAoTeQg01BNLrZs/7x0Re9KGE7CaT3dyf\n6+KSTXbP3CPwYzhz5j5Ka40QQgjn4WJ2AUIIIexLgl0IIZyMBLsQQjgZCXYhhHAyEuxCCOFkJNiF\nEMLJSLALIYSTkWAXQggnI8EuhBBOxs2Mg5YtW1ZXq1bNjEMLIYTDio6Ovqi1Dr7Z+0wJ9mrVqhEV\nFWXGoYUQwmEppU7k530yFSOEEE5Ggl0IIZyMBLsQQjgZCXYhhHAyEuxCCOFkJNiFEMLJSLALIYST\ncahgjzp+iS/WxJpdhhBC3LLMnFz++8teLqRkFvqxHCrYf9t9lnf/OMDWY5fMLkUIIW7JRysOMXPT\ncQ6fTy30YzlUsL/QrS5Vgrx5cdEuMnNyzS5HCCHyJeZUItPXHaV/ZBXa1ipb6MdzqGD38XDjnT5N\nOHYxjY/+PGR2OUIIcVPZFitjFu6inL8XY++uXyTHdKhgB2hbqywDWoQyff1RYk4lml2OEELc0Ger\nj3DwfAoTejciwMu9SI7pcMEO8PJd9Sgf4MULC2PIssiUjBCieNp/NpnPVh/h3qaVuL1++SI7rkMG\ne4CXO2/3bsyh86l8tlpWyQghih9LrjEFU8rHndfvaVikx3bIYAfoVK8cfZpV5vPVR9h3JtnscoQQ\n4h+mrz/G7tNJvNGzEaV9PYr02A4b7ACv3dOAUj4ejFkUgyXXanY5QggBQGx8Kh+tOET3hhW4q3GF\nIj++Qwd7KR8P3rq3IXtOJzNt/VGzyxFCCHKtmjELd+Ht7sqb9zZEKVXkNTh0sAN0b1SRuxpX4OM/\nD3P4fIrZ5QghSriZm44TfeIyr/VoQDl/L1NqsEuwK6WeVUrtVUrtUUrNVUoV6dm82asRfl5uPDt/\nJzkyJSOEMMmRC6m898cBOtcrR5/wyqbVYXOwK6UqA08BEVrrRoAr0N/WcW9FWT9P3u7diD2nk5my\n6khRHloIIQBjFcxz83fi7eHKO30amzIF8zd7TcW4Ad5KKTfABzhjp3HzrXujivRuVpkpq4+wK04e\nXBJCFK3P18QSE5fEW/c2olyAOVMwf7M52LXWp4H3gZPAWSBJa7383+9TSg1TSkUppaLi4+NtPew1\n/bdnQ4L9PBk9P0Z6yQghisye00lMXnmYe8Iq0aNJJbPLsctUTGmgF1AdqAT4KqUe+vf7tNbTtNYR\nWuuI4OBgWw97TYHe7rx3fxOOXEjl/WUHC+UYQghxtcycXEbP30mQrwfjexXtg0jXY4+pmC7AMa11\nvNY6B1gMtLHDuAXSoU4wD7UKZcbGY2w+mmBWGUKIEuKjPw9x6Hwq797XhFI+Rfsg0vXYI9hPAq2U\nUj7KuFtwO7DfDuMW2Ni76hMa5MPzC2JIzbKYWYoQwoltO36JaeuPMqBFFTrVK2d2OVfYY459C7AQ\n2A7szhtzmq3j2sLHw40P+oZxOjGDCb/tM7MUIYSTSsuy8Nz8GEJKezPu7gZml/MPdlkVo7V+XWtd\nT2vdSGs9SGudZY9xbRFRLYhhHWowd+spVh04b3Y5QggnM+H3/Zy6nM7794fh5+lmdjn/4PBPnt7I\n6K51qFfBnxcW7CI+xfS/a4QQTmL53nN8v+Ukj7evQcsaZcwu5384dbB7urkyeUAzUrMsvLAwBq21\n2SUJIRzc+eRMXly0i4aVAnj+jrpml3NNTh3sAHXK+zPu7vqsORjPzE3HzS5HCOHArFbNc/NjyMjJ\n5ZP+zfBwK54RWjyrsrNBrapye71yTFx6gAPnpHe7EKJgZmw4xoYjF3mtR0NqlfMzu5zrKhHBrpTi\n3fubEODlzlNzd8hTqUKIW7bndBLvLTtAt4blGdCiitnl3FCJCHYwGoV98EAYh86nMvF3U5fZCyEc\nTEZ2Lk//sIMgXw/e6dPE1AZf+VFigh3gtjrBPNq2Ot/+dUKWQAoh8m38b/uIjU/jwweaFvk2dwVR\nooIdYEz3urIEUgiRb38vbRzWoQZta5U1u5x8KXHB7uXuyqd5SyBHz9+J1SpLIIUQ13YmMaPYL228\nlhIX7AC1y/vz354NWX/4Il+sjTW7HCFEMZSTa2XU3B1kW6x8OqD4Lm28Fsep1M76R1ahV9NKfLD8\nIFukC6QQ4l8+WH6I6BOXebtPY2oEF9+ljddSYoNdKcWE3o2pVsaXp37YQUKqzLcLIQyrD1zgy7Wx\nDGgRSq+m5u1dWlAlNtgB/DzdmDIwnMvpOTw7P0bm24UQnE3KYPT8ndSr4M/r9xSvro35VaKDHaBB\npQBev6cB6w7Fy3y7ECWcJdfKqO+NefXPHgzHy93V7JIKpMQHO8DAFqHcE2bMt289dsnscoQQJvng\nz0NE5c2r13SwefWrSbBjzLe/3bsRoUE+PDVX5tuFKInWHLzAF2tiGdCiikPOq19Ngj2Pv5c7UwaG\ncyk9m2fm7SRX5tuFKDHOJGYwen5M3rx68diQ2hZ2CXalVCml1EKl1AGl1H6lVGt7jFvUGlUO5L/3\nGOvbP15xyOxyhBBFIDMnlxGzox1+Xv1q9trP6RPgD631/UopD8DHTuMWuQEtqrDj5GU+XXWEJiGl\n6NqgvNklCSEK0RtL9hITl8SXDzV36Hn1q9l8xa6UCgQ6ADMAtNbZWutEW8c1i1KK8fc2onHlQEbP\n28mxi2lmlySEKCTztp1k7tZTjOxYk+6NKhT+AdOLZnGGPaZiqgPxwDdKqR1Kqa+UUr52GNc0Xu6u\nfPFQOG6uiuHfRZOebTG7JCGEne2KS+TVn/fSvnZZnivsPjDpl2D5q/BhAzjxV+EeC/sEuxsQDnyh\ntW4GpAEv/ftNSqlhSqkopVRUfHy8HQ5buEJK+zB5QDMOX0jhxUW7Zb9UIZxIQmoWw7+LJtjPk0/6\nN8PVpZD6q2cmw5p34OMmsOlTqH8PBFQsnGNdxR7BHgfEaa235L1eiBH0/6C1nqa1jtBaRwQHB9vh\nsIWvfe1gnrujLktizjBjwzGzyxFC2IEl18pTP+zgYlo2Xz7UnKDC6K+enQ4bP4FPwmDNRKhxG4zY\nBPdNh9LV7H+8f7H55qnW+pxS6pRSqq7W+iBwO7DP9tKKh5EdaxJzKpGJSw/QqHIgrWqUMbskIYQN\n3l9+iI1HEnjvviY0Dgm07+CWLNg+C9ZNgtTzUPN26PwKVP6fa91CZa917KOAOUqpXUBT4G07jWs6\npRQfPBBG1SAfnvx+O6cTM8wuSQhRQL/tOsuXa2MZ2DKUByLtuG9prgW2fwefRsDvz0NQDXhkKQxa\nXOShDnYKdq31zrxpliZa63u11pftMW5x4e/lzrTBzcnKsfL4t1FyM1UIB7TndBLPLdhJeGgp+zX3\nslphzyL4vCX88iT4BMFDi4xQr9rGPscoAHnyNJ9qlfNn8sBmHDiXzHPSCVIIh3IhOZPHZ0UR5OPB\n1EEReLrZ+BCS1nBwKUztAAsfBRd36Dcbhq2BWl3A5M2uJdhvQae65Rh7V32W7jnHxysPm12OECIf\nMnNyGfZdNInpOUx/OIJgf0/bBjy6Br7qAnP7Q3Yq9JkOIzYaK15MDvS/2evJ0xLjsXbVOXguhckr\nD1OnvB89mlQyuyQhxHVorRm7eDc7TyXy5UPhNKxkw83Sk1tg1Xg4vh4CKsM9k6HpQHB1t1/BdiLB\nfouUUrzVuxHHLqbx/IIYqgb52v/OuhDCLqauO8riHacZ3bUO3RsVcP342V2w6i04vAx8g6H7O9D8\nEXD3sm+xdiRTMQXg6ebKl4OaU8bXk8dnRXEhOdPskoQQ/7Ji33ne/eMAPZpUZFTnWrc+QPwhmP8w\nTG0Pp7bA7a/D0zHQakSxDnWQYC+wsn6eTB8cQXJmDo9/F01mTq7ZJQkh8hw8l8LTP+ygUaVAJt0f\nhrqVue/Lx+HHEcZKlyMroMMYI9DbjwYPx+iWIsFugwaVAvioX1N2xSXy7LydslJGiGLgfHImj3yz\nFV9PN6YPjsDbI58rYJLPwq+jjbXoexZBq5FGoHceB96lCrdoO5Ngt1G3hhUYl7dSZuLS/WaXI0SJ\nlppl4dGZ20jMyOHrIZFUCMzHlElaAix/BSY3he3fQvggeHondJsAvmULv+hCIDdP7eCxdtWJu5zB\n9PXHqFzKmyFtq5tdkhAljiXXyn/mbOfAuRS+ejiCRpVvsqghMwk2TYHNn0NOOjTpD7eNgSDH//Mr\nwW4HSile7dGA04kZvPHrPiqV8uaOhkXQ21kIARjLGl/9eQ9rD8Xzdu/GdKpb7vpvzk6DLVONJl2Z\nidCgF3QaB8GF3Lq3CMlUjJ24uigm929Gk5BSPPXDDnaecti9RoRwOJ+vib2yYcbAlqHXfpMlywj0\nT5rCyjegSgt4Yh08MMupQh0k2O3K28OVGXlPtj02cxsnE9LNLkkIp/fzztNMWnaQXk0r8fy1NszI\ntUD0tzA5HJaOgbJ14NFl8OACqBhW9AUXAQl2Oyvr58nMR1qQqzVDvtnK5bRss0sSwmn9FZvACwt2\n0bJ6EO/d3wSXqzfMsFph1wL4LBKWPAX+5WHQjzDkVwhtZV7RRUCCvRDUDPZj+uAI4hIzeGTmNtKy\npBukEPa253QSw2ZFEVrGh2lXN/bSGg78Bl+2g8VDwc0b+s+FoSuhZudi08+lMEmwF5LIakF8OqAZ\nu+ISGT47miyLPMAkhL0cu5jGkG+24u/lxqxHWxDo424E+pGVML0z/DAQLJlw3wwYvgHq3VUiAv1v\nEuyFqFvDCrxzXxPWH77I6Hkx5MoDTELY7FxSJg99tQWrhu+GtqRSKW9jg+iZd8PsPpAWDz2nwH+2\nQuP7waXkxZwsdyxkD0RUISk9hwm/7yfA2523eze6tcebhRBXXE7LZtCMLSSmZ/PDsNbUzDkCs8cb\nj/77loM7J0Hzh8HNxta8Ds5uwa6UcgWigNNa6x72GtcZPN6hBpfSs/liTSxBvu680K2e2SUJ4XDS\nsiw8MnMbJy6lM693II03/gf2LwHv0tDlDWgxDDx8zC6zWLDnFfvTwH4gwI5jOo0x3eqSmJ7DZ6tj\nKe3jwdD2NcwuSQiHkWXJZfjsaBLjDrKm9moqLVkCHn5w20vQeiR4Sevsq9kl2JVSIcDdwARgtD3G\ndDZKKd66txHJGTm89dt+Arzc7buZrhBOypJr5Y3Zf3Lnsan091qHyxl3aDMK2j4DvmXMLq9YstcV\n+8fAGMDfTuM5JVcXxYf9wkjOzOHFxbtwd1P0bhZidllCFFu5KRfY8PXLvH7pZ9zcNS4Rj0CH58Ff\nWnbciM23i5VSPYALWuvom7xvmFIqSikVFR8fb+thHZanmyvTB0fQukYZnpsfw887T5tdkhDFT8Zl\nrCvexPJhE9pfWsTRinfi+tR2uPt9CfV8UFrbtgRPKTURGARYAC+MOfbFWuuHrveZiIgIHRUVZdNx\nHV16toVHvtlG1InLTO7fjLubFHDbLiGcSVYqbPkSvWkyKjOJJbmtuNzieQbf09XsyooFpVS01jri\nZu+z+Ypda/2y1jpEa10N6A+sulGoC4OPhxtfD4kkPNRoGvbHnnNmlySEeXIy4a/P4ZMwWDWe/e6N\nuDNrIoc7fCqhXgAlb+V+MeLr6cY3j7QgLCSQUXO3s2LfebNLEqJo5eZA1DfwaTgsexldviFTa0/l\nrvj/0Om2zjzbpbbZFTokuwa71nqNrGG/NX6ebsx8tAUNKgUycs52Vh+4YHZJQhQ+ay7EzIMpkfDr\nMxBQGT34F94MmsjE3f480aEGL3SrKw/zFZBcsRcDAV7uzHq0BXUr+PPEd9Es2yvTMsJJaQ37foEv\n2sKPw8DTDwbOx/rIMl7dFcQ3G4/zaNvqvHRnPQl1G0iwFxOB3u7MHtqSRpUDGDlnu6yWEc5Fazi8\nAqZ1hPmDwGqBvjNh2DosNbvy/KJdzN58kiduq8GrPepLqNtIesUUI4He7nz3WEuGfhvFM/N2kpGd\nS/8W19kNRghHcXwjrBoPJ/+CUqHQ63No0g9c3ci2WHl23k5+232W0V3rMKpzLQl1O5BgL2aMG6qR\njJgdzUuLd5OWnctj7Rx/c11RAp2OhlVvQewq8KsAd38AzQaDmwcAmTm5jJyznVUHLvDK3fWlzYYd\nSbAXQ17urkwdFMHTP+xg/K/7yMi28GRnWR0gHMT5vbD6bTjwK3gHwR1vQeRQcPe+8pa0LAuPz4ri\nr6MJTOjdiAdbVjWxYOcjwV5Mebi58OmAZrywcBfvLz9EWnYuY2SVgCjOEmKNQN+zCDz9oeNYaDUC\nvP7ZFzApI4dHZ25jx8nLfPhAmLTVKAQS7MWYm6sLH/QNw9vDlS/WxHIxJYu3+zTG3VXueYtiJPEU\nrHsPdswx+qC3ewbaPAU+Qf/z1rNJGQz5ehtHL6by2cBw7mwsT1wXBgn2Ys7FRTHh3kaU9fNk8srD\nxKdm8dnAcHw95ZdOmCz1Aqz/AKK+Nl63eBzajTY2jb6Gg+dSGPLNVlIyLXz7SAva1CpbhMWWLJIO\nDkApxeiudagY6MW4H3czYPpmvh4SSVm/kr1LjDBJ+iXYNBm2TAVLFjR7EDqMgVLXb0O9+WgCj8+K\nwtvdlflPtKZBJdm2oTBJsDuQAS1CCfbz5Mm527nvi018+0gLqpX1NbssUVJkpRj9XP6aYvy88f3Q\n8WUoU/OGH/t11xlGz4shtIwPMx+JJKS07HJU2GSy1sF0aVCeuY+3Ijkjhz5fbGLnqUSzSxLOLicD\nNn1qNOha8zZU7wAjNsJ9X9001GdsOMaouTtoEhLIwuGtJdSLiAS7A2oWWppFI9rg6+nKgGmbpTOk\nKByWbNj2FUxuBstfgYphMHQV9J8D5Rve+KO5Vt5Yspfxv+7jjgblmT20JaV8PIqocCHB7qBqBPux\neERb6lTwZ/jsaKasOoytvfWFAIwGXTu/hykR8NtzULoaDPkdBv0IIc1v+vGkjBwembmNbzYeZ0ib\nanz+YHO83F0Lv25xhcyxO7Bgf0/mDWvFS4uMte4Hz6cy6f4m8odIFIzVCvt/MdaiXzwIFZvC3R9C\nrdshn89PHI1PZeisKE4mpDOxT2MGSEsMU0iwOzgvd1c+6teUOhX8mbTsICcS0pg2KIIKgV5mlyYc\nhdZweLnx+P+5XRBcDx74Durfk+9AB1h/OJ7/zNmOq4ti9tCWtKohG02bRaZinIBSipEdazH1oeYc\nuZBKzykbiJGbqiI/jq2Dr7vB9w9AVjL0ngYjNkGDnvkOda013246zpBvtlEx0JtfnmwnoW4yCXYn\nckfDCiwe2QYPNxcemPoXi6LjzC5JFFdxUfBtT/j2HkiKgx4fw5NRENYPXPI/lZeZk8vYH3fz+i97\n6VQ3mEUj21AlSFa+mE2mYpxMvQoB/Pyftoycs53nFsQQdeISr9/TUObdheHcblg1AQ4tBZ+y0G0i\nRDwK7rc+dXciIY2Rc7az90wyIzvW5Lk76uLqIr2MigObg10pVQWYBZQHNDBNa/2JreOKgivj58mc\noS354M9DfLEmlphTSXzxUDhVy8jDTCXWxcPGTdG9i8ErEDq/Ci2HGzsYFcCyved4fkEMCvhqcARd\nGly7jYAwh7J1iZxSqiJQUWu9XSnlD0QD92qt913vMxEREToqKsqm44r8Wbn/PKPnx2DVmvf7htGt\nYQWzSxJFKfEkrHkXYr4HN2+j22KbUeBdqkDD5eRambTsINPWHaVx5UA+fzBcpl6KkFIqWmsdcbP3\n2XzFrrU+C5zN+3mKUmo/UBm4brCLonN7/fL8OqodT36/nSe+i+bx9tUZ072edIh0dinnYN37ED0T\nlAu0HAHtngW/4AIPeS4pk1Fzt7Pt+GUGtarKKz3q4+kmU3zFkc1X7P8YTKlqwDqgkdY6+V/fGwYM\nAwgNDW1+4sQJux1X3FyWJZcJv+1n1l8naFqlFB/3ayp9ZpxR+iXY8BFsnQ7WHGj2kNGgK7CyTcMu\n33uOlxbvJjMnl4l9GtOrqW3jiYLJ7xW73YJdKeUHrAUmaK0X3+i9MhVjnt92neXlxbuwWDWv9mhA\n/8gqsnmHM8hMhr8+M35kpxp7inZ8EYJs224uLcvCm0v2MS/qFA0qBjB5QFNqlfO3U9HiVhXZVEze\nwdyBRcCcm4W6MNfdTSoSXrUUzy+I4eXFu1m5/wLv3NdYWgA7qux02DoNNn4MGZeNh4o6jYNy9W0e\nOvrEZUbP38nJS+mM6FiTZ7vUwcNNpvAcgT1unirgW+CS1vqZ/HxGrtjNZ7Vqvt54jPeWHSTAy413\n72vC7fVlZYPDsGRB9Lew/n1IPQ+1ukDnV6BSM5uHzsm18unKw0xZfYSKgd581K8pLar/725IougV\n2VSMUqodsB7YDVjzvjxWa/379T4jwV58HDyXwtM/7ODAuRT6R1bh5bvqE+jtbnZZ4npyLRAzF9a+\nC0mnoGpbY+li1dZ2Gf7AuWTGLNzFrrgk+oRX5r89GxLgJb8fiosin2O/FRLsxUuWJZcPlx9i+vqj\nlPXz5M1eDeneSPaiLFasVmMN+pqJkHAEKoUbV+g1O99SP5fryczJZcqqI3y5NhZ/Lzcm9G7MXbIf\nabEjwS5uWcypRF5avJv9Z5O5o0F53uzVSJqJmU1rOLgUVk+A83ugXANjDr3e3XYJdDC2rRu7eDdH\nL6bRp1llXunRgCBf6Z1eHEmwiwLJybUyY8MxPvrzEO6uLrzYvS4PtqyKizwqXrS0hqNrYNV4OB1t\nrG7pNA4a9gEX+9zATErPYeLS/fyw7RRVgrx5u3dj2tcu+Dp3Ufgk2IVNTiSkMfbH3Ww8kkB4aCne\n6NmIxiGBZpdVMpzcYgT68fUQEAK3jYGmA8HVPnPdVqvmp52nefv3A1xOz2Zou+o806UO3h7ysFFx\nJ8EubKa1ZtH200z8fT+X0rO5PzyEF7rVpVyATM8UirMxRk/0w8vBNxjaPw8Rj4Cb/ZaiRp+4zJu/\n7iPmVCJhIYFM6N2YRpXlL2xHIcEu7CY5M4cpq47wzcZjeLi6MLJTLR5rV106RtpL/EFjDn3fz+BV\nCto9Ay2GgYf9ngw+k5jBu38c4OedZyjn78mL3evRu1llmWJzMBLswu6OXUzj7d/38+e+84SU9mbs\nXfW5s1EFeXK1oC4fNxp07foB3H2g9X+MH172u4JOz7Ywde1Rpq6LxaphWPsajOhYE19P6djtiCTY\nRaHZeOQiby7Zx8HzKYSFBPJs1zrcVidYAj6/ks/AukmwfRa4uEHkUKNBl29Zux0iMyeXOVtO8sWa\nWC6mZnF3k4q8fGc9QkpLJ0ZHJsEuCpUl18qi7XFMXnmE04kZNK9amtFd69CmZhkJ+OtJu2g06Nr2\nFVgtEP4wdHgBAuy3XjzLksu8baf4bPURzidn0aZmGZ67ow7Nq8qTo85Agl0UiWyLlQXRp5iy6ghn\nkzJpUT2I0V3ryJ6XV8tMgk1TYPPnkJMOTfobDbpKV7PbIbItVhZGxzFl1WHOJGUSWa00o7vWpXVN\n+XVwJhLsokhl5vz/leKFlCwiq5XmsXY16NqgfMndLi07DbZMhY2fQGYiNLgXOo2F4Lp2O0RSRg7z\ntp1k5sbjnEnKpFloKZ7rWpe2teRfTs5Igl2YIjMnl7lbTzJjwzHiLmcQGuTDo22r0TeiSsm5YZeT\nCdHfwPoPIC0eaneDzuOgYpjdDnHqUjpfbzzG/G2nSMvOpVWNIJ64rSYd5V6HU5NgF6ay5Fr5c995\nvtpwjOgTl/H3cmNgy1AGt65G5VLeZpdXOHJzYOccWPseJJ+Gau2NBl2hLe0yvNaaqBOX+XrDMZbt\nPYeLUvQMq8Sj7arLWvQSQoJdFBvbT15mxvpjLN1zFg20q1WWvhFVuKNBeedYC2/NhT2LjAZdl45C\n5Qi4/VWo0dEuw19IyWTx9tMsiDpFbHwaAV5uPNiqKg+3ria9fEoYCXZR7MRdTmdBVBwLo+M4nZhB\ngJcb9zarTN/mVWhUOcDxphC0hgO/wqoJEL8fyjcyOi7W6W5zg66cXCurDlxgQdQpVh+MJ9eqiaha\nmr4RIfRoUqnkTGuJf5BgF8WW1arZFJvAguhTLN1zjmyLlTrl/ejeqCLdG1agfkX/4h3yWkPsSuPx\n/zM7oEwt46Zog942NejKtljZFHuRZXvPs3zvORLSsinn70mf8BD6RoRQM9jPjichHJEEu3AISek5\n/BJzmiUxZ9l24hJaQ5Ugb7o3rEC3hhUIDy1dvB57P7EJVo6Hk5sgMNRYttikP7gW7Ao6LcvC2kPx\nLNt7jlX7L5CSZcHXw5WO9cpxX3hlOtQOxs1VtqMTBgl24XDiU7JYsf88y/aeY+ORi+Tkasr6edC6\nZlna1CxD6xplqFrGx5yr+dPbjSv02JXgV954sCh88C036LLkWtl1Oom/YhPYfDSBrccukWWxUtrH\nna4NytOtYQXa1irrHPcehN0VabArpboDnwCuwFda63du9H4JdnEzyZk5rD5wgVUHLvBXbAIXUrIA\nqBjoResaZWhVswxNq5SiRlnfwr2ivbDfCPQDv4J3aePR/8jHwSN/j+anZVk4cC6Z6BOX2RSbwLZj\nl0jLzgWgbnl/2tQqwx0NKhBZrbRcmYubKso9T12BQ0BXIA7YBgzQWu+73mck2MWt0FoTG5/GX0cT\n2Jx3pZuQlg2Ap5sL9Sr406BSIA0rBdCwUgA1y/nZvk9nQiyseQd2LwAPP2jzJLQaCV4B13y71aqJ\nT83i4LkU9p5JZu+ZJPadTebYxTT+/iNWM9iX1jXL0LpGWVrVCKKMn/3a8YqSoSiDvTXwX611t7zX\nLwNorSde7zMS7MIWVqsmNj6VPWeS2Hs6+UqQJmdarrwnwMuNKkE+hJT2pkpp478VAr0I8HInwNud\nAC93Ar3d8fNy++eTsUlxxjr0HbPB1YPcyMdJDB9JEv4kZeSQnGkhMT2bs0mZnLqUzqnLGcRdTifu\ncgbZFuuVYSqX8s77iyaQBpVKPUTJAAAUxElEQVQCCAsJlD72wmb5DXZ7rJmqDJy66nUcYJ8nMoS4\nBhcXRe3y/tQu70/vZsbXtNacTsxg75lkTiSkEXc5g1OX0omNT2PtoXgyc6zXHc/b3ZUyKonH1U/0\n509csDKfLnyRcy9xqwNh9fZrfq60jzshpX2oV8GfLvXLU6W0NzXL+dGwYiCBPvbZ7UiIgiiyxbBK\nqWHAMIDQ0NCiOqwoIZRShJT2uWZbWq01F1OzuZiaZVx15115J2XkkJWSQNjJWUSen4+bNZtdZe9k\nfaVHSfasyN1K4e/lRoC3cXVvXO27EeDlToVAL/xtne4RopDYI9hPA1Wueh2S97V/0FpPA6aBMRVj\nh+MKkS9KKYL9PQn2v2pOOysVtnwBOz+FrCRodB90HEvTsrVoal6pQtiFPYJ9G1BbKVUdI9D7AwPt\nMK4Q9peTCVEzYP2HkH4R6t4FncZBhUZmVyaE3dgc7Fpri1LqSWAZxnLHr7XWe22uTAh7ys2BHd/B\n2kmQcsbo49L5VQi56X0oIRyOXebYtda/A7/bYywh7MqaayxZXDPR2GO0SkvoMw2qtze7MiEKjXQS\nEs7JaoX9v8Dqt+HiQajQGAbOh9p32NygS4jiToJdOBet4cgKWDUezsZA2TrQdybU72VTgy4hHIkE\nu3AexzcYDbpObYZSVeHeL6HJA+AifVdEySLBLhxfXLRxhX50NfhXhLs/gGaDwc3D7MqEMIUEu3Bc\n5/YYc+gHfwOfMnDHWxA5FNyddOs9IfJJgl04noRYI9D3LAJPf2MdeqsRxs+FEBLswoEknoS178LO\nuUYf9HbPQJunwCfI7MqEKFYk2EXxl3Ie1r8P0TON1y2GQfvR4FfO1LKEKK4k2EXxlX4JNn4MW6ZB\nbjY0ewhuGwOBIWZXJkSxJsEuip/MZNj8Ofz1GWSlQOO+0PElKFPT7MqEcAgS7KL4yMmArdNhw0eQ\ncQnq9TBujJZvYHZlQjgUCXZhPks2bP8W1r0Pqeeg5u3Q+RWoHG52ZUI4JAl2YZ5cC+yaB2vfMVa8\nhLaBvt9A1TZmVyaEQ5NgF0XPaoV9Pxlr0RMOQ8Wm0OMj40pdGnQJYTMJdlF0tIZDy2DVW3B+NwTX\nh36zjbl0CXQh7EaCXRSNo2uNQI/bCqWrQZ/pxnZ00qBLCLuTYBeF69Q2WPUmHFsHAZWhx8fGenRX\n2QhaiMIiwS4Kx9ldsHoCHPoDfMpCt4kQ8Si4e5ldmRBOz6ZgV0pNAu4BsoFY4BGtdaI9ChMOKv4Q\nrHkb9v4IXoHGvqIth4Onn9mVCVFi2HrF/ifwct6G1u8CLwMv2l6WcDiXTxgNumLmgps3tH8e2owC\n71JmVyZEiWNTsGutl1/1cjNwv23lCIeTfDavQde3oFyg5Qho9yz4BZtdmRAllj3n2B8F5l3vm0qp\nYcAwgNDQUDseVpgiLQE2fmS0ALBaoNkg6PACBFY2uzIhSrybBrtSagVQ4RrfGqe1/jnvPeMACzDn\neuNoracB0wAiIiJ0gaoV5stMMppz/fU5ZKdCk35Gg66g6mZXJoTIc9Ng11p3udH3lVJDgB7A7Vpr\nCWxnlZ0GW6fBho8hMxHq9zQadJWrZ3ZlQoh/sXVVTHdgDHCb1jrdPiWJYsWSZWxwse59SLsAtboa\nDboqNTW7MiHEddg6xz4F8AT+VMYj4Zu11sNtrkqYL9cCMd/D2vcg6RRUbQf9voPQVmZXJoS4CVtX\nxdSyVyGimLBaYe9io0HXpVio3Bx6ToYanaSfixAOQp48FQat4eBSo5/Lhb1QriH0nwt175RAF8LB\nSLCXdFrD0TVGoJ+OgqCacN8MaNgHXFzMrk4IUQAS7CXZyc2wcjyc2ACBVaDnpxA2EFzlt4UQjkz+\nBJdEZ3YaV+hH/gTfcnDnJGj+MLh5ml2ZEMIOJNhLkgsHjI6L+38B79LQ5Q1oMQw8fMyuTAhhRxLs\nJcGlY7DmHdg9H9x94baXoPVIo/uiEMLpSLA7s6TTsG4S7PgOXNyg9ZPQ9hnwLWN2ZUKIQiTB7ozS\nLsL6D2HbV6Ct0HyI0UY3oKLZlQkhioAEuzPJSIRNn8LmL8CSAWED4LYXoXRVsysTQhQhCXZnkJUK\nW76ETZON7osNe0PHsRBcx+zKhBAmkGB3ZDmZEPU1bPgQ0uKhTnej42LFJmZXJoQwkQS7I8rNgR2z\njRujyaehegfo/D1UaWF2ZUKIYkCC3ZFYc2H3QlgzES4fg5BIuPcLqHGb2ZUJIYoRCXZHoDXsX2J0\nXIzfD+Ubw4B5UKebNOgSQvwPCfbiTGs4shJWjYezO6FMbbj/G2hwrzToEkJclwR7cXV8oxHoJ/+C\nUqHQ63Njf1Fp0CWEuAlJieLmdLTRoCt2FfhVgLveh/CHwc3D7MqEEA7CLsGulHoOeB8I1lpftMeY\nJc75fUaDrgO/gncQdB0PkUOlQZcQ4pbZHOxKqSrAHcBJ28spgRJijVUuuxeCp7/xYFGrEeAVYHZl\nQggHZY8r9o+AMcDPdhir5EiKMzaK3jEbXD2g7dPGD58gsysT4pbk5OQQFxdHZmam2aU4DS8vL0JC\nQnB3dy/Q520KdqVUL+C01jpGybK7/Em9YDToipphvI4cCu2fA//y5tYlRAHFxcXh7+9PtWrVkByw\nndaahIQE4uLiqF69eoHGuGmwK6VWABWu8a1xwFiMaZibUkoNA4YBhIaG3kKJTiLjMmycbPR0sWRB\n04FGg65SVcyuTAibZGZmSqjbkVKKMmXKEB8fX+AxbhrsWusu1zl4Y6A68PfVegiwXSnVQmt97hrj\nTAOmAUREROgCV+xoslJg85dG18WsZGh0H3QaC2Vqml2ZEHYjoW5ftv7/LPBUjNZ6N1DuqkKOAxGy\nKiZPTgZsm2E06EpPgLp3Q+dxUL6h2ZUJUWKkpqbSsWNHLl26xIYNG6hUqdKV7z344INERUXh7u5O\nixYtmDp1aoHntIsbeXzR3izZRqBPbgbLx0GFxjB0JQz4XkJdiCJksVh44IEHGDRoEJMmTaJXr14k\nJydf+f6DDz7IgQMH2L17NxkZGXz11VcmVmtfdntASWtdzV5jOSRrLuyabyxdTDwBVVpCn+lQvb3Z\nlQnh1LZt28Zjjz3G1q1byc3NpUWLFsybN4+PPvqIO++8k1GjRgHg6upK//79+fnnn3F3d+euu+66\nMkaLFi2Ii4sz6xTsTmld9NPdEREROioqqsiPWyisVtj/s9Gg6+IhqBgGnV+FWl2kQZcoEfbv30/9\n+vUBeGPJXvadSb7JJ25Ng0oBvH7Pjf+1+8orr5CZmUlGRgYhISG8/PLL+R4/JyeHli1b8sknn9C+\nffG5ELv6/+vflFLRWuuIm31WWgoUlNZweLnx+P+5XVC2LjwwC+r3lEAXooi99tprREZG4uXlxeTJ\nk2/psyNHjqRDhw7FKtRtJcFeEMfWGYF+aguUrga9p0LjvuDianZlQpjqZlfWhSUhIYHU1FRycnLI\nzMzE19c3X5974403iI+PZ+rUqYVcYdGSYL8VcVGw8k04thb8K0GPj6DZIHB1jjvpQjiqJ554gvHj\nx3Ps2DFefPFFpkyZctPPfPXVVyxbtoyVK1fi4mRtsCXY8+PcHuMK/dBS8CkL3d6GiEfB3dvsyoQo\n8WbNmoW7uzsDBw4kNzeXNm3asGrVKjp37nzDzw0fPpyqVavSunVrAPr06cNrr71WFCUXOgn2G7l4\n2LgpuncxeAZC51eg5Qjw9DO7MiFEnsGDBzN48GDAWPmyZcuWfH3OYrEUZlmmkmC/lsSTsOZdiPke\n3LyNXi5tRoF3abMrE0KIm5Jgv1rKOVj3PkTPBOUCLYdDu9HgF2x2ZUIIkW8S7ADpl2DDR7B1Olhz\noNlD0OEFCAwxuzIhhLhlJTvYM5Nh8+ewaQpkp0KTB6DjSxBUw+zKhBCiwEpmsGenw7bpsOFjyLgE\n9e+BTuOgXP2bf1YIIYq5khXslmzY/q0xj556znjsv/MrUKmZ2ZUJIYTdONeq/OvJtRhb0H3aHH5/\n3uiF/shSeGiRhLoQTua///0v77//fr7f/8svv/DOO+8U6Fg//fQT+/btu/L6tddeY8WKFQUay56c\n+4rdaoV9P8LqiZBw2Ajxez6Gmp2ln4sQAovFQs+ePenZs2eBPv/TTz/Ro0cPGjRoAMCbb75pz/IK\nzDmv2LWGg0thagdY+Ci4uEG/2fD4aqh1u4S6EE5mwoQJ1KlTh3bt2nHw4EEAYmNj6d69O82bN6d9\n+/YcOHAAgCFDhjB8+HBatmzJmDFjmDlzJk8++SRJSUlUrVoVq9UKQFpaGlWqVCEnJ4fp06cTGRlJ\nWFgY9913H+np6WzatIlffvmFF154gaZNmxIbG8uQIUNYuHAhf/zxB3379r1S35o1a+jRowcAy5cv\np3Xr1oSHh9O3b19SU1Pt/v/D+a7Yj64xHv+P2walqxs90RvdJw26hCgKS1+Cc7vtO2aFxnDn9adK\noqOj+eGHH9i5cycWi4Xw8HCaN2/OsGHD+PLLL6lduzZbtmxh5MiRrFq1CjA24N60aROurq7MnDkT\ngMDAQJo2bcratWvp1KkTv/76K926dcPd3Z0+ffrw+OOPA0aL4BkzZjBq1Ch69uxJjx49uP/++/9R\nU5cuXRg2bBhpaWn4+voyb948+vfvz8WLF3nrrbdYsWIFvr6+vPvuu3z44Yd2b2XgPMF+aqvRoOv4\negioDPd8Ak0flAZdQji59evX07t3b3x8fADo2bMnmZmZbNq06R9XzVlZWVd+3rdvX1xd//dir1+/\nfsybN49OnTrxww8/MHLkSAD27NnDK6+8QmJiIqmpqXTr1u2GNbm5udG9e3eWLFnC/fffz2+//cZ7\n773H2rVr2bdvH23btgUgOzv7Sq8ae7I52JVSo4D/ALnAb1rrMTZXdSvO7jKu0A8vA99g6P4ONH8E\n3L2KtAwhBDe8si5KVquVUqVKsXPnzmt+/3ptfXv27MnYsWO5dOkS0dHRVxqJDRkyhJ9++omwsDBm\nzpzJmjVrblpD//79mTJlCkFBQURERODv74/Wmq5duzJ37twCn1t+2DTHrpTqBPQCwrTWDYH834q2\nVfwhmP8wTG1v9EW//XV4OgZajZBQF6IE6dChAz/99BMZGRmkpKSwZMkSfHx8qF69OgsWLABAa01M\nTMxNx/Lz8yMyMpKnn36aHj16XLmqT0lJoWLFiuTk5DBnzpwr7/f39yclJeWaY912221s376d6dOn\n079/fwBatWrFxo0bOXLkCGDM4x86dMim878WW2+ejgDe0VpnAWitL9he0k1cPg4/joDPW8KRFdBh\njBHo7UeDR/6a6wshnEd4eDj9+vUjLCyMO++8k8jISADmzJnDjBkzCAsLo2HDhvz888/5Gq9fv37M\nnj2bfv36Xfna+PHjadmyJW3btqVevXpXvt6/f38mTZpEs2bNiI2N/cc4rq6u9OjRg6VLl165cRoc\nHMzMmTMZMGAATZo0oXXr1ldu6tqTTXueKqV2Aj8D3YFM4Hmt9babfa7Ae56unQRr3zVuhEYOhXbP\ngm/ZWx9HCGE319qbU9iuUPc8VUqtACpc41vj8j4fBLQCIoH5Sqka+hp/WyilhgHDAEJDQ2922Gsr\nFQrhg6HD8xBQqWBjCCGEk7tpsGutu1zve0qpEcDivCDfqpSyAmWB+GuMMw2YBsYVe4GqDetn/BBC\nCHFdts6x/wR0AlBK1QE8gIu2FiWEEKLgbF3u+DXwtVJqD5ANPHytaRghhHPTWqPkiW67sTVGbQp2\nrXU28JBNFQghHJqXlxcJCQmUKVNGwt0OtNYkJCTg5VXwZdvO8+SpEMIUISEhxMXFER//P7fWRAF5\neXkRElLwHdwk2IUQNnF3d6d69epmlyGu4pzdHYUQogSTYBdCCCcjwS6EEE7GppYCBT6oUvHAiQJ+\nvCzOs1ZezqX4cZbzADmX4sqWc6mqtQ6+2ZtMCXZbKKWi8tMrwRHIuRQ/znIeIOdSXBXFuchUjBBC\nOBkJdiGEcDKOGOzTzC7AjuRcih9nOQ+QcymuCv1cHG6OXQghxI054hW7EEKIG3DIYFdKjVdK7VJK\n7VRKLVdKOeyuG0qpSUqpA3nn86NSqpTZNRWEUqqvUmqvUsqqlHLI1QtKqe5KqYNKqSNKqZfMrqeg\nlFJfK6Uu5HVddVhKqSpKqdVKqX15v7eeNrumglJKeSmltiqlYvLO5Y1CPZ4jTsUopQK01sl5P38K\naKC1Hm5yWQWilLoDWKW1tiil3gXQWr9oclm3TClVH7ACUzG2SCzA3ofmUUq5AoeArkAcsA0YoLXe\nZ2phBaCU6gCkArO01o3MrqeglFIVgYpa6+1KKX8gGrjXQX9NFOCrtU5VSrkDG4CntdabC+N4DnnF\n/neo5/EFHO9vpzxa6+Vaa0vey81AwVu6mUhrvV9rfdDsOmzQAjiitT6a1476B6CXyTUViNZ6HXDJ\n7DpspbU+q7XenvfzFGA/UNncqgpGG1LzXrrn/Si03HLIYAdQSk1QSp0CHgReM7seO3kUWGp2ESVU\nZeDUVa/jcNAQcUZKqWpAM2CLuZUUnFLKVSm1E7gA/Km1LrRzKbbBrpRaoZTac40fvQC01uO01lWA\nOcCT5lZ7Yzc7l7z3jAMsGOdTLOXnPISwN6WUH7AIeOZf/1p3KFrrXK11U4x/lbdQShXaNFmx7cd+\no020/2UO8DvweiGWY5ObnYtSagjQA7i9OG8teAu/Jo7oNFDlqtcheV8TJsqbj14EzNFaLza7HnvQ\nWicqpVYD3YFCucFdbK/Yb0QpVfuql72AA2bVYiulVHdgDNBTa51udj0l2DagtlKqulLKA+gP/GJy\nTSVa3g3HGcB+rfWHZtdjC6VU8N8r3pRS3hg36Qsttxx1VcwioC7GKowTwHCttUNeXSmljgCeQELe\nlzY74gofpVRv4FMgGEgEdmqtu5lb1a1RSt0FfAy4Al9rrSeYXFKBKKXmAh0xugieB17XWs8wtagC\nUEq1A9YDuzH+rAOM1Vr/bl5VBaOUagJ8i/F7ywWYr7V+s9CO54jBLoQQ4voccipGCCHE9UmwCyGE\nk5FgF0IIJyPBLoQQTkaCXQghnIwEuxBCOBkJdiGEcDIS7EII4WT+D0JMm6YIdH6LAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "x = np.linspace(-3,3)\n",
    "x_squared, x_squared_der = s.run([scalar_squared,derivative],\n",
    "                                 {my_scalar:x})\n",
    "\n",
    "plt.plot(x, x_squared,label=\"x^2\")\n",
    "plt.plot(x, x_squared_der, label=\"derivative\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why that rocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    }
   ],
   "source": [
    "my_vector = tf.placeholder('float32',[None])\n",
    "\n",
    "#Compute the gradient of the next weird function over my_scalar and my_vector\n",
    "#warning! Trying to understand the meaning of that function may result in permanent brain damage\n",
    "\n",
    "weird_psychotic_function = tf.reduce_mean((my_vector+my_scalar)**(1+tf.nn.moments(my_vector,[0])[1]) + 1./ tf.atan(my_scalar))/(my_scalar**2 + 1) + 0.01*tf.sin(2*my_scalar**1.5)*(tf.reduce_sum(my_vector)* my_scalar**2)*tf.exp((my_scalar-4)**2)/(1+tf.exp((my_scalar-4)**2))*(1.-(tf.exp(-(my_scalar-4)**2))/(1+tf.exp(-(my_scalar-4)**2)))**2\n",
    "\n",
    "der_by_scalar = tf.gradients(weird_psychotic_function, my_scalar)[0]\n",
    "der_by_vector = tf.gradients(weird_psychotic_function, my_vector)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNXZwPHfmcmeyUoWlgAJ+yZb\nEEUWwbpgtW5Vwa31tRVta7X1Vatt32rfKrV16VtqFbVutSruqLhTCUUQkB1kkS1AICwJ2ZPJet4/\nzkwIIcvsM5k8388nn5vM3Ln3nASeOfPcc5+jtNYIIYQIf5ZgN0AIIURgSMAXQohuQgK+EEJ0ExLw\nhRCim5CAL4QQ3YQEfCGE6CYk4AshRDchAV8IIboJCfhCCNFNRAS7AS2lpaXp7Oxsj19fVVVFfHy8\n7xoUJOHSD5C+hKJw6QdIX5zWrl1bpLVO72y/kAr42dnZrFmzxuPX5+XlMX36dN81KEjCpR8gfQlF\n4dIPkL44KaX2ubKfpHSEEKKbkIAvhBDdhAR8IYToJkIqh9+W+vp6CgoKsNvtne6blJTEtm3bAtAq\n/wpEP2JiYsjKyiIyMtKv5xFChI6QD/gFBQUkJCSQnZ2NUqrDfSsqKkhISAhQy/zH3/3QWlNcXExB\nQQE5OTl+O48QIrSEfErHbrfTo0ePToO9cJ1Sih49erj0qUkIET5CPuADEuz9QH6nQnQ/XSLgCyFE\nWFv5FOlHl/v9NBLwXTRv3jyGDx/Odddd55Pj5efn8+qrrzb/vGbNGm6//XafHFsI0cWsfIq0olV+\nP03IX7QNFU8++SSLFy8mKyvLJ8dzBvxrr70WgAkTJjBhwgSfHFsI0cXYS6mPt/n9NDLCd8Gtt97K\nnj17uPDCC0lKSuLRRx9tfm7UqFHk5+eTn5/P8OHDufnmmxk5ciTnn38+NTU1AOzatYtzzz2XMWPG\nMH78eHbv3s29997LsmXLGDt2LH/5y1/Iy8vj4osvBuD48eNcdtlljB49mjPPPJNNmzYB8MADD3DT\nTTcxffp0BgwYwLx58wL/yxCiq1v8e3g/hD5NNzWBvZyGCP/XBOpSI/zff/ANWw+Vt/t8Y2MjVqvV\nrWOO6J3I/d8b2eE+8+fP55NPPmHJkiU88cQT7e63c+dOXnvtNZ599lmuvvpq3n77ba6//nquu+46\n7r33Xi6//HLsdjtNTU08/PDDPProoyxatAgwdTSc5s6dy7hx41i4cCFffPEFP/jBD9iwYQMA27dv\nZ8mSJVRUVDB06FB+8pOfyFx6IVylNax/GaqOwZCZMOy7wW4R1JYDmoYIGeF3KTk5OYwdOxaA3Nxc\n8vPzqaio4ODBg1x++eWAueEpLi6uw+OsXLmSG264AYBzzjmH4uJiysvNG91FF11EdHQ0aWlpZGRk\ncOTIET/2SIgwc3yPCfbKCh/fA3VVwW4R2EsBAhLwu9QIv7OReCBuvIqIiKCpqan555Zz2aOjo5u/\nt1qtzSkdX2p9joaGBp+fQ4iwdcBxYfSiR2HRL2Hpn+G83we3TTUm4NdHygg/5GRnZ7Nu3ToA1q1b\nx969ezvcPyEhgaysLBYuXAhAbW0t1dXVJCQkUFFR0eZrJk2axCuvvAKYVE9aWhqJiYk+7IUQ3dT+\nlRCTBONvhHHXw1dPwJGtwW1T8wjf/zl8Cfhu+v73v8/x48cZOXIkTzzxBEOGDOn0NS+//DLz5s1j\n9OjRnHXWWRw+fJjRo0djtVoZM2YMf/nLX07a/7777mPt2rWMHj2ae++9l5deeslf3RGie9m/Evqe\nARYLnPu/EJ0An/9PcNtUIymdkJOfn9/8/WeffdbmPlu2bGn+/q677mr+fvDgwXzxxRen7N/6Mefi\nB6mpqc2fCFp64IEH2j2fEKIT1cehaAeMmWV+ju8BI6+ALW8Ht10ywhdCCB87sNps+5554rHUHBNw\na0qC0yYI6AhfAr4Qons4sBIskdBn/InHkvubbYlLKwT6h70ULBE0WmP8fiqfBHyl1PNKqaNKqS0t\nHntAKXVQKbXB8RUCE16FEN3W/lXQawxExp54LCXbbEuDGPBrSiEmGQJQ0NBXI/wXgZltPP4XrfVY\nx9dHPjqXEEK4p6EODq2Dfmee/HiKc4SfH/AmNbOXQmxyQE7lk4Cvtf4PcNwXxxJCCJ8r3AgNdjND\np6WYJIhNCXJKp8yM8APA3zn825RSmxwpnxQ/n0sIIdp2YKXZth7hg8njB3OEXxO4Eb4/p2U+BfwB\n0I7tY8BNrXdSSs0B5gBkZmaeVFMGzPqu7d2g1FpjY6PL+3pj7ty52Gw2l8sZf/TRR2zfvp0777zT\npf1b9mPRokUMGjSIYcOGAfDggw8yefJkZsyY4VnjW7Db7af8vn2tsrLS7+cIlHDpS7j0A1zvy8gt\ni7DF9GTVmq3AyTdajWiIx3ZoG6uD9DuZWFJIRUN8QP4ufgv4WuvmIi9KqWeBRe3s9wzwDMCECRO0\ncy6607Zt21wulxCoNW2jo6OJjo526VwNDQ3MmjXLreO37Menn35KZGQkp59+OgB/+tOf3G9wO2Ji\nYhg3bpzPjteWvLw8Wv9Nu6pw6Uu49APc6MumX8LASW3vW78EVn7N9GlTweJe8UWfWFVLXL8h2Gw2\nv/9d/JbSUUr1avHj5UCXvkvooYceYsiQIUyZMoUdO3YAsHv3bmbOnElubi5Tp05l+/btANx4443c\neuutnHHGGdxzzz28+OKL3HbbbZSVldG/f//mWjxVVVX07duX+vp6nn32WU4//XTGjBnD9ddfT3V1\nNStWrOD999/n7rvvZuzYsezevZsbb7yRt956i08++YSrrrqquX0tyyt/9tlnTJo0ifHjx3PVVVdR\nWVkZ4N+WECFEayg/BEntrGWR0h8a66CiMLDtAtM2e1nXSukopV4DpgNpSqkC4H5gulJqLCalkw/c\n4vWJPr4XDm9u9+nYxgawutmlnqfBhQ93uMvatWtZsGABGzZsoKGhgfHjx5Obm8ucOXOYP38+gwcP\nZtWqVfz0pz9tvnu2oKCAFStWYLVaefHFFwGTnho7dixLly5lxowZLFq0iAsuuIDIyEiuuOIKbr75\nZgDuvvtunnvuOX7+859zySWXcPHFF3PllVee1KZzzz2XOXPmUFVVRXx8PK+//jqzZ8+mqKiIBx98\nkMWLFxMfH8+f/vQnHn/8cX73u9+593sRIlxUHzcXbBP7tP28c2pmyb723xT8pbYCdKO5aFvv/9P5\nJOBrra9p4+HnfHHsULBs2TIuv/zy5rLGl1xyCXa7nRUrVpw0yq6trW3+/qqrrmqzNv+sWbN4/fXX\nmTFjBgsWLOCnP/0pYMok/Pa3v6W0tJSKigpmzmxrlusJERERzJw5kw8++IArr7ySDz/8kD//+c8s\nXbqUrVu3MnnyZADq6uqYNGmS178DIbqs8oNm217AT24xNTN7ckCa1MxRVoHYLhTwA6aTkXhNgHL4\nAE1NTSQnJzcvTNJafHzbdTEuueQSfv3rX3P8+HHWrl3LOeecA5g00MKFCxkzZgzz589n5cqVnbZh\n9uzZPPHEE6SmpjJhwgQSEhLQWnPeeefx2muved45IcJJZwE/qS8oS3BuvnKUVSAmGdpf28lnpLSC\nC6ZNm8bChQupqamhoqKCDz74gLi4OHJycnjzzTcB0FqzcePGTo9ls9k4/fTTueOOO7j44oubPwVU\nVFTQq1cv6uvreeONN5r376iM8tlnn826det49tlnmT17NgBnnnkmy5cvZ9euXYC5TvDtt9961X8h\nurTmgN+77ecjosybQTCmZrYc4QeABHwXjB8/nlmzZjFmzBguvPDC5hkzr7zyCs899xxjxoxh5MiR\nvPfeey4db9asWfzrX/86afbOH/7wB8444wwmT57M4MGDmx+fPXs2jzzyCOPGjWP37t0nHcdqtXLx\nxRfz8ccfN1+wTU9P58UXX+Saa65h9OjRTJo0qflishDdUvkhsESALaP9fVKygxTwy8w2QDdeobUO\nma/c3Fzd2tatW095rD3l5eUu7xvKAtUPd363nlqyZInfzxEo4dKXcOmH1i725e05Wj8+suN93v2p\n1o8M8Umb3LL2n1rfn6j18Xyv/i7AGu1CjJURvhAivJUfbD9/75SSDZWHod73y5J2SFI6QgjhQ+UH\n28/fOzVXzdzv9+acpKbUXDCOCsxkky4R8M0nFuFL8jsV3YLzpqtOA36QqmbaS00BN0tgQnHIB/yY\nmBiKi4slQPmQ1pri4mJiYvy/4IIQQeW86aqzG6pa3nwVSM5a+AES8vPws7KyKCgo4NixY53ua7fb\nwyKIBaIfMTExZGUF+K5CIQKtsymZTvHpEBkXnBF+gPL30AUCfmRkJDk5OS7tm5eX5/diYIEQLv0Q\nIujKD5ltZxdtlQpOmeQAj/BDPqUjhBAeKy8w284CPpg8fqDvtg3wCF8CvhAifLly05VTUl8oO+D/\nNrVU47hoGyAS8IUQ4avsICT0cq3OfXy6ufO1MQBVzOBEaWRJ6QghhA+4MgffKS7VbKuL/deeluqr\noaleUjpCCOETrszBd4pPM9tABfyWlTIDRAK+ECI8ae1aWQWnOEfAryryX5taCnBZBZCAL4QIVzUl\nHa901VrzCD9AAb+rjvCVUs8rpY4qpba0eCxVKfW5UmqnY5vii3MJIYRLypxTMl3N4fcw2+rj/mlP\na114hP8i0HpNvnuBf2utBwP/dvwshBCB4bzpytV1amMdF20DldLpqiN8rfV/gNZvi5cCLzm+fwm4\nzBfnEkIIl5S7OcK3RkBsSuBSOs4RfgDn4StfFSVTSmUDi7TWoxw/l2qtkx3fK6DE+XOr180B5gBk\nZmbmLliwwOM2VFZWYrPZPH59qAiXfoD0JRSFSz+g477k7HmZfvvfYenZb4FyYR4+MHHVT6m0ZbN1\n5D2+bGabsve+Qva+N8g7+x1QVq/+LjNmzFirtZ7Q6Y6urJLiyheQDWxp8XNpq+dLOjtGWyteuSNc\nVvIJl35oLX0JReHSD6076cs7t2j92Aj3DviP87V+4SKv2uSyD+/Sem7f5h+7+opXR5RSvQAc26N+\nPJcQQpysrMD1dI5TfFrg5uHbyyA2cOkc8O+0zPeBHzq+/yHg2grfQgjhC+7cdOUU1yOwF20DeMEW\nfDct8zXgK2CoUqpAKfUj4GHgPKXUTuBcx89CCBEYlUdNHR13xPUwI/xALLgU4EqZ4KN6+Frra9p5\n6ju+OL4QQrilvgbqKk7cTOWq+DTQjY5g7Odbh2pKIX2of8/RitxpK4TwjYZaKN4Nu/5ttsHkTMvE\np7v3uubyCgHI49sDWxoZusCKV0KILmDhz2DDK4AjFZKSA7evNytJBUOVY46IuwE/3nm3bREwyKdN\nOoW9POABX0b4QgjvVB6Fja/CkAvgsqdgyp1QshcObwpemzwe4TsDvp9H+I310FAjAV8I0cVseRt0\nE5z7exh7LUy6zdzotPX94LWp6pjZ2jxN6fh5pk5thdlGJ/j3PK1IwBdCeGfTG9BzNGQMMz/H94Ds\nybB1YWBmu7TFGfA9HuH7O+CXm210on/P04oEfCGE54p2waF1MPrqkx8fcSkU74Kj24LTrqoiiIyD\nqHj3XhcVZ17n74u2dmfAlxG+EKKr2PwGoGDUlSc/Pux75vFtQUrrVB1zf0qmU1wA7rZ1pnRiZIQv\nhOgKtIZNr0PONEhsdYNTQib0mwRbg3SDfeVR99M5TvE9ApjSkRG+EKIrKFgDJfkwelbbz4+4FI5u\nhaKdAW0WYFI6ngb8uLQAXrSVWTpCiK5g0+sQEQPDv9f2887HgzHKrzrmRcDv4f9Vr+xlZisjfCFE\nyNMatn8Ig89vPw+d1AeyTodtHwS2bU1NJiXjcUonLQApHcnhCyG6irICqDgE2VM73q//ZDjyjbnR\nKFDspdDU4N0Iv74a6qp9266WasvBEmE+IQWQBHwhhPsKVptt34kd75cxHJrq4fhe/7fJydO7bJ2c\ns3v8OcqvrTBz8ANcekICvhDCfQdWm/nqmaM63s9ZDfJYAOfjN9905em0zACUV7CXBzx/DxLwhRCe\nOLAaeo83C393JG2I2R7b4f82OXlaOM0pEBUzaysCnr8HCfhCCHfV15jCaJ2lc8Dc6ZrcD45t93+7\nnJwpHVuGZ68PSEqnPOBlFSAA5ZGVUvlABdAINGhXVlYXQoSuQ+vNRVFXAj5A+rAAj/CPAQpiUz17\nfZzjdf5M6dSWQ2If/x2/HYEa4c/QWo+VYC9EGDiwymyz3Aj4RTuhscF/bWqp6pgJ2p2lm9oTk2xm\n0Pjz5it7cEb4ktIRQrjnwNeQOvDEYiGdSR8GjbVQus+/7XLy5qYrMDNn4vxcXqG2Imwv2mrgM6XU\nWqXUnACcTwjhL1qbEb6r6RwwAR8CVznTm7IKTnFp/rtoq7VJ6QThom0gljicorU+qJTKAD5XSm3X\nWv/H+aTjTWAOQGZmJnl5eR6fqLKy0qvXh4pw6QdIX0KRN/2IqSnkzOoidtQkU+jiMawN1UwF9qz+\nmP1HbB6dtz1t9WXisX1U2nLY6sXfakydBUvhHtb74e9taaxlWlMDew4Wsb/F8QPy70trHbAv4AHg\nrvaez83N1d5YsmSJV68PFeHSD62lL6HIq35seE3r+xO1Ltzs3useG6H1Wz/2/LztaLMvc/tq/eHd\n3h34jR9qPW+8d8doT/lh8ztc9cxJD3vzdwHWaBdisF9TOkqpeKVUgvN74Hxgiz/PKYTwowOrISrB\n3EHrjoxhgZma2VALtWU+Sun4KYffXCkz/FI6mcC7ytw+HAG8qrX+xM/nFEL4y4HVkJULFqt7r0sf\nBvlfQlOj+691R3NZBQ/vsnWK6+GoyeOH9tY6KmWGWw5fa70HGOPPcwCUVtfx5093kGhvYES5nYzE\nwBYkEqJbqKuGo9/AlDvdf236UGiwQ+l+SM3xfducPF3LtrXYZLO1l52Yl+8rQVrAHAJz0dbv9hRV\n8cGGQ1TUNjB/478ZkBbPGQNSOT07lYk5qWSlxAW7iUJ0fYc3g26CPuPdf61zps6x7X4O+F4WTnOK\nTTHbmhLfB3x7cBYwhzAJ+OP7pbDh/vN5+YMvqE/J4as9xSzaVMhrqw8A0CsphgnZqZyenUJu/xSG\n9UzEaglslTohurzCDWbbe5z7r22uqbMdhl7ouza15m3hNKcYxwi/ptS747RFRvjes1oU2UlWpk8b\nwM3TBtDYpNlxuILVe4tZs6+E1XuL+WDjIQBs0RGM65fM+H4pjO+fwrh+ySTGRAa5B0KEuEPrwZYJ\nCb0637e12GRI6O3/EgvOwmme1tFxaIhOIgLILzhIlWUQabZoMn2VKnauZxsT2OUNIYwCfmtWi2JE\n70RG9E7kxsk5aK0pKKlh7b4S1uw7zpr8Ev72xU6atLmxbnCGzbwB9EthfP9kBqTZsMinACFOOLQe\neo31vIZ7+lD/z9SpOmYWFYnyfL7/0XI7v38vn78Dj7+/ivebrFgtirmXj2LW6f28b6OM8P1PKUXf\n1Dj6psZx2ThTtKjCXs/GA2Ws3VfCuv0lfLS5kAVfmzRQQkwEY/smM65vMmP7JTMmK5ketuhgdkGI\n4KmtNKPzEZd5foz0obDuZXOnqb8W/nDeZevh8TcXlHHzP9cQUaPACreekcp3B+byyqp9/OrtzRwq\ntfOLcwejvGm/vQwiYsEa+KxCtwn4bUmIiWTK4DSmDDb5vqYmzZ6iKtbvL2H9gVLW7SvhiSW7aNJm\n/76psYzOSmZMVhKjs5IZ2TuRBEkFie7g8GZAe5a/d0ruD/VVZoFwV+vwuKvqmMf5+yXbj/KTV9bS\nIz6aZ+acC8/BiOQmRozqyXeGZ3DfO5v56793crjMzh+vOM3zDECQ6uhANw/4rVksikEZNgZl2Lhq\nQl8Aqmob2HywjA0HStlUUMqG/aV8uKmw+TUD0uIZ2SeJEb1M+mh4rwTSbdHejQCECDWH1ptt77Ge\nHyPZkQ4p209NpLkoGhvl4znuVcfMdQY3HS6z84vXNzAw3cZLN00kzRYNkfFmLj4QabXwyJWjyUyM\n5u9LdjO2XzLXTPQwvROkOjogAb9T8dERnDmgB2cOODEiKaqsZXNBGZsPmq+1+cebLwgDpMRFMjgz\ngcEZNgak2xiQFk92Wjy9k2OIjvDjTSdC+Muh9eZibUJPj15eUFLNO+tquR34+VPv8UH9IaKsFi4d\n25sfTc1hWE8fBcCqos6XXWxFa83db22krqGJJ64db4I9mAvNNSXN+ymluOv8oazJL+Hhj7dz3ojM\nE/u6Q0b4XUuaLZoZwzKYMezETIDS6jq2FVawrbCcnUcr+PZIJR9sPES5/eQa4OkJ0fRJjiUjIZr0\nhGjSbNEkx0WSGBNJYmwksZFWdpY00qOgDIsFFAqloElrGho1DU2a+sYm6hqaqG1wbhubv69vbKKu\nsYnGRt18TqUgKsJCdISVmEhL87mSYiPJTIyhR3yUXKAWHTu03qN0ztFyO/O+2MnrXx8giSZuj4RL\nsxsZNmAoB0treGddAW+uLeDsIek8dvUYzwKok9YepXReXrmPZTuLePCyUeSkxZ94IjbllGmZSike\nunwUF/51GXM/3Mbjszz4xBOkWvggAd9nkuOimDSwB5MGnvgkoLWmpLqevUWV7C2q5mBJDQdLqzlU\namdfcTVr9pVwvKqu7QOu+jJALYcIiyIzMYaslFj694ijf494BmXYGJxho19qHBFWWTahW7OXQ/FO\nGH21Wy/7ancxP39tHWU19Vw9oS8/mz4Q5idybq9azp0xCIB7LhjKK6v287cvdnL1/K94+cdn0Cc5\n1sN2lkFjHcS7PiVz19FK5n60jelD07nujFYpmpiTR/hOgzISuGXaQJ5YsosrJ2Rx1kA3rxnUVng9\nbdRTEvD9SClFanwUqfGp5PZv+269hsYmKuwNlNvrKa9pwN7QyOo16xk6YhSNWqO1eeOwWBQRFoXV\nooiyWoiOtBBltRIVYSEm0ozeI62KqAgLkVYLVovCOWZv0lDn+FRQU99IeU09ZTX1lFbXcaS8liPl\ndgrL7Ow/Xs0X249RVFnQ3L4oq4VBGTaG9UxgWK8EhvVMZHivRNITZMZSt3F4k9m6OMLXWvPcl3v5\n48fb6d8jjtduPpPBmY4URlJfKDvQvG9yXBQ/mzGIiTmp3PTi11z51Ape/tEZDMrwYFqlm3fZaq25\n751NxERa+fP3R5963S02GY7vafO1t50ziPc3HuK3C7fw8R1T3UvV1lYEZQ4+SMAPugirhZT4KFLi\no5ofq8q3Mn2E+xeeOhIVYQFHjO5sBFVZ28Duo5XsPFrJziMVbD9cwfLdRbyz/mDzPmm2KIY4rlMM\nzkxgYLqNnLR4MhKiJT0UbpwXbHt1nr6obWjkV29tYuGGQ5w/IpPHrh5z8ky25H6mnk4rp2en8vqc\nSfzg+dVc/fRXvPOTs8humV5xhZt32X6wqZCv80v44xWntV1/K7btET5ATKSV3186kv964WveWFPA\nDWf2d72dteWSwxehwxYdwZi+yYzpm3zS48er6th+uJzthRVsP1zOjiOVvLW2gKq6xuZ9YiItZKXE\n0Ssphp6JMdSU1LHTsofkuEgSYiKJibQQG2k+mShlPoVozCed+kZzfcJe30hNfSP2+kaq65xfDWZb\n20i14zl7fSO19eaaRUNTEw2Njk9EmOsXFqWaP+04zxsbZSU+KgJbTAQJMZEkx0aSEh9JSlwUqfFR\npNmi6WGLCu7Fda2hvhrsZejYFIrsFg6W1nCk3E5xZR1FlbVU1jZQ32j6bLUoR/sjyUiMYUSvRLJS\nYn03U+zQekjMAlvHI+eymnpueXkNK/cc587zhnDbjEGnvvkn94V9y9t8/Yjeibx56yQuf3I5t7y8\nlnd/dhZxUW6EKDcKp9XUNfLHj7YxsnciVztm5J2ijRx+S9OHpJPbP4Wnluxi1oS+ZlDVmaYmx0Vb\nyeGLEJcaH8VZA9NOyllqrTlYWsPeoiryi6vJL6riUGkNhWV2vj1yjKKKehbt8X5pO6tFEecI2HFR\nVmIinV8WEiIjiLRaiLAoLC2CXJM+cZG7tqGJ4qo6qksaqaptoMLeQGVt+4tqJ8ZEkJ4QTUZCjGMb\nTcWxekqSCkizRTe/MaTERRHpxTWO+sYmE8QP7iJyy5tk5L9HQs0BIrRpW5WOJa/xdN5rmsyKppE0\nOZawiI20EmE1ab6GRk1Fq74kxUYyOiuJGUMzOG9EJn1TvSggeGh9p9MxD5bWcOPzq8kvruL/Zo1t\nvrnxFMn9zAi3pvRERcoWctLimTd7HD98YTX3vr2Zv84e6/oblxsBf/7S3RSW2fnr7HHt19WKSYaG\nGqi3Q+SpnwCUUtz+ncH88PnVvL2uwLVpmnWVgJYRvuialFJkpcSRlRLH1MGnPr9kyRJyJ02htKqe\ncns9tQ2N2OvN9QSNbr7pMsJiIcJqrk/EOAJ7TKSV+CjzfZTV4vN7GxqbNBX2eo5X1VFSXUdxZR3F\nVXUUVdRSVFnL0Qrztf5ACccqarHXN/H6jo2nHCchOoKkuEhs0RHYoiOIi44gymoh0qqIsFpo0ppG\nxwyr6roG84ZT28DxqjqyanZwd8QbnG01efKVTcP5xnIRltgUom3JjNQ7ubR0KVc1/IeaHqOouOJf\nJGf2P2U0Wd/YRGl1PQUl1WwtLOebQ+V8vfc4/7toK/+7aCsjeiVy1YQsrhiX5d4vqfq4yWOPva7d\nXZwXZ2sbmnjppokdX8R0zsUv3d9mwAeYNiSdu84fyiOf7mBs32RumuJidU1nDj+u45u6DpbWMH/p\nbi4e3YuJOR1UwnRWzLSXQmTb01GnDU5jTN9k/r5kF1fmZnX+5u8sqyDz8EU4UkqZaaAheEey1aJI\njosiOS6q03211ny8OI+hY0+nqKKW4qo6iitrKamup6S6jtLqeiprTTAvq6mn3jFFtqFJY1HmXFaL\nhfgoK8lxUYy2lXNlxAuMafoMe2QKe4bejh49i+F9h3JmbKvfVX0NbH2P2A/vInbBd+Ha16HX6JN2\nibRaSHdM9R3XL6X58fyiKhZvO8L7Gw/x+w+28vDH28nNsJA8sJSxfdsOuCcp+Nps+57R5u/kH8v2\n8vAn28nuEcfTN+QyKKOTkWux7+d0AAAgAElEQVSSI31SduCUPrT0k7MHsuFAKQ99tI0xfZPanfRw\nkqqjZlQe0fHfc+5H5hPnfd/tZNUu5xtSTUm79x8opbjjO4O46cU1vLv+YPvpISdn4TQZ4QsRupRS\nxEUqBqbbGJjuxULcjfWwYh4s/bP5ecqdxEz5BQM6mrURGQtjZkPP0+CVq+H5mXDVizDk/E5Pl50W\nz4+nDuDHUwew5WAZC77ez1tf7+eyvy9nfL9kbpycw8yRPdvPP+9fCcoKfXJPfri4mv9dtJXF245w\n4aiePHLVGGzRLoSTZMfFzTYu3LZksSgeu3oMF81bxu2vbeCjO6aS1PqNsLWqY51Od1y28xgfbirk\nl+cO6Xz6Z3NN/I5LJM8YmsGoPok8uWQXV4zr0/E05ubCaWE6S0cpNRP4K2AF/qG1ftjf5+w26qrM\nGp7RCUEpxOQXNaVQuNHkjYt2muXgaitMP2OSTQ0WWyZkjIBeYyB1IFi6yH0CB76GD+4wq0YN/x7M\nfBiS3EixZI6Em/8Nr86CN34Ac/LMWrEuGtUniQf7nMZkWxGFMdm89FU+t7+2nh7xUVwxvg+zTu97\n6gj9wGozEo8y1wDKqut5YslOXlqxD6tF8duLhvOjKTmup9viUiEyDkoPdLprYkwk82aP46r5X/Gb\ndzfzt2vGdXweZ+G0dtQ2NPK7974hJy2eW84e0HlbnTXx7R0HfKUUt58zmDkvr+X9jYe4YnwHf1N7\nGI/wlVJW4O/AeUAB8LVS6n2t9VZ/njcs2cthTx7sWgxHtkDJPqhuschyRCwk9YH+kyF7Kgw4O2g3\nd7it8ih8sxC2vAUHVp14PKGXGWVF2SAiGsoKzBtBdRE0OS5SRtmg/1kw8Dsw8BxIG+y/SoyeqimF\nf/8vrHkeEnvD7Ndg2Hc9O1ZCT7j2DZg/Gd76L7j5C/MJwA2xEYqbpuTww7Oy+c+3x3j96wO8sDyf\nZ5ftpV9qHGfkmJXi0uMtTC1Yw5HBs3kvbzdf7jrG1/kl1Dc2ceX4LP77/KH0THKzRrxSjqmZ+1za\nfVy/FH553hAe+XQH04akd5wyqTrW4eLqzyzdw96iKv5500RiIl2YhdVy1atOnDcik+G9Ennii11c\nOrZP+xeCm2vhh2cOfyKwy7G2LUqpBcClgAR8VzTWw46PYO1LsHepCXLRieYGmGEXQUp/U+CptsKM\nhIt2mcC57iWwRJhR5MRboN+ZoRcEAQo3wbLHYNv7Zum8jJEw/deQNcH0sb2l5RrrTV31wk1wcK15\nI9z5mXmuxyAYcSkMv8R8Aghmv7WGLW/DJ/eZN6kzfwIzfu396C4hEy6fD//6Pnz6G7j4cY8OY7Wo\n5hIhxypq+XDTIVbsLubzbUd4c20Bo9VupkfbeXBTIh9t2M6wngnccGZ/vj8+ixG9vQhYrW6+6syt\nZw/ky51F3P/eN4zvl9z+dYKqYxA/rc2n9hdX88SSXVw0uhfThri4/GFzDr/zVa+cufxb/7WODzYe\nan+WUpBz+Epr3flenh5cqSuBmVrrHzt+vgE4Q2t9W4t95gBzADIzM3MXLFjg8fkqKyux2bzIr4aI\n2pJChpYuoVfhZ0TXlWCPTudoxlSKe0ygPHEo2tLB+7RuxFa5l8wjy+h5+HMiG6qosA1k98AfUpri\n9/XkT9HW38RWsYucva/S4/haGqxxHOo9kyOZ06myuXHzSisxNUdIPb6W9GNfkVy6BUUT1bG9OJox\nlaMZ06iO7+Rimgvc+feVWLaNgbtfIKl8B+UJg/h2yE+pTBjodRtaGrD7BfodWMiWkb+iKP0sl1/X\nWT+atOZwlSan8AMmHXyefw57lsSUNJKjfZM6G/ztfDKOfsnyKf9y+TUl9ibuX1FDbITif86MxRZl\n3sidfVFNDZz9n++zN/sa9mXPPqU/j62xs7u0iT9OjSUlxsV+6CbOXnoF+/pfRX5O+7OUWp7nd8tr\naNAwd0rsSVOEnbIOLGTQ7hdYNuU1GiNOnirrTfyaMWPGWq31hE531Fr77Qu4EpO3d/58A/BEe/vn\n5uZqbyxZssSr1wddaYHWH9+rG36frvX9SVr/6yqtd3yidWODZ8errdT66+e0fnyU1vcnav3K1Vof\n3eHbNnfipL9JeaHW7/7EtOXhbK2X/lnr6hLfn7SySOs1L2r94ve0fiDZnO/Js7Re+ojWxbs9PqxL\n/74Ortd6wXXmnI8M0XrNC57//TpTX6v109O1fri/1tXHXX6Zy/9PXv+B+bfja8v+Yn4/9nK3XrYm\nv1gP/vVH+tpnv9J1DY1a6xZ9KS80x1z97Cmvm/vRVt3/V4v0q6v2ud/WP/bT+sO7XN590cZDuv+v\nFumF6wva3uHfD5r/242NpzzlTfwC1mgXYrK/r3YdBFoOrbIcj4mWyg/Bh3fBvLGw6mmOpZ8FP1sF\n170BQy4Ai4d3fUbFw4Sb4Lav4dwHIH85PDUJFj8AddU+7EAnGurgy7/A33Jh85sw+Q64YyNMu7vd\nudheie8BuT+EH74Pd24zF0cj4+CLP8C8cTB/Ciz5IxzaYNIu3mqsh22L4IXvwjNnw+4lJjV1+zrI\nvdHzv19nIqLgknkm5fCfR317bK3N9ZS+E317XDB324JLF25byu2fytwrTmP5rmL+sKhVVridm64W\nrj/I00v3cP2Z/TyrX99BeYW2XDiqJ0Mybfzti100NrXxb8tZGjlIEw38ncP/GhislMrBBPrZwLV+\nPmfXUX4Ivvw/WPsi6EZzc8vUO9m+MZ+e6UN9d57IGJjySxh7PSy+3wTfb96Fix6DQef67jxtSCr9\nBp7+lcm5D/0uXPAQpLowQ8JXEnqa3PmZPzFTAb9ZaK6LLP0TLH3YVFbMnmwudvedCGlDOr8IqjVU\nFJqAuP0j2PmpqdSY1A/OfwjG3xC44lg9T4Nx18Gqp+H0H/nud1t2wPSx35m+OV5LLadmZo5w66VX\n5max80gFT/9nD9ERFs6IdQTVSsfi5S0C/qaCUn719ibOyEnl/u+N9KytnZRXaM1iUfz8nMH8/LX1\nvLv+IFfmtpqxE8Q6OuDngK+1blBK3QZ8ipmW+bzW+ht/nrNLKDtogu66f5oLsWOvhWl3QUq2Y4d8\n/5zXlg6XPWnmdC/6pbnoN/IKuGAuJPby7bkqjsDiBxi38VUTCK9ZAEMv9O053JXcDybfbr6qiuDb\nT83F8PwvzRsggLKYv0NKtpmWF5tsFsWurWBEwV7Y+4iZVukc9cWmwrDvwfCLYdB5YA3CrS0zfgtb\n3jWf3K7+p2+OeWC12fpjhN/y5isP3DNzGNV1jTy7bC9LUyyMyrXTs7lSZgYNjU28uno/j366gzRb\nNE9eN97z8hftlEjuyEWn9eKF5Xt58MOtTB+afnKN/3YC/uKtRyiqavKsjW7w+79OrfVHwEf+Pk+X\ncHQbrHgCNr0OaBPop9wJqS7eOu4rOdPg1uWw/P9g2eOw83M457dmhOjtfP6GOlg139xY1GBnX7/v\n0//6v5n0UiiJTzMj43HXmRF7ST4UbjALdR/dZoJRyT4zB7uhDqLisTVaIDrLzALKGGnmp/eZEJwg\n31JiL5Mmy5sL+76C/pO8P+b+lWbKa4aHI+OO2DLMm6iLUzNbs1oUf7hsFBOyU7jnzQ1cNG8Zc3tu\n5gLgvV11zF+1nG2F5Zw1sAdzLz+NHt4sqhKb4vYbk8Wi+NP3R3PRvC954P1veOLa8SeebGPxk93H\nKrl9wXoGJcHsizxvqivkTlt/a2wwUwbXPGfm0EfGmbzuWT830yqDJTIGpt8Lp10FH90Fn/wKVj0F\n0+8zj7mbd26ohY2vmU8uJfkwZCZcMJe9mw/QP9SCfWtKmTfdTt54V+flMX369MC0yV1n3WZSg5/9\nBn78b++nox5YZe6u9cebmVLmhjM3c/itXTq2D1UFO1hYEMuBgv3UYeWOhXvonRTLk9eN58JRPb2v\nv+RmDt9pcGYCPz9nEI99/i2Xjj3Cec5y57UVJ+b3A/b6Rm57dT3RERb+a6T/w7EEfH/QGo5uhc1v\nwYZXofKwuTv0nN/ChB+1P788GHoMhOvfMemNLx6Ed28xo/7cG2HUFZ2vYVqSD1vfg5XzoeKQmT//\n3Udh8HmOHbz7Ty1cFBUP5/wG3vuZuUYxzIuhor3M3Nw39S7fta+1duriu6u3zcIbt05CL3wZvSuD\nT34yjewe8a7dWOUKZw7fWeXPDbecPZAPNxfy24WbmZiTakpD1JafNNCb+9E2thWW8/yNE7Ac9r6q\nbGck4PtKXTUUrIbdX8C2D0yFQWWBwefD+B+YbaiWP1AKhs40bdz2Pnz5OHx6H3z6a8ieYkrjpmSb\nXHx9NVQeMR9zd+fBkc3mGP3OgkufMHe7huJNXt3B6NnmRrYlc2HIhZ7PBPn2M3MjnD8v6Cf3MzfO\n+YiqKkLZ0n23GLpTTLKZUFFb4fbdsVERFv585Wguf3IFVzy5nL/OHseoFguYf7KlkH9+tY8fTcnh\nnGGZ5EnAD1G1lVC8y4yCDm+BQ+vg4Dpoqjd3uOacDWfdbkZZXaW8AZgAMfIy83XsW1PqYNsiWPUM\nNNa22jfC5K/Pf9D0M5Azb0TbrBFw9r3w7hzzxj3yMs+Os+09U9Yi63Tftq+lpL7m7uO66uY6PV6p\nOuby0oZuaVki2YNyCKOzkvnnTRP57zc2ctnfl7MtppRvi+H+p1awZl8Jo7OS+NVM1+sheat7B3yt\nzYIEtRWOr0qoc3xvLzNf1cfNP6aqIpOyKNkHNcdPHCMiFnqOgkk/M6PhvmcErU6GT6UPMWUAZvza\nrNJTedh8BI+Kd9S4Se06Rcu6k9OuhGWPQt4fTWkNd6/F1FXBzsUw7nr//n2dM3XKD5r6R96qOmam\n1PpayxLJyR7M4wcmD0rjk19M5X/e3UDkzlo+3VVNZXoDd18wlGsm9nNtpSwfCc+AX283wal0n/kq\nLzQBq/IoVBebIF5TYvJpupOpUMpqZnTEp5vReq+x5g+fOgAyR5kLff66sSZUWCym6Fdi72C3RHTG\nYjUX49+6yUw1Pe1K916/a7FZ5Wn49/zTPidnldCyAu8DvtaOEb5ra9m6xcUSyZ1Jjoti3mUD4BG4\nfsZo7jy37Zo//hYeAb90P6x5nlHblsPGX5gLibS4y01ZTbC2ZZp/FKkDzB8yJsnk06ITzFSp6AQz\nFS06wbyzxyRBVPDuihPCIyMuhwzHKH/EZe7NtNn6vvn01n+y/9oHJwd8b9VVQoPdP+nTmBYjfC8p\nexkAGemZXh/LU+ER8GsrYMXfiI3pDdnjzY1FqQPMHX3J/cw/hHAfhQvhZLHAjN/A69eZyqmn/8i1\n1zXUmtlaI918k/BEYm9A+Sbgu7GWrdta5vC95Qj4AbsLuw3hEfDTh8NvDvP1suWhO09aiEAadpEZ\npS+Za9I6rgSZPXnmGtaIS/3ePKyRZsqvTwK+8y5bfwR8343wmwN+dPCu8YVHrsJiCd0pj0IEg1Km\nblF1kZmq6Yqt75tglBOg/HJSlsflFU7SPML3Qw4/Mg6sUV7n8IGQGOGHR8AXQpyq9zgYcw2sfMpx\nXasDjfWw40Nzh3SEF6UI3JGUFfopHaU8qqfTpubVriTgCyH84Zz/MZMWFj/Q8X4rnzJB7bSrAtIs\nwAT88oPel6iudAT8OD+M8MHk8cMkhy8BX4hwltTHFFb75l3Y8Frb+xTthCUPwdCLWpTECETb+prZ\nNdXF3h2n6phJRUW6ub6uqzysp3MKe5m5+z4qeKvyhcdFWyFE+6b+N+z/Ct6/zZTIbvnfvqnR1N+J\niDFr4wayLEbz1MwD3uXf/XWXrVNsilm7wlv2MvPGFMRp3jLCFyLcRUTBrH+Z2Wyv/wBbxa4Tz616\n2lTGvPBPnRfK87VEx0Lf3ubx/R3wY5J9d9E2iOkckBG+EN1DTCJc/xb84zzGbvgdHHzJjO6Ld5oL\ntaNnBb5NzQuheBvwi0zVV3/xZQ4/yGVX/DbCV0o9oJQ6qJTa4Pj6rr/OJYRwQUJPuOFdjqeONaPr\n1BwT6L83LzgVTuNSTS2qUB/hxyabGTaNDd4dx15+4s7dIPH3CP8vWmsfr64shPBY2iC2jryHjFC4\nQdG5EIo3c/F1o7no6+8cPphRvjfXGuxlgV/drhXJ4Qshgicpy6zx7KGoulJA+7cMeXMBNS9n6oRA\nDt/fAf82pdQmpdTzSqmUzncXQnQrXt58FV1bfOI4/hLrWKEuDAK+VykdpdRioK1L+78BngL+gClb\n+QfgMeCmNo4xB5gDkJmZSV5ensftqays9Or1oSJc+gHSl1AUSv3oX9JATuVhln7xOdrifnkUm+PT\nwZpvD1FZmOfj1hkJ5XvIBTavWkrx7mqPjqGaGjm7roK9h0vY187vPiB/F62137+AbGBLZ/vl5uZq\nbyxZssSr14eKcOmH1tKXUBRS/Vj3stb3J2pdvMejl3/78n+b11cc9XHDWijebc6x/lXPj1FVbI7x\n1ZPt7uLN3wVYo12Ixf6cpdOrxY+XA1v8dS4hRBflZV386NpiU9wsrocPG9VKcw7/eMf7dcRZRyeI\nlTLBv7N0/qyUGotJ6eQDt/jxXEKIrsjLufjRtUVmyU1/3r0anWRKIniTww+BOjrgx4Cvtb7BX8cW\nQoQJ57KZHgf845Dk56U3LRYzf77aixF+iAR8mZYphAieyFgzh77cixF+INZajksNixG+BHwhRHAl\n9vFshK+1yeEHIuDHpniXw5eAL4QQeD4Xv6YEi66HhEAEfG9H+MFf/AQk4Ashgi2prwn47i6EUu64\nQzdQI/xqb1M6KuizdCTgCyGCKykL6irdH0E7a9Q7yyz7ky9y+NEJQa2FDxLwhRDBlpJttp2tu9ta\nQEf4qVBXAQ11nr0+BMoqgAR8IUSwOStIlux173XlhWgsYMv0fZtai3WUNfa0Lr4EfCGE4MQI//ge\n915Xfoi6qGSwBmAdpzhHATVP5+JLwBdCCCAq3twte9zdEf5BaqP9WFKhJW9LJNdKwBdCCCN1gPsj\n/IpCaqO9WJDEHc0lkmWEL4QQ3knN8SilUxud6p/2tObtCF8CvhBCOKTkQOURqKtybX97OdSWB26E\n700Ov6nJtDfIc/BBAr4QIhSkDjBbV/P4FYUAgcvhR9nAEuHZCL+uAtAywhdCCKBFwHcxreO46Spg\nI3ylHOUVPBjhh0gdHZCAL4QIBc65+G4H/ADl8MFRQM2DEX6I1NEBCfhCiFAQkwRxaW4H/LqoAKV0\nwOTxPcnhywhfCCFaSc1x/W7bikMQ14Mma5R/29RSbArUeHCnbbgEfKXUVUqpb5RSTUqpCa2eu08p\ntUsptUMpdYF3zRRChL3UAa5ftC0/FJgaOi1JDp8twBXAf1o+qJQaAcwGRgIzgSeVUlYvzyWECGep\nA0yZ5Hp75/uWHwxMHfyWYpM9zOGHScDXWm/TWu9o46lLgQVa61qt9V5gFzDRm3MJIcJc6gBAQ+m+\nzvcNxgg/LhXqq117Q2qp1nHRNgTm4fur6lAfYGWLnwscj51CKTUHmAOQmZlJXl6exyetrKz06vWh\nIlz6AdKXUBSq/UgoLyEX2Lz0PYrT2h8fWhrrmFZdzN7iOipV4PrS69AxhgIrvviQOjfm/w/ctYVe\n1hi+XPZlh/sF4u/SacBXSi0Gerbx1G+01u952wCt9TPAMwATJkzQ06dP9/hYeXl5ePP6UBEu/QDp\nSygK2X5UnQbr7uG0PjaYNL39/Y59C8sgZ+xU9pXaAteXb0rh26c4a+wwyBzp+uvK3oTyHp22MxB/\nl04Dvtb6XA+OexDo2+LnLMdjQgjRtrhUiE7qfGrm4U1m23MUlBb7v11OntbTCZE6OuC/aZnvA7OV\nUtFKqRxgMLDaT+cSQoQDpVwrola4EaxRkD4sMO1y8rSeTrgEfKXU5UqpAmAS8KFS6lMArfU3wBvA\nVuAT4Gda60ZvGyuECHOulEk+vAkyhoM1MjBtcvJmhB8CF2zB+1k672qts7TW0VrrTK31BS2ee0hr\nPVBrPVRr/bH3TRVChL3UAVC6Hxrr235eayjcBD1HB7Zd4HlNfHt5eIzwhRDCp1IHgG5sf0Hz8oMm\n4PYaE9BmARAZC9ZoyeELIYRP9B5ntvtXtv18ofOCbRBG+Eq5X09Hawn4QgjRpozhpoha/rK2nz+8\nCVDuTYv0JXcrZtZVmU8sMWGQwxdCCJ9SCrKnwN5lZnTcWuEm6DEIom2Bbxs46um4EfCri8w2LkB1\n+zshAV8IEVpypppqmG3N1jm8CXoFIZ3j5G49ncpjZhuf7p/2uEkCvhAitGRPM9vWaZ3q41B2IDj5\neyd3c/hVR83WJgFfCCFOlTYYbJkmrdOS8w7boI7wHTn8ttJNbal0BPz4DP+1yQ0S8IUQoUUpyJ5q\nRvgtA2vzDJ0gTMl0ik2FxlpTNdMVVY4cvqR0hBCiHTlTofIIFO088djhTZDYB+IDuKxha+6WV6g6\nCjHJEBHAlbk6IAFfCBF6sqeabX6LtZWCdYdtS87UjDM335nKo2ALjXQOSMAXQoSi1AFmNO/M49dW\nQPHO4ObvARIcleIrDru2f9WxkEnngAR8IUQoas7jfwlf/R2eOB10k5mjH0wJvcy2otC1/SuPSsAX\nQohO5Uw1Ny59+mtzs9UP3oOcacFtU3waKKsbI/zQSun4a4lDIYTwzohLoXgXDP0u9A2RJbEtVhPA\nXRnhN9SaOjohMiUTJOALIUJVdAKc+0CwW3GqhJ6ujfCdUzJD5KYrkJSOEEK4J6GXiwE/tG66Au9X\nvLpKKfWNUqpJKTWhxePZSqkapdQGx9d875sqhBAhIKGnaymdEKujA96ndLYAVwBPt/Hcbq31WC+P\nL4QQoSWhF1QXQ0NdxzdUhVgdHfB+icNtWusdvmqMEEKEPOdc/MojHe8XYnV0wL85/Byl1Hql1FKl\n1FQ/nkcIIQKneS5+J3n8qmMQZYOoOP+3yUVKd1L1TSm1GOjZxlO/0Vq/59gnD7hLa73G8XM0YNNa\nFyulcoGFwEitdXkbx58DzAHIzMzMXbBggcedqaysxGYL0sIIPhQu/QDpSygKl35AcPpiq9jDhLW/\nZMvIeylKn9TufsO3PkZi+besOrOtjPepvOnLjBkz1mqtJ3S6o9ba6y8gD5jg6fPOr9zcXO2NJUuW\nePX6UBEu/dBa+hKKwqUfWgepLxVHtb4/UeuVT3e834vf0/of57l8WG/6AqzRLsRqv6R0lFLpSimr\n4/sBwGCgjeVrhBCii4nrAZaIzmfqhFgdHfB+WublSqkCYBLwoVLqU8dT04BNSqkNwFvArVprN5aJ\nEUKIEGWxgM2Fm69CrI4OeDktU2v9LvBuG4+/DbztzbGFECJkdTYXv7HBTN0MoTo6IHfaCiGE+xJ6\ndjwts7oY0CE3wpeAL4QQ7kro1fEIv/mmKxnhCyFE15bQ0yxmXm9v+/kQvOkKJOALIYT7mu+2befC\nbYgtXu4kAV8IIdzV2VKHIVhHByTgCyGE+zpb6rDyKFijIToxcG1ygQR8IYRwV2f1dKqOmQu2SgWu\nTS6QgC+EEO6KTQFrVMcj/BDL34MEfCGEcJ9SHS91GGKLlztJwBdCCE90tNRh5TGITwtse1wgAV8I\nITzR3gi/qQmqi0JuDj5IwBdCCM+0V0DNXgpNDZLSEUKIsJHQE2rLoK7q5MePOyrBJ/YJfJs6IQFf\nCCE80d7UzH3LzbbvGYFtjwsk4AshhCdSB5jtkS0nP75vBfQYBAmZgW9TJyTgCyGEJ/rkQnQS7Pz8\nxGNNjbDvK+g/OXjt6oAEfCGE8IQ1AgbOgF2LwazdDUe+MXn9cAz4SqlHlFLblVKblFLvKqWSWzx3\nn1Jql1Jqh1LqAu+bKoQQIWbweeZuW2daZ98Ks80Ow4APfA6M0lqPBr4F7gNQSo0AZgMjgZnAk85F\nzYUQImwMOtdsd35mtvu+hOR+kJQVvDZ1wKuAr7X+TGvd4PhxJeDs5aXAAq11rdZ6L7ALmOjNuYQQ\nIuQk9ISeo2GnI62zbwX0nxLsVrXLq0XMW7kJeN3xfR/MG4BTgeOxUyil5gBzADIzM8nLy/O4AZWV\nlV69PlSESz9A+hKKwqUfEBp9yYkaQr/977D+/fmMry5mu70Hhz1oUyD60mnAV0otBnq28dRvtNbv\nOfb5DdAAvOJuA7TWzwDPAEyYMEFPnz7d3UM0y8vLw5vXh4pw6QdIX0JRuPQDQqQvA2Lg+bcYX/Ih\nAMMuuIlhzimbbghEXzoN+Frrczt6Xil1I3Ax8B2tnZeqOQj0bbFbluMxIYQIL30mQEwS5C8zN2Ol\n5AS7Re3ydpbOTOAe4BKtdXWLp94HZiulopVSOcBgYLU35xJCiJBkjYCB3zHf958ccouetOTtLJ0n\ngATgc6XUBqXUfACt9TfAG8BW4BPgZ1rrRi/PJYQQoWnweWbb/6zgtqMTXl201VoP6uC5h4CHvDm+\nEEJ0CcMvgcNbYOTlwW5Jh3w5S0cIIbqnaBvMnBvsVnRKSisIIUQ3IQFfCCG6CQn4QgjRTUjAF0KI\nbkICvhBCdBMS8IUQopuQgC+EEN2EBHwhhOgm1Il6Z8GnlDoG7PPiEGlAkY+aE0zh0g+QvoSicOkH\nSF+c+mut0zvbKaQCvreUUmu01hOC3Q5vhUs/QPoSisKlHyB9cZekdIQQopuQgC+EEN1EuAX8Z4Ld\nAB8Jl36A9CUUhUs/QPrilrDK4QshhGhfuI3whRBCtKPLB3yl1PNKqaNKqS3Bbou3lFJ9lVJLlFJb\nlVLfKKXuCHabPKWUilFKrVZKbXT05ffBbpM3lFJWpdR6pdSiYLfFG0qpfKXUZscKdWuC3R5vKKWS\nlVJvKaW2K6W2KaUmBbtNnlBKDXX8PZxf5UqpX/jlXF09paOUmgZUAv/UWo8Kdnu8oZTqBfTSWq9T\nSiUAa4HLtNZbg9w0tymlFBCvta5USkUCXwJ3aK1XBrlpHlFK3QlMABK11hcHuz2eUkrlAxO01l1+\n7rpS6iVgmdb6H0qpKPeKp4AAAAJESURBVCBOa10a7HZ5QyllBQ4CZ2itvbknqU1dfoSvtf4PcDzY\n7fAFrXWh1nqd4/sKYBvQJ7it8ow2Kh0/Rjq+uuToQimVBVwE/CPYbRGGUioJmAY8B6C1ruvqwd7h\nO8BufwR7CIOAH66UUtnAOGBVcFviOUcaZANwFPhca91V+/J/wD1AU7Ab4gMa+EwptVYpNSfYjfFC\nDnAMeMGRavuHUio+2I3ygdnAa/46uAT8EKSUsgFvA7/QWpcHuz2e0lo3aq3HAlnARKVUl0u5KaUu\nBo5qrdcGuy0+MkVrPR64EPiZIyXaFUUA44GntNbjgCrg3uA2yTuOtNQlwJv+OocE/BDjyHe/Dbyi\ntX4n2O3xBcdH7SXAzGC3xQOTgUscue8FwDlKqX8Ft0me01ofdGyPAu8CE4PbIo8VAAUtPjW+hXkD\n6MouBNZprY/46wQS8EOI40Lnc8A2rfXjwW6PN5RS6UqpZMf3scB5wPbgtsp9Wuv7tNZZWutszMft\nL7TW1we5WR5RSsU7JgPgSH+cD3TJ2W1a68PAAaXUUMdD3wG63OSGVq7Bj+kcMB+LujSl1GvAdCBN\nKVUA3K+1fi64rfLYZOAGYLMj9w3wa631R0Fsk6d6AS85Zh1YgDe01l16SmMYyATeNeMKIoBXtdaf\nBLdJXvk58IojFbIH+K8gt8djjjfg84Bb/Hqerj4tUwghhGskpSOEEN2EBHwhhOgmJOALIUQ3IQFf\nCCG6CQn4QgjRTUjAF0KIbkICvhBCdBMS8IUQopv4f+i7zgW+v+L9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plotting your derivative\n",
    "scalar_space = np.linspace(1, 7, 100)\n",
    "\n",
    "y = [s.run(weird_psychotic_function, {my_scalar:x, my_vector:[1, 2, 3]})\n",
    "     for x in scalar_space]\n",
    "\n",
    "plt.plot(scalar_space, y, label='function')\n",
    "\n",
    "y_der_by_scalar = [s.run(der_by_scalar, {my_scalar:x, my_vector:[1, 2, 3]})\n",
    "     for x in scalar_space]\n",
    "\n",
    "plt.plot(scalar_space, y_der_by_scalar, label='derivative')\n",
    "plt.grid()\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Almost done - optimizers\n",
    "\n",
    "While you can perform gradient descent by hand with automatic grads from above, tensorflow also has some optimization methods implemented for you. Recall momentum & rmsprop?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_guess = tf.Variable(np.zeros(2,dtype='float32'))\n",
    "y_true = tf.range(1,3,dtype='float32')\n",
    "\n",
    "loss = tf.reduce_mean((y_guess - y_true + tf.random_normal([2]))**2) \n",
    "\n",
    "optimizer = tf.train.MomentumOptimizer(0.01,0.9).minimize(loss,var_list=y_guess)\n",
    "\n",
    "#same, but more detailed:\n",
    "#updates = [[tf.gradients(loss,y_guess)[0], y_guess]]\n",
    "#optimizer = tf.train.MomentumOptimizer(0.01,0.9).apply_gradients(updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd81fW9x/HX95wMNoQ9E0CWKCgk\nYnDXVbetG2mrdWCdvc7eaqe299qhtVZvLVLraFAE98BZFRXCOEAgEEgYORBICBkkgZBxzvneP87J\nMYSEHDDJGXk/Hw8e5HfOL+d8OJy8zzff33cYay0iIhJbHOEuQERE2p7CXUQkBincRURikMJdRCQG\nKdxFRGKQwl1EJAYp3EVEYpDCXUQkBincRURiUFy4nrh///525MiR4Xp6EZGo5HK5Sqy1A1o7L2zh\nPnLkSFasWBGupxcRiUrGGHco56lbRkQkBincRURikMJdRCQGKdxFRGKQwl1EJAYp3EVEYpDCXUQi\nkstdztOfbcLlLg93KVEpbOPcRUSa43KXk5Hp5u2snXh9lsR4Bxk3pZOakhTu0qKKwl1EIoLLXc5T\n/8nl840lNN7Zuabex+0ZLu46axzXnpgctvqijcJdRMLO5S5nxuxM6ry+Zu8vqqzlwTfWAijgQ6Q+\ndxEJu8wtpdQ3CXaHOfi8t7N2dFBF0U8tdxEJu/TR/XA6DB7fNx0yg3slsquqlsaZv6aggj9/tJEu\ncQ6mH9Vf/fCHoHAXkbBLTUni4UuP5VdvZQcDfmdFLXEOw7SRfaj1+Dh93AA+WFfEU//ZBECcI4+n\nZ07lu8cMDmfpEUvhLiIR4doTkxk/uCdPfJLLV3n+i6oenyXe6eBn5x9NakoSCXEOHvsoN3jfLS+5\nOHVsf65KG8E5EwexbmclmVtKSR/dr9O36o21tvWz2kFaWprVkr8i0pTLXc7MOZnU1vuCo2a6BIZD\nAsyck0m9x0ec08Glxw/l602l7Nizn+4JTmo8Pqy1JMTF7vBJY4zLWpvW2nlquYtIRElNSSLjpnSe\n+CSXL/NKAKit97Fkcwl3nDmWjJvSD2id+3yWxZtL+f1768kpqgL8wyf/+kkef7xiMjv27O+UrXm1\n3EUkIjVtwU8a2ouzJw7ilLEDmg1pl7uca5/NpM7jAwMN0eYIfJ0Q52DuzdHfmg+15a5wF5GI5XKX\nk7mlhOwdlSzMLgIg8RAh7T/f30rv1z2Bn7+xliWbS4P39++RwMXHDWVQry7sr/Ny2rjmPygimcJd\nRGLG059t4s8fbgz2wZ8+bgDPXX8CzuYGwzfS0Pqv8/hwGMPEIb3IKaqk3ut/pEN9UEQq9bmLSMxI\nH92PxHgH9R5/F80Xubu5/O+LueHkkWwv399if3pD/33jPvcnP83liU/y8Fmo9fh4YfHWmOyTV8td\nRKJCsMtlVF92VtTw4OtrqKr14jAc1uiYxq15X6N++WgZYRNqy13LD4hIVEhNSeL274whdWRfLj5u\nKCeM6geAz0K9x0fmltJWHuGbx8m4KZ17zx3P+ccOPqLHiAbqlhGRqOPKL+PzjcXBY6fTQfrofiF/\nf2pKEqkpSbjc5Xy4rgifhfi4w3uMSKeWu4hEDZe7nD98sIE7Xl4V7FIxwBWpw4+oOyU1JYn7vzse\ngLvPHhfxXTKHQ+EuIlHBlV/GNbOX8PfPN1NYUYPTYXAaSIx3cPnU4Uf8uDedOpreXeN5dUVBTO36\npG4ZEYloLnc5C9cW8s6ancEhjA4DV58wgmF9un7rUS5rCirYV+uhYn89M2Yv4eVZ02OiBa9wF5GI\ntWxrKdc+uzS4UqQzMN00Ps7fWm+LEM7cUoovMGqwzmuZs2gLmcN7R/3QSIW7iEQkn8/y23fWfxPs\nbdhabyx9dD8S4vxj6L0WFq4r4sP1RVEzNLIlCncRiTjWWh5+dz3rdlYS5zDYNm6tN9Z4olNOYSXv\nrin0T3Cq9/HaygKFu4hIW3C5y3nso40s3lzKDSeP4sJJg8ncWtau3SSNh0YuzC7E6wMLLFhR0C4f\nKB1B4S4iEWN5fhkzZmfi8VmcxnDBpMGkjuxL6si+HfL8qSlJXJOWTMaybQDUeX28uCQ/Kpcn0FBI\nEYkIxZU13PtqVqN9VC1Lt5Z1eB2XpQ6nS7wjuEH3W6t38thHG5k5JzOqhkoq3EUk7L7I3c35f/2S\noor9xDv949fDNWO08fIElxw3BPAvT1BT7+OJT3KjJuDVLSMiYVPv9fH4x7n8/fPNjB/Uk1dmpVNZ\n4wl7N0jjPviP1u2ixuMD4Ku8Epbnl0XFKBqFu4h0OH9oFvHZxmJyd+1lxrRkfn3xRLrEOwEiJjhT\nU5LIuDmdv3ycy1eb/Jt2R8soGoW7iHQol7uca2YvCc42veeccdx11tgwV9Wy1JQk7j5nHEu3llLv\ntVhg3vJtGOCyCB5J02qfuzFmhDHmM2PMemPMOmPMT5s5xxhjnjTGbDLGrDHGTG2fckWk3WRkwMiR\n4HD4/87IaPOn2FNdxy/fXHvAMgKt7aYUCVJTkrgqbUTw2OuDjKXbIvoiaygXVD3AvdbaiUA6cLsx\nZmKTc84Hxgb+zAL+3qZVikj7ysiAWbPA7fbvJu12+4/bMOA/21DMuX9ZxMaiKuICi34lRNEyu5dN\n9Y+iafxRVBvBF1kPeycmY8xbwFPW2o8b3fYP4HNr7cuB443AGdbawpYeRzsxiUSQkSP9gd5USgrk\n53+rh66sqed3767n1RUFjB/Uk8euOo7awMYY0TZ23OUu57WVBcxfsT3424fBvzJlR11kbZc9VI0x\nI4EpwNImdw0Dtjc6LgjcdkC4G2Nm4W/Zk5ycfDhPLSLtadu2w7s9RF/llfDAgiyKKmu47Yyj+OnZ\nY0mMi6yLpoejYRTN5VOHB2fRWvzDJP/6SS7XnTSSDUVVB3xoBbcH7OAPspDD3RjTA3gN+C9rbeWR\nPJm1djYwG/wt9yN5DBFpB8nJzbfcj7AR9vWmEv704UZWb9/D6AHdee3Wk5iSHH1h3pLUlCTuPXc8\nKwN7sQIsyithUV4J4L+WcPq4AfTqGs97awrx+iwJcQ5+ffExZO+s6JCLsSGFuzEmHn+wZ1hrX2/m\nlB3AiEbHwwO3iUg0+P3v/X3s1dXf3Natm//2w/TPr7bwu3dzsPgvlv7+e8fGVLA3aLzgWProfry3\ntpB/fbUVi3/S04r8cqrrvXgDM25rPT4efGNt8Pvnuwp4+eb268oJZbSMAf4J5FhrH2/htLeBHwVG\nzaQDFYfqbxeR0Ljc5Tz92ab2v2A3cybMnu3vYzfG//fs2f7bQ1S2r467563mkUCwA2AtK7ftaZeS\nI0Fw0+6UJC6cNITEeAcOIM5h+PkFR/PIJce0+L3tvSF3KC33k4EfAmuNMasDtz0IJANYa58B3gcu\nADYB1cCP275Ukc7F5S5n5rOZ1Hh8OAycffQgbjn9qPb7VX7mzEOGeUt9x9Za3ly9g4ffWc/eWg9X\npg7jnaxC6r2+mNt0uuE1SEtJYlT/7pRV11G+r5491XWUVddx6tgBfJKzC4/P8tCba4lrMszTQPCD\nr71fm1bD3Vr7VaCmQ51jgdvbqigR8e8Q1DDt3Wfho/W7+M/GYuaFYRu4t1bt4J75Wfh89oCRIdvL\nqnnozWwW5e5mSnIfHr1sMuMH9+SaaSlRORqmJTv27OdfX23lua+3Bjfmbo210LNLHGX76oO3nTNx\nEP17JkZOn7uIdLykbgkH3ebx2g6Z+u5yl/PZhmJq6r0syy9jTUFF8L56j4/Fm0tYta2cxz7KxWHg\nt5ccww/SU4ITkhpGlUSrfbUe3s7ayTtZO9m5Zz/5pdUH3G+AsycO4nvHDyOpWzxJ3RNI6pbA1pK9\n/Pj55dR7/L+13HfuBH7zzrrgcbv+5tWEwl0kQpVX1zV7+6p26H9v6G44YWQSWdsr+N+FOcEWakq/\nblx/UgoZS7fh8VocDsMbK3ewpWQfZ00YyCPfO5ahfboe0fO11LL3+izL80tZuqWMU8b0b3U998Md\nbug/v4RxA3vRLdHJ5t172Vy8l82797F5914KK2qC5xoDP5qewgkjk7h/wZpgUP+kmaAe3LvLARdZ\nU1OSGD+4Z2QPhRSRjpU+uh+JcQ5qA10zDXKKqvi/z/I4fkQSq7bvCSk0moZfw/GJo/pSsreWu15e\nTb3XR9MeB4eBq9JGkD66H3OXbsMC9V5L6b46nrp2ChdOGoJ/zEXLzwVQXechd9deNhRWsihvNx9k\nFwU/PHokOjEY6n0+vD6Lx2dpPLfyL5/kEe80dEuIo0u8gy7xThLj/H93iXNS6/GydkcFPuuvd/Lw\nPnRPdFLvsdR5fdR7fXi8lnqvjzqvj321Hsqr62mqR2IcRw3ozvTR/di9t5av8vwLhTmAQb26cPFx\nwxjap1urQd30t5Zw/RajcBeJUKkpScy9OZ37569mS8mB3QJ//DAX8HcPxDsNV6aNCPbhuvLL+Chn\nF/17JOLzWRZv9o+/bgjMft0TKNtXd1CQNzhtbH+W5ZcFW6hTk/vwt0/zqGs0I/O66SlcNHnoQd+7\nIr+MmXOWUufx4XQYUlOS2FVZg7usOvj88U4TDHYDjB3UkykjkohzGpwOQ7zD4HLvYfFmf7ga4PgR\nfZg4pBc19T5qPF5q6r3UenzU1HsprqoNPp7PQlHFfoYndSPe6aBnfBwJTgfxTgfxcQ7iHYa84r2U\nV1cEn/+qtOHcc+54BvZMDH5QudzlLG/0GjRc+Iym7qbDXn6grWj5AZHQ/HtJPr94a13wOMHp4NSx\n/fl0Q/EB5xn8reCqWu8Bt/dIdLI3cJsB+vdIYPfeuuDxCSOTWL29Aq/PH2QZN6UD8PWm3VTu9/c9\nF1fVBncminP4P0zOnzSErvEOcgqryCmsJKewkrU7KoLT8gGSusczfXQ/JgzuxYTBPTl6SC92Vdbw\ng38uDQZnc9P2Xe5yZs7JPOQ5R3Lu4ZwfrpmlrQl1+QGFu0iYNRciLnc5H64rYuvuvXySUxxswZ4z\n0T8cEmDmnExq6w/sShncK5FdlbX+7gQDd3xnDKePH3hAmP3qomN4+N11B4QbEKxh8vDeLHAV8LdP\n89hZUcO0UX2588wxbCutZr6rgDUFew4aMdKrSxwThvSif48EPl6/yz8j0+kgo4VJOqEE5+GE65H1\nuUdecIdC4S4SYeYu3cZzX20BY7jh5FFcfcIIPsgu5O55q6n3WpwOw2nj+lO2r56s7XsO6jZxGrjn\n3PHc/p0xwDeLWC1wFeD1thzcjfvYm/a5Nw43j9fHE5/m8dISNxX7/X3SEwb3pEu8k3U7D2yRg//D\n5vtThnHvd8cztHeXA7o0ojU4o4HCXSSCzF267YCp5+BvWTdtAXdPcNKzSxxFlbWAP0CdDoO19pBd\nGK0F96H4fJZ31xZy18urDrovMc7B5OG9mZqSRGqyv1/8toyVIXeBSNtrl1UhReTIzFt+8OqKjYPd\n4A/Shy6cSPbOCuav2I7H6w/031x8DOXVdS2G9ZGOzrDW8trKHdw3P+ug+wwwMz2FX100kYS4A1cp\naTrUTyKTwl2kAwzs1QWoaPF+C9Q0WVgKwOvzsb/eQ88ucbyTtZPy6jpOHzeAeKfjiPqt/V0521m3\no5KsggPrmXXaaF5cnB9cNuD7U4YdFOwQXSNGOjN1y4h0AJe7nKv/sRiPz98qvvT4oSzMLqLe6x8y\nOOvU0azdURFcMrY13RKc7K/zYvEPLfyvs8aCMcEhe0s2lxDncPD4J7nUB4Ylpo/uy1ebDlyoygAL\nbp3O1OQkjDHqL48C6nMXiTCt9Y03DNGr8/jw2cAY9jgHZ4zzL0bVcNtJR/Vnf73nW6+2aID7vvvN\nBVqJDupzF4kwrfWNN14fPKlbQrCfHWBR3u7gRcx7zh0HEBzeiDHBNcMbM/inzjfclRjnwOP10TDo\nJdZWbJQDqeUuEgVaGgvf8EHQMPzR6XSAtf6wNw0LzBoumjyE315yDJt37+O1lQUdsiqhtA91y4h0\nIg1BPzW5Dwuzi5i3fDu1Hh+XTx3OPeeOY9hhLuwlkUvdMiKdyPEj+rC1ZB/3vprFzooazhg/gJ+d\nN4Gjh/QKd2kSJgp3kShmreXz3N38YeEGNhRVMXl4b/581XGcdFT/cJcmYaZwF4lCLnc5b67awapt\n5WTvrCS5bzf+NsO/BK+jydZu0jkp3EWizFebSrjuuWXBETI3nTKKB86b0OyEI+m89G4QiSKLN5Vw\n279dwWB3GkjqnqBgl4Oo5S4SBfbVevjDBxt4cYmbIb27UOPxBVeC1Fh1aY7CXSTCLdtaxn3zs9he\nXs0NJ4/i/u+OZ31hpZYJkENSuItEqP11Xv704Ub+tXgrI5K68crN6ZwYhdu9SXgo3EUikMtdxn3z\n17C1ZB/XTU/hZ+dPoFuCflwldHq3iESQmnovj3+cy7NfbmFo767MvelEThqjMety+BTuIhFi1bZy\n7pufxebd+7j2xGQevOBoeiTqR1SOjN45ImGWuaWEv3ycx7KtZQzp3YUXb5jGaeMGhLssiXIKd5Ew\nmrd8G//92los4DSGP1wxmVPHKtjl29PMB5EwqKn38ujCDcFg97OsKWh5Kz6Rw6GWu0gHW7mtnPsD\nfetnHz2Qr/JKgvuWakKStBWFu0gHaRgJM+fLLQzu1YUXbpjG6eMGaN9SaRcKd5EO4HKXcf/8NWwp\n8Y+E+fn5E+jZJR7QhCRpH62GuzHmOeAioNhae2wz958BvAVsDdz0urX24bYsUiRa7a/z8uePNvLc\n11sZ2rsr/77xRE4Zq3Hr0v5Cabk/DzwFvHiIc7601l7UJhWJxIhlW8t4YEEW+aXV/DDdP8tU49al\no7T6TrPWLjLGjGz/UkSin8tdzpd5u8ndVcXC7CKGJ3Vl7s0namck6XBt1YyYbozJAnYC91lr17XR\n44pEDZe7nBmzM6nz+gC4YNJg/nTFcXRXa13CoC3edSuBFGvtXmPMBcCbwNjmTjTGzAJmASQnJ7fB\nU4tEjjlfbgkGu8PAMUN7K9glbL71JCZrbaW1dm/g6/eBeGNMs7+DWmtnW2vTrLVpAwZoFp7EBmst\nT36ax8LsIhzGvztSgsasS5h962aFMWYwsMtaa40x0/B/YJR+68pEokC918cv38zmleXbuWzqMK5O\nG8EKd7nGrEvYhTIU8mXgDKC/MaYA+DUQD2CtfQa4ArjVGOMB9gPXWGttCw8nEjP21Xq4LWMlX+Tu\n5s4zx3DPOeMwxgQ31BAJp1BGy8xo5f6n8A+VFOk0iqtquOH55eQUVvG/l01ixjRdQ5LIoqs9Iodp\nU3EV1z23nPLqOub8KI3vTBgY7pJEDqJwFzkMS7eUcvOLK0iIczJv1nQmDe8d7pJEmqVwFwmBy13O\n819vZWF2ESn9uvH8j6cxom+3cJcl0iKFu0grlueXMWN2Jh6fxWHgVxdNVLBLxNNmHSKHUFixn7vn\nrcbj8w8AM0D2zsrwFiUSArXcRVrwQXYRP3ttDTX1XuKdBp/PakMNiRoKd5Em9td5eeS99cxduo1J\nw3rz5IwplO2r04YaElUU7iKN5BRWctfLq8gr3sstp43m3nPHkxDnYFT/7gp1iSoKdxH868O8sDif\n/1m4gd5d43npxmmcOlbrH0n0UrhLp1e6t5YHFqzh0w3FnDlhIH+8YjL9eySGuyyRb0XhLp3aV3kl\n3PPqavZU1/Priydy/UkjMcaEuyyRb03hLp3S0i2lPPbxRpZtLeeoAd15/sfTmDi0V7jLEmkzCnfp\ndD7N2cVNL67AWnA6DI9ceqyCXWKOJjFJp1JYsZ/7FmQRXJTaWlZt3xPWmkTag8JdOo1tpdVc+cwS\naup8JDgdOA2alCQxS90y0ilsKt7LD+YspcbjZd4t6dR7rSYlSUxTuEvMyyms5AdzlmIMvDIrnQmD\n/f3rCnWJZQp3iWmrt+/huueW0S3BScZNJzJ6QI9wlyTSIRTuErOWbinlxhdW0Ld7Ahk3nahleqVT\n0QVViUmLcndz3b+WMahXIq/eMl3BLp2OWu4Scz5aV8Qdc1dx1MAevHTjNC0lIJ2Swl1iyhOf5PLX\nT/MYM6AHr9ycTu9u8eEuSSQs1C0jMWF/nZc75q7kiU/ysBa2l1WzaffecJclEjYKd4l6Lnc5Fz75\nJe+uKQzeVu/1kbmlNIxViYSXwl2iVk29l/99P4crn1lMrcfHby6eSJd4zTwVAfW5S5TK2r6He+dn\nsal4LzOmJfPgBRPo2SWeScP7aOapCAp3iTK1Hi9PfprHM19sYUCPRF64YRqnj/tmx6TUlCSFuggK\nd4ki2TsquG9+FhuKqrgydTi/uGgivbtqNIxIcxTuEvHqvT6e/mwTT/1nE327J/DP69I46+hB4S5L\nJKIp3CWi5RRWcu+rWawvrOT7U4bx64sn0qdbQrjLEol4CneJOC53OYs3l1C4Zz/zXQX07hrPP36Y\nynePGRzu0kSihsJdIorLXc61z2ZS6/EBcPKY/vxtxhT6dldrXeRwaJy7RAyfz/LUZ5uCwe4wcNJR\n/RTsIkeg1XA3xjxnjCk2xmS3cL8xxjxpjNlkjFljjJna9mVKrCsor+baOZl8tqEYhwGngQRNRBI5\nYqF0yzwPPAW82ML95wNjA39OBP4e+FukVdZaFrgK+O0767HW8sfLJ3PUgO5kbi3TRCSRb6HVcLfW\nLjLGjDzEKZcCL1prLZBpjOljjBlirS08xPeIULK3lp+/vpaP1+9i2qi+PHblccF111NH9g1zdSLR\nrS0uqA4Dtjc6LgjcpnCXFn2QXcRDb6ylqsbDQxcczY2njMLhMOEuSyRmdOhoGWPMLGAWQHJyckc+\ntUSIypp6fvv2el5bWcAxQ3sx9+bjGT+4Z7jLEok5bRHuO4ARjY6HB247iLV2NjAbIC0tzbbBc0sU\nWbyphPvmZ7GrqpY7zxzDnWeOJSFOA7ZE2kNbhPvbwB3GmFfwX0itUH+7NHC5y/kqbze5u6p4b20R\no/p3Z8FPpjMlWRdKRdpTq+FujHkZOAPob4wpAH4NxANYa58B3gcuADYB1cCP26tYiS4udzkzns2k\nLjBu/fxjB/P4VcfTNcEZ5spEYl8oo2VmtHK/BW5vs4okZjy7aHMw2B0Gjh3WW8Eu0kG0/IC0uTqP\nj4ffXccH63bhMGDQzkgiHU3hLm2quKqG2zNWsjy/nFmnjebsoweyPL9cE5JEOpjCXdrM6u17+MlL\nLvbsr+Ov1xzPpccPA2DaKLXYRTqawl3axKvLt/OLN7MZ2CuR1249iWOG9g53SSKdmsJdvpV6r49H\n3l3Pi0vcnDymH0/NmEqSVnEUCTuFuxyx3VW13J6xkmX5Zdx86ih+dt4E4pyalCQSCRTuckSytu/h\nlmb610UkMijc5bC43OXM+XILH+fsYlDPLupfF4lQCncJ2bKtpcx4dilen8Vh4HffP1bBLhKh1EEq\nISmurOHueavx+vzrvRlg/c7K8BYlIi1Sy11a5XKXceu/V7Knup54p8Hns5pxKhLhFO7SImstL2W6\nefid9QxP6sqLN05jX62XzC2lmnEqEuEU7tKsmnovD76xltdX7uCsCQN5/Orj6d01HkChLhIFFO5y\nkO1l1dzykoucokruPnscd545RlvgiUQZhbsc4Ivc3dz18iqstfzzujTOnDAo3CWJyBFQuAsAPp/l\n/z7fxGMf5zJ+UE+e+UEqI/t3D3dZInKEFO5CZU09976axcfrd3HJcUN59PJJdEvQW0MkmuknuJN7\na9UOfvPOOir21/PLiyZyw8kjMUb96yLRTuHeiT3/9VZ+8856ABKcDo4f0UfBLhIjNEO1k/oidzeP\nvJcTPPb6fGRuKQ1jRSLSlhTundCnObu4+YUVjEjqSmKcA6fRHqcisUbdMp3Mh+uKuGPuSiYM7sVL\nN05j8+59mnEqEoMU7p3Ie2sK+ekrqzh2WG9euGEavbvGk5qSoFAXiUHqlukk3lq9gztfXsnxI/rw\n0o3TgksJiEhsUsu9E1jgKuCBBVmcMLIvz11/At0T9d8uEuv0Ux7jXlm2jZ+/sZaTj+rPsz9Ko2uC\nM9wliUgHULjHKJe7nL9/sZlP1u/i9HED+McPU+kSr2AX6SwU7jHIlV/G1bMz8QS2w7v1jNEKdpFO\nRhdUY0xVTT0PvpGNp9F2eC73nvAWJSIdTi33GLJuZwW3Z6xkW1k1cQ6DtdoOT6SzUrjHAGstc5dt\n47fvrKdvtwTm3TIdhzGanCTSiSnco9zeWg8Pvr6Wt7N2ctq4AfzlquPo1yMR0HZ4Ip2Zwj2K5RRW\ncnvGSvJL93HfueO47QxthycifiFdUDXGnGeM2WiM2WSM+e9m7r/eGLPbGLM68Oemti9VGlhrmbd8\nG997+muqaj1k3JTOHWeOVbCLSFCrLXdjjBN4GjgHKACWG2Pettaub3LqPGvtHe1QozRSXefhF29k\n8/qqHZw8ph9PXD2FAT0Tw12WiESYULplpgGbrLVbAIwxrwCXAk3DXdpZ7q4qbstYyebde7n77HHc\nceYYnGqti0gzQgn3YcD2RscFwInNnHe5MeY0IBe421q7vekJxphZwCyA5OTkw6+2k3K5y5nz5RY+\nzSmmV9c4/n3jiZw8pn+4yxKRCNZWk5jeAUZaaycDHwMvNHeStXa2tTbNWps2YMCANnrq2Pb1phKu\nemYJC7OL8Ph8PHrZZAW7iLQqlHDfAYxodDw8cFuQtbbUWlsbOJwDpLZNeZ3b8vwybstYidd+M9t0\n466q8BYlIlEhlHBfDow1xowyxiQA1wBvNz7BGDOk0eElQA5yxGo9Xh5duIGr/rGExDhDglNb4YnI\n4Wm1z91a6zHG3AF8CDiB56y164wxDwMrrLVvA3cZYy4BPEAZcH071hzT1u+s5J5XV7OhqIoZ00bw\n0IUT2VhUpdmmInJYjA38yt/R0tLS7IoVK8Ly3JHI67P8Y9Fm/vJxLr27JvDHKyZx5oRB4S5LRCKM\nMcZlrU1r7TzNUI0A7tJ93PNqFi53ORdMGszvvjeJvt0Twl2WiEQxhXsYNSz49fv3cnA6DE9cfTyX\nHj8UYzR2XUS+HYV7mOyqrOGBBWv4Inc3p4zpzx+vmMzQPl3DXZaIxAiFexi8k7WTX7yZTa3Hy28v\nOYYfpqdoXRgRaVMK9w70xcbHtJLtAAAJUklEQVRiHl24gZyiKo4b0YfHrzqOowb0CHdZIhKDFO4d\n5NlFW/j9+/7h/3EOw0MXTFCwi0i70R6q7ayypp4HFmQFgx38F1KX55eHsSoRiXUK93b0Zd5uzvvL\nIha4CrhsyjC6xGumqYh0DHXLtIN9tR7+5/0cMpZuY/SA7rx260lMSU5iZnqKZpqKSIdQuLexJZtL\nuX9BFjv27OemU0Zx33fH0yXeCfj3NFWoi0hHULi3kf11Xv7wwQaeX5xPSr9uvHrLdE4Y2TfcZYlI\nJ6VwbwMudxn3vppFfmk1101P4WfnT6Bbgl5aEQkfJdC3sGRzCY9/nMvy/HKGJ3Vl7s0nctJR2khD\nRMJP4X6E/p3p5pdvZmMBp8Pw6OWTFewiEjE0FPIw7a318Ku3svlFINgBsJas7XvCWZaIyAHUcj8M\nn28s5qE3stlZsZ8LJw3m05xi6r0+jVsXkYijcA9B+b46HnlvPa+v3MGYgT1Y8JOTSE1JwuUu17h1\nEYlICvdDsNayMLuIX72VzZ7qeu46cwy3nzmGxDiNWxeRyKZwb0FxZQ2/fCubD9ftYtKw3rx4w4lM\nHNor3GWJiIRE4d6EK7+MZ7/cyqK83Xh9lp+fP4EbTxlFnFPXnkUkeijcG1m4tpDb567EZ8Fh4MkZ\nU7ho8tBwlyUictgU7oDPZ8lY6ubhd9fjC4xvNIC7tDqsdYmIHKlOH+7u0n08sGANS7eWcdzw3mwo\nqsKj4Y0iEuU6bbh7fZbnF+fzpw83EO908MfLJ3Nl2nBWbtuj4Y0iEvU6Xbi73OW8t7aQrzeVsLGo\nijMnDOR/vj+Jwb27ABreKCKxoVOF+9ItpcycsxRPoGP9p2eN4b/OHocxJsyViYi0rU4R7tZa/rOh\nmPvmZwWD3WkgIc6pYBeRmBTz4Z63q4pH3sthUe5uhvbpwr5aD16f1QVTEYlpMRvue6rreOKTPF7K\ndNMtwckvL5rIj6ansKagQhdMRSTmxVS4u9zlLN5cQsX+eha4CqjcX8+Macncc844+vVIBHTBVEQ6\nh5gJd5e7nBmzM6nz+gA4dmgv/nRzOkcP0XowItL5xMSCKV6f5c8fbggGu8PA+ZMGK9hFpNMKqeVu\njDkP+CvgBOZYax9tcn8i8CKQCpQCV1tr89u21IO53OV8mrOLL3KLWbezCqcxQMPFUm15JyKdV6vh\nboxxAk8D5wAFwHJjzNvW2vWNTrsRKLfWjjHGXAP8Abi6PQpu4O+GWUKd1z+08Y4zx/Cd8QN1sVRE\nhNBa7tOATdbaLQDGmFeAS4HG4X4p8JvA1wuAp4wxxlpraWMNrfX31xYGg91hoGu8UxdLRUQCQgn3\nYcD2RscFwIktnWOt9RhjKoB+QElbFNmgaWv9wG4YjVkXEWnQoaNljDGzgFkAycnJh/39mVtKgzNM\nHQaunjaCYX26qhtGRKSJUMJ9BzCi0fHwwG3NnVNgjIkDeuO/sHoAa+1sYDZAWlraYXfZpI/uR0Kc\ng3qPf0ney6cOV6iLiDQjlHBfDow1xozCH+LXANc2Oedt4DpgCXAF8J/26G9PTUki46Z0XTQVEWlF\nq+Ee6EO/A/gQ/1DI56y164wxDwMrrLVvA/8EXjLGbALK8H8AtAtdNBURaV1Ife7W2veB95vc9qtG\nX9cAV7ZtaSIicqRiYoaqiIgcSOEuIhKDFO4iIjFI4S4iEoMU7iIiMci0w3D00J7YmN2A+wi/vT9t\nvLRBO4umelVr+1Ct7aMz1ppirR3Q2klhC/dvwxizwlqbFu46QhVN9arW9qFa24dqbZm6ZUREYpDC\nXUQkBkVruM8OdwGHKZrqVa3tQ7W2D9XagqjscxcRkUOL1pa7iIgcQkSHuzHmPGPMRmPMJmPMfzdz\nf6IxZl7g/qXGmJEdX2WwltZqvccYs94Ys8YY86kxJiUcdQZqOWStjc673BhjjTFhG40QSq3GmKsC\nr+06Y8zcjq6xSS2tvQ+SjTGfGWNWBd4LF4SpzueMMcXGmOwW7jfGmCcD/441xpipHV1jo1paq3Vm\noMa1xpjFxpjjOrrGRrUcstZG551gjPEYY65ot2KstRH5B//ywpuB0UACkAVMbHLObcAzga+vAeZF\ncK3fAboFvr41kmsNnNcTWARkAmmRWiswFlgFJAWOB4aj1sOodzZwa+DriUB+mGo9DZgKZLdw/wXA\nQsAA6cDSML6urdV6UqP///MjudZG75P/4F9p94r2qiWSW+7BjbmttXVAw8bcjV0KvBD4egFwljHG\ndGCNDVqt1Vr7mbW2OnCYiX9Hq3AI5XUFeAT4A1DTkcU1EUqtNwNPW2vLAay1xR1cY2Oh1GuBXoGv\newM7O7C+b4qwdhH+vRdacinwovXLBPoYY4Z0THUHaq1Wa+3ihv9/wvuzFcrrCnAn8BrQru/VSA73\n5jbmHtbSOdZaD9CwMXdHC6XWxm7E3yoKh1ZrDfwKPsJa+15HFtaMUF7XccA4Y8zXxphMY8x5HVbd\nwUKp9zfAD4wxBfhbbnd2TGmH7XDf05EinD9brTLGDAO+D/y9vZ+rQzfIFjDG/ABIA04Pdy3NMcY4\ngMeB68NcSqji8HfNnIG/xbbIGDPJWrsnrFW1bAbwvLX2MWPMdPw7mB1rrfWFu7BoZ4z5Dv5wPyXc\ntRzCE8DPrLW+9u5kiORwb7ONuTtAKLVijDkbeAg43Vpb20G1NdVarT2BY4HPA2++wcDbxphLrLUr\nOqxKv1Be1wL8faz1wFZjTC7+sF/eMSUeIJR6bwTOA7DWLjHGdMG/5kg4u5OaE9J7OlIYYyYDc4Dz\nrbXhyIBQpQGvBH62+gMXGGM81to32/yZwnXhIYQLE3HAFmAU31ycOqbJObdz4AXVVyO41in4L7aN\njfTXtcn5nxO+C6qhvK7nAS8Evu6PvyuhXwTXuxC4PvD10fj73E2Y6h1JyxcpL+TAC6rLwlFjiLUm\nA5uAk8JZYyi1NjnvedrxgmrEttxthG3M3Qa1/gnoAcwPfGpvs9ZeEqG1RoQQa/0QONcYsx7wAvfb\nMLXcQqz3XuBZY8zd+C+uXm8DP+kdyRjzMv6urP6B/v9fA/GBf8cz+K8HXIA/NKuBH3d0jQ1CqPVX\n+K+1/V/gZ8tjw7SYWAi1dlwtYXhfiYhIO4vk0TIiInKEFO4iIjFI4S4iEoMU7iIiMUjhLiISgxTu\nIiIxSOEuIhKDFO4iIjHo/wEly2lkMyrRMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "s.run(tf.global_variables_initializer())\n",
    "\n",
    "guesses = [s.run(y_guess)]\n",
    "\n",
    "for _ in range(100):\n",
    "    s.run(optimizer)\n",
    "    guesses.append(s.run(y_guess))\n",
    "    \n",
    "    clear_output(True)\n",
    "    plt.plot(*zip(*guesses),marker='.')\n",
    "    plt.scatter(*s.run(y_true),c='red')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic regression example\n",
    "Implement the regular logistic regression training algorithm\n",
    "\n",
    "Tips:\n",
    "* Use a shared variable for weights\n",
    "* X and y are potential inputs\n",
    "* Compile 2 functions:\n",
    " * `train_function(X, y)` - returns error and computes weights' new values __(through updates)__\n",
    " * `predict_fun(X)` - just computes probabilities (\"y\") given data\n",
    " \n",
    " \n",
    "We shall train on a two-class MNIST dataset\n",
    "* please note that target `y` are `{0,1}` and not `{-1,1}` as in some formulae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y [shape - (360,)]: [0 1 0 1 0 1 0 0 1 1]\n",
      "X [shape - (360, 64)]:\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "mnist = load_digits(2)\n",
    "\n",
    "X,y = mnist.data, mnist.target\n",
    "\n",
    "print(\"y [shape - %s]:\" % (str(y.shape)), y[:10])\n",
    "print(\"X [shape - %s]:\" % (str(X.shape)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      " [[ 0.  0.  5. 13.  9.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0. 12. 13.  5.  0.  0.  0.  0.]\n",
      " [ 0.  0.  1.  9. 15. 11.  0.  0.  0.  0.]]\n",
      "y:\n",
      " [0 1 0 1 0 1 0 0 1 1]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f4bcbeeb2b0>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAAD8CAYAAABaQGkdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAACxNJREFUeJzt3fuLXPUZx/HPp5vErRqTYqxKNjSh\naEAqNZqmhIjQBEusokJL3YCWSmGhoCiGihZL239A0h+KIFErmBpsVBDrpVIVK6QxF1M1txKDJRvU\nRLwHTLLm6Q87gShp92zmnO+ZeXy/YHEvw36fQd45Z2ZnztcRIQA5fa3tAQA0h8CBxAgcSIzAgcQI\nHEiMwIHECBxIjMCBxAgcSGxKE790mk+JQZ3WxK9u1dissvfpnHPeL7bWvoMzi601OHqk2FpxZKzY\nWiV9poM6HIc80e0aCXxQp+n7XtbEr27Vez9eXHS9X61cW2yt32y+ptha59/2drG1xt55t9haJW2I\nv1e6HafoQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRWKXDby23vsr3b9h1NDwWgHhMGbntA0h8l\nXSHpAkkrbF/Q9GAAulflCL5I0u6I2BMRhyWtlVTudY0ATlqVwGdL2nvc16Od7wHocbW92cT2iKQR\nSRrUqXX9WgBdqHIE3ydpznFfD3W+9wURcW9ELIyIhVN1Sl3zAehClcA3SjrP9jzb0yQNS3qi2bEA\n1GHCU/SIGLN9k6RnJQ1Iuj8itjU+GYCuVXoMHhFPSXqq4VkA1IxXsgGJETiQGIEDiRE4kBiBA4kR\nOJAYgQOJETiQWCM7m2RVcqcRSRqe/kGxtVbN/LTYWn/d8myxtS753S+LrSVJs+5dX3S9iXAEBxIj\ncCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSq7Kzyf2299t+o8RAAOpT5Qj+J0nLG54DQAMmDDwi\nXpL0foFZANSMx+BAYmxdBCRW2xGcrYuA3sMpOpBYlT+TPSxpvaT5tkdt/6L5sQDUocreZCtKDAKg\nfpyiA4kROJAYgQOJETiQGIEDiRE4kBiBA4kROJBY329dNLb0kmJrDU/fWmwtSbpi+XCxtWa8trPY\nWj99eVmxtd5f8HmxtSRpVtHVJsYRHEiMwIHECBxIjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxKpc\ndHGO7Rdsb7e9zfYtJQYD0L0qr0Ufk7QyIrbYni5ps+3nImJ7w7MB6FKVvcnejogtnc8/kbRD0uym\nBwPQvUm9m8z2XEkLJG04wc/YugjoMZWfZLN9uqRHJd0aER9/+edsXQT0nkqB256q8bjXRMRjzY4E\noC5VnkW3pPsk7YiIu5sfCUBdqhzBl0i6QdJS21s7Hz9qeC4ANaiyN9nLklxgFgA145VsQGIEDiRG\n4EBiBA4kRuBAYgQOJEbgQGIEDiTW93uTfXZmubtw1/4Li60lSUcL7hdW0sbXv932CF8ZHMGBxAgc\nSIzAgcQIHEiMwIHECBxIjMCBxAgcSIzAgcSqXHRx0PYrtv/V2bro9yUGA9C9Kq/zPCRpaUR82rl8\n8su2n46IfzY8G4AuVbnoYkj6tPPl1M5HNDkUgHpU3fhgwPZWSfslPRcRJ9y6yPYm25uO6FDdcwI4\nCZUCj4jPI+IiSUOSFtn+zgluw9ZFQI+Z1LPoEfGhpBckLW9mHAB1qvIs+lm2Z3Y+/7qkyyXlfKMy\nkEyVZ9HPlfSg7QGN/4PwSEQ82exYAOpQ5Vn01zS+JziAPsMr2YDECBxIjMCBxAgcSIzAgcQIHEiM\nwIHECBxIrP+3LvpGuX+j1qxfXGwtSTpfrxRdr5QpMw4XW2vso2nF1upFHMGBxAgcSIzAgcQIHEiM\nwIHECBxIjMCBxAgcSIzAgcQqB965NvqrtrkeG9AnJnMEv0XSjqYGAVC/qjubDEm6UtLqZscBUKeq\nR/BVkm6XdLTBWQDUrMrGB1dJ2h8Rmye4HXuTAT2myhF8iaSrbb8laa2kpbYf+vKN2JsM6D0TBh4R\nd0bEUETMlTQs6fmIuL7xyQB0jb+DA4lN6oouEfGipBcbmQRA7TiCA4kROJAYgQOJETiQGIEDiRE4\nkBiBA4kROJBY329dNPhBuTe4fe/CN4utJUkfFVxryjlnF1vrugv+7/uWavXI05cWW6sXcQQHEiNw\nIDECBxIjcCAxAgcSI3AgMQIHEiNwIDECBxKr9Eq2zhVVP5H0uaSxiFjY5FAA6jGZl6r+ICLea2wS\nALXjFB1IrGrgIelvtjfbHmlyIAD1qXqKfmlE7LP9TUnP2d4ZES8df4NO+COSNKhTax4TwMmodASP\niH2d/+6X9LikRSe4DVsXAT2myuaDp9mefuxzST+U9EbTgwHoXpVT9LMlPW772O3/HBHPNDoVgFpM\nGHhE7JH03QKzAKgZfyYDEiNwIDECBxIjcCAxAgcSI3AgMQIHEiNwILG+37rojF3lNvj57dCTxdaS\npJ+N3FZsranXHii2Vknz7lzf9git4ggOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRWKXDb\nM22vs73T9g7bi5seDED3qr5U9Q+SnomIn9ieJnHhc6AfTBi47RmSLpP0c0mKiMOSDjc7FoA6VDlF\nnyfpgKQHbL9qe3Xn+ugAelyVwKdIuljSPRGxQNJBSXd8+Ua2R2xvsr3piA7VPCaAk1El8FFJoxGx\nofP1Oo0H/wVsXQT0ngkDj4h3JO21Pb/zrWWStjc6FYBaVH0W/WZJazrPoO+RdGNzIwGoS6XAI2Kr\npIUNzwKgZrySDUiMwIHECBxIjMCBxAgcSIzAgcQIHEiMwIHECBxIrO/3Jjv62s5ia113z8pia0nS\nXSsfLrbWqjeXFVtr40UDxdb6quMIDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kNmHgtufb\n3nrcx8e2by0xHIDuTPhS1YjYJekiSbI9IGmfpMcbngtADSZ7ir5M0psR8Z8mhgFQr8m+2WRY0gnf\nAWF7RNKIJA2y+SjQEyofwTubHlwt6S8n+jlbFwG9ZzKn6FdI2hIR7zY1DIB6TSbwFfofp+cAelOl\nwDv7gV8u6bFmxwFQp6p7kx2UdGbDswCoGa9kAxIjcCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxR0T9\nv9Q+IGmybymdJem92ofpDVnvG/erPd+KiLMmulEjgZ8M25siYmHbczQh633jfvU+TtGBxAgcSKyX\nAr+37QEalPW+cb96XM88BgdQv146ggOoWU8Ebnu57V22d9u+o+156mB7ju0XbG+3vc32LW3PVCfb\nA7Zftf1k27PUyfZM2+ts77S9w/bitmfqRuun6J1rrf9b41eMGZW0UdKKiNje6mBdsn2upHMjYovt\n6ZI2S7q23+/XMbZvk7RQ0hkRcVXb89TF9oOS/hERqzsXGj01Ij5se66T1QtH8EWSdkfEnog4LGmt\npGtanqlrEfF2RGzpfP6JpB2SZrc7VT1sD0m6UtLqtmepk+0Zki6TdJ8kRcThfo5b6o3AZ0vae9zX\no0oSwjG250paIGlDu5PUZpWk2yUdbXuQms2TdEDSA52HH6s71yPsW70QeGq2T5f0qKRbI+Ljtufp\nlu2rJO2PiM1tz9KAKZIulnRPRCyQdFBSXz8n1AuB75M057ivhzrf63u2p2o87jURkeWKtEskXW37\nLY0/nFpq+6F2R6rNqKTRiDh2prVO48H3rV4IfKOk82zP6zypMSzpiZZn6ppta/yx3I6IuLvteeoS\nEXdGxFBEzNX4/6vnI+L6lseqRUS8I2mv7fmdby2T1NdPik52b7LaRcSY7ZskPStpQNL9EbGt5bHq\nsETSDZJet721871fR8RTLc6Eid0saU3nYLNH0o0tz9OV1v9MBqA5vXCKDqAhBA4kRuBAYgQOJEbg\nQGIEDiRG4EBiBA4k9l/q9J/njqg1MAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('X:\\n',X[:3,:10])\n",
    "print('y:\\n',y[:10])\n",
    "plt.imshow(X[0].reshape([8,8]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# inputs and shareds\n",
    "weights = tf.Variable(np.zeros((X.shape[1],), dtype='float32')) #vetor linha -> uso de broadcast com x, ver teste abaixo\n",
    "input_X = tf.placeholder('float32', shape=(None, X.shape[1]))\n",
    "input_y = tf.placeholder('int32', shape=(None,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "[[1 1 1]\n",
      " [2 2 2]\n",
      " [3 3 3]\n",
      " [4 4 4]]\n",
      "[[ 1  2  3]\n",
      " [ 2  4  6]\n",
      " [ 3  6  9]\n",
      " [ 4  8 12]]\n"
     ]
    }
   ],
   "source": [
    "w = np.array([1,2,3])\n",
    "print(w)\n",
    "x = np.array([[1,1,1],[2,2,2],[3,3,3],[4,4,4]])\n",
    "print(x)\n",
    "print(w*x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10 20 30]\n",
      "[ 6 12 18 24]\n"
     ]
    }
   ],
   "source": [
    "print(np.sum(w*x, axis=0))\n",
    "print(np.sum(w*x, axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predicted_y = tf.math.sigmoid(tf.reduce_sum(-weights*input_X, axis=1))\n",
    "loss = tf.losses.log_loss(input_y, predicted_y)\n",
    "optimizer = tf.train.MomentumOptimizer(0.01,0.9).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train_function = lambda xx, yy: s.run([loss, weights, optimizer], feed_dict={input_X:xx, input_y: yy})\n",
    "predict_function = lambda xx: s.run(predicted_y, feed_dict={input_X:xx})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss at iter 0:0.6931\n",
      "train auc: 0.9998901642045143\n",
      "test auc: 1.0\n",
      "loss at iter 1:0.1377\n",
      "train auc: 1.0\n",
      "test auc: 1.0\n",
      "loss at iter 2:0.0336\n",
      "train auc: 1.0\n",
      "test auc: 1.0\n",
      "loss at iter 3:0.0190\n",
      "train auc: 1.0\n",
      "test auc: 1.0\n",
      "loss at iter 4:0.0127\n",
      "train auc: 1.0\n",
      "test auc: 1.0\n",
      "resulting weights:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATYAAAD8CAYAAAD9uIjPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFclJREFUeJzt3X+QXWd93/H3xyvJJsLyLzlGkUyt\nBpUZfrR22JqmJIyx/ENJXMRMzUQk8SgZU01mShompcEuLXScMjVNJ4SZUtKtLFCCiUMMFA3jWhUG\nDzBtHa1sR+BfkVBIvaqMLcuKjAqStfvpH/csXTa7umd97z279zmfF3NG9/x8njOWvjzPeX7JNhER\nJTlnsTMQEdFvCWwRUZwEtogoTgJbRBQngS0iipPAFhHFSWCLiOIksEVEcRLYIqI4ywbx0JFVK738\n0gsH8ehFtWxkstH0Vi3/QWNpHT/9isbSmjo5kL92c/KKZkfWaKSZ9F567jiTJ06ql2fc+LaVfv5Y\nvb/T+/af2m17Uy/pNWkgf8OWX3oh6/7drw/i0X+DevpPuzCrL/hec4kBN6x5srG0vvhXb2wsre/v\nu6SxtH6w7qXG0gJYfv6pRtKZuP0Pen7G88cm+bPdr6517ciaA6t7TrBBqYpGtJSBqZr/60bSJklP\nSToo6bY5zr9V0sOSzki6eda5rZIOVNvWfrxbc3WCiFhSjHnJvX9ekTQCfBy4HpgA9kraZfvxGZf9\nb+BXgffNuvdi4EPAKJ1Yu6+694Ve8pQSW0SL9anEdjVw0PYh26eBe4DNMy+w/R3b++FvPOxGYI/t\nY1Uw2wP0/C0vJbaIljJmsj/Tlq0Fnp6xPwG8uYd71/aaoQS2iBabonZgWy1pfMb+mO2xAWSpLxLY\nIlrKwGT9wHbU9ug85w4Dl8/YX1cdq+MwcM2sex+sm6n55BtbRItN4VpbF3uBDZLWS1oBbAF21czC\nbuAGSRdJugi4oTrWkwS2iJYy8JJdazvrc+wzwHvoBKQngM/afkzSHZLeDiDp70uaAN4J/GdJj1X3\nHgN+h05w3AvcUR3rSa2qqKRNwMeAEWC77Tt7TTgiFpfxQqqiZ3+WfR9w36xjH5zxey+dauZc9+4A\ndvQlI5Wuga1mH5WIGDaGyULXcqpTFe3aRyUihk9n5EG9bdjUqYr20kclIpYsMUmDg60b1LfuHpK2\nAdsAlq2+oF+PjYgB6TQetDew1eqjUnXWGwM47yfXFlpzjyhHpx9bewPbD/uo0AloW4BfGmiuIqIR\nU20tsdk+I2m6j8oIsMP2YwPPWUQMVNtLbHP2UYmI4WbEZKF99DNWNKLFWlsVjYgyGXHaI4udjYFI\nYItoqU4H3VRFI6IwrW48iIjy2GLSKbFFRGGmUmKLiJJ0Gg/KDAFlvlVEdJXGgyXsvHObW+n7vjfc\n3VhaAD+z99bG0vr+weYmLvi7Gw80ltb+p+ec2zAqk+nHFhElyciDiCjSVFpFI6IknUHwCWwRURAj\nXsqQqogoiU066EZEaZQOuhFRFpMSW0QUKI0HEVEUo/ZONClpB3AT8KztNww+SxHRhM7ye2WWbeqU\nQz8FbBpwPiKicZ0Fk+tsw6bOKlVfk3TF4LMSEU0yGXkQEQUaxtJYHX0LbJK2AdsAlq1ubqaIiHh5\nbBVbYuvbW9kesz1qe3Rk1cp+PTYiBqTTeDBSa+tG0iZJT0k6KOm2Oc6fK+lPqvMPTX/eknSFpO9L\nerTa/qAf75aqaERr9WfNA0kjwMeB64EJYK+kXbYfn3HZrcALtl8jaQvwEeAXq3Pftn1lzxmZoetb\nSfpj4H8Cr5U0Iam52Q8jYmA6jQeqtXVxNXDQ9iHbp4F7gM2zrtkM7Kx+3wtslDSwD3x1WkXfNajE\nI2Jx9WnkwVrg6Rn7E8Cb57vG9hlJfw1cUp1bL+kR4ATwr2x/vdcMpSoa0VILHHmwWtL4jP0x22N9\nyMYR4NW2n5f0JuC/Snq97RO9PDSBLaLFFrCYy1Hbo/OcOwxcPmN/XXVsrmsmJC0DLgCet23gFIDt\nfZK+DfwdYJwelNnWGxFd2fDS1Dm1ti72AhskrZe0AtgC7Jp1zS5ga/X7ZuArti3p0qrxAUl/G9gA\nHOr13VJii2ipTlW097JN9c3sPcBuYATYYfsxSXcA47Z3AXcBfyTpIHCMTvADeCtwh6SXgCng120f\n6zVPCWwRLdavkQe27wPum3XsgzN+/wB45xz3fQ74XF8yMUMCW0RLTXf3KFECW0RrlTukKoEtosWy\n5sEStfqVJxtL689Pv6KxtADW/IcVjaW17C8ONpbWG7/8fxpL6xy5sbQA9h/+iUbT60WnVTTL70VE\nQVo9NXhElCtV0YgoSlpFI6JIaRWNiKLY4kwCW0SUJlXRiChKvrFFRJES2CKiKOnHFhFFam0/NkmX\nA38IXEanWj5m+2ODzlhEDJYNZ7pPIjmU6pTYzgD/3PbDks4H9knaM2tprYgYQq2tito+QmfBBWy/\nKOkJOivOJLBFDLF8Y6tUqzdfBTw0x7ltwDaAZasv6EPWImLQXGhgq13BlvRKOlP4vneupbFsj9ke\ntT06smplP/MYEQMyhWptw6ZWiU3ScjpB7W7bnx9sliKiCXaLv7FVy9DfBTxh+/cGn6WIaIaYLLRV\ntM5bvQW4BbhW0qPV9vMDzldENMBWrW3Y1GkV/QYMYSU7Is4qY0UjojzufGcrUQJbRIsNY4tnHQls\nES3lghsPEtgiWixV0YgozjC2eNaRwBbRUnYCW0QUKN09IqI4+ca2RK1debyxtD7/wmhjaQEsP3ys\nsbTOPPdcY2k9cvzyxtLauuZ/NJYWwP7D72g0vV4YMdWnVlFJm4CPASPAdtt3zjp/Lp0Ja98EPA/8\nou3vVOduB24FJoF/Znt3r/kps603Impxze1sJI0AHwd+Dngd8C5Jr5t12a3AC7ZfA3wU+Eh17+uA\nLcDrgU3Af6qe15MEtoi2ct/Gil4NHLR9yPZp4B5g86xrNgM7q9/3AhurCTY2A/fYPmX7L4GD1fN6\nksAW0Wb1i2yrJY3P2LbNeMpa4OkZ+xPVMea6xvYZ4K+BS2reu2BD/40tIl6+BXT3OGq72Y/MPUhg\ni2gpA1NTfenucRiY2SK0rjo21zUTkpYBF9BpRKhz74KlKhrRVgasetvZ7QU2SFovaQWdxoBds67Z\nBWytft8MfMW2q+NbJJ0raT2wAfizXl8tJbaIFutHPzbbZyS9B9hNp7vHDtuPSboDGLe9i84s3H8k\n6SBwjE7wo7rus3RWvTsD/FPbk73mKYEtos361EHX9n3AfbOOfXDG7x8A75zn3g8DH+5PTjoS2CJa\nazin/a6jzmIu5wFfA86trr/X9ocGnbGIaECLh1SdAq61/b1qGb5vSPpvtv/XgPMWEYNkcH9aRZec\nOou5GPhetbu82gqN8xFtU2Zgq9XdQ9KIpEeBZ4E9th+a45pt072SJ0+c7Hc+I2IQ+jFYdAmqFdhs\nT9q+kk7nuaslvWGOa8Zsj9oeHVm1st/5jIhBaHNgm2b7OPBVOqPwI2KY9a+D7pLTNbBJulTShdXv\nVwDXA08OOmMRMXh2vW3Y1GkVXQPsrOZIOgf4rO0vDTZbEdGIFreK7geuaiAvEdEwDWFprI6MPIho\nqyFtGKgjgS2itYazYaCOBLaINkuJLSKKM7XYGRiMBLaItprux1agBLaIFkuraESUp9DAljUPIqI4\nQ19ie/0rjzSW1kXLmp215KlVr2k0vaasOOdMY2m9atnxxtIaRqmKRkRZTHuHVEVEwVJii4jSpCoa\nEeVJYIuI4iSwRURJ5FRFI6JEaRWNiNKkxBYR5Sk0sNUeUlWtLfqIpKx3EFEC///vbN22YbOQsaK/\nCTwxqIxExCJo87qiktYBvwBsH2x2IqJJmqq39ZSGdLGkPZIOVH9eNM91W6trDkjaOuP4g5KekvRo\ntf14tzTrlth+H/htip1vMyIG6DbgAdsbgAeq/R8h6WLgQ8CbgauBD80KgL9s+8pqe7ZbgnUWTL4J\neNb2vi7XbZM0Lml88kSzs2BExMvUTFV0M7Cz+r0TeMcc19wI7LF9zPYLwB5g08tNsE6J7S3A2yV9\nB7gHuFbSp2dfZHvM9qjt0ZFVK19ufiKiKc01Hlxme3p+sWeAy+a4Zi3w9Iz9ierYtE9W1dB/Lalr\n57s6CybfDtwOIOka4H22f6XbfRExBOoHrdWSxmfsj9kem96R9GXgVXPc94EfSc62tOBQ+cu2D0s6\nH/gccAvwh2e7If3YItqsfog5ant03sfY1813TtJ3Ja2xfUTSGmCub2SHgWtm7K8DHqyefbj680VJ\nn6HzDe6sgW1BU4PbftD2TQu5JyKWJtFMqyiwC5hu5dwKfHGOa3YDN0i6qGo0uAHYLWmZpNUAkpYD\nNwHf6pZg1jyIaKvmvrHdCVwv6QBwXbWPpFFJ2wFsHwN+B9hbbXdUx86lE+D2A4/SKdn9l24Jpioa\n0WYNdL61/TywcY7j48C7Z+zvAHbMuuYk8KaFppnAFtFmQziqoI4EtogWG8ZxoHUksEW0WQJbRBTF\nfWnxXJIS2CLaLCW2iChNvrEtUX9+Yl1jaX1m/VcbSwvg0+tXNZbW+Sde3Vhab7vkkcbSevLUTzSW\n1lBKYIuIogzpJJJ1JLBFtJRIVTQiCpTAFhHlSWCLiOIksEVEUYZ0ab06Etgi2iyBLSJKkyFVEVGc\nVldFqxWqXgQmgTNnm/s8IoZEOugC8DbbRweWk4hoXgJbRJSk5JEHdRdzMfDfJe2TtG2QGYqI5mjK\ntbZhU7fE9jPVgqU/DuyR9KTtr828oAp42wCWrb6gz9mMiL4r+BtbrRLbjAVLnwW+QGfB0tnXjNke\ntT06smplf3MZEQPR0PJ7jesa2CStrJaWR9JKOguZdl2wNCKGgGtuQ6ZOVfQy4AuSpq//jO37B5qr\niGjEMJbG6uga2GwfAv5eA3mJiKa1NbBFRKGySlVElKbkfmwJbBFt5jIjWwJbRIulxBYRZRnSrhx1\nJLBFtFipjQd1x4pGRIE0VW/rKQ3pYkl7JB2o/rxonuvul3Rc0pdmHV8v6SFJByX9iaQV3dJMYIto\nK9NpPKiz9eY24AHbG4AHqv25/C5wyxzHPwJ81PZrgBeAW7slOPRV0W8996rG0pq8otly+zNbTjWW\n1pF/uLaxtH6eRxpL69/vv6GxtADOnG7mn5StvjynocaDzcA11e+dwIPA+2dfZPsBSdfMPKbOkKdr\ngV+acf+/AT5xtgSHPrBFRA+aCWyX2T5S/X6GzjDNui4Bjts+U+1PAF3/XziBLaKlFthBd7Wk8Rn7\nY7bHfvgs6cvAXNWnD8zcsW1p8OXEBLaItvKCJpE8era1TmxfN985Sd+VtMb2EUlrgGcXkMvngQsl\nLatKbeuAw91uSuNBRJs1M23RLmBr9Xsr8MXa2bMNfBW4eSH3J7BFtFhDE03eCVwv6QBwXbWPpFFJ\n23+YF+nrwJ8CGyVNSLqxOvV+4LckHaTzze2ubgmmKhrRVgYaWM/A9vPAxjmOjwPvnrH/s/Pcf4g5\nZu0+mwS2iDbLkKqIKE0GwUdEcYZxab06ajUeSLpQ0r2SnpT0hKSfHnTGImLA6raIDmHsq1ti+xhw\nv+2bqwGoPzbAPEVEAzoddIcwatXQNbBJugB4K/CrALZPA6cHm62IaESLpy1aDzwHfFLSI5K2V+uL\nRsSQk11rGzZ1Atsy4KeAT9i+CjjJHNOOSNomaVzS+OSJk33OZkT0XcHf2OoEtglgwvZD1f69dALd\nj7A9ZnvU9ujIqhToIpa+zljROtuw6RrYbD8DPC3ptdWhjcDjA81VRDSjmYkmG1e3VfQ3gLurFtFD\nwK8NLksR0Yi2L5hs+1Fg3ilLImJIDWFprI6MPIhoszLjWgJbRJtpqsy6aAJbRFuZYjvoJrBFtJQY\nzs63dSSwRbRZAltEFCeBLSKKkm9sEVGitIpGRGGGc7hUHUMf2E6dWt5YWm/71j9uLC2Af/LGbzSW\n1qdW/IPG0vqPn/1HjaXl85r9h+sLJ5tJqB/JmAS2iChQmTXRBLaINks/togoTwJbRBTFhsky66IJ\nbBFtlhJbRBQngS0iimJgCNczqCOBLaK1DM43togoiSm28aDrKlWSXivp0RnbCUnvbSJzETFgDaxS\nJeliSXskHaj+vGie6+6XdFzSl2Yd/5Skv5wRg67slmad5feesn2l7SuBNwH/F/hCzXeKiKWsmeX3\nbgMesL0BeIA5Flyv/C5wyzzn/sV0HKoWlzqrOgsmz7QR+Lbtv1rgfRGx5NQMar0Hts3Azur3TuAd\nc+bGfgB4sdfEYOGBbQvwx3OdkLRN0rik8ckTJ3vPWUQMloGpqXobrJ7+911t2xaQ0mW2j1S/nwEu\nexm5/bCk/ZI+KuncbhfXbjyoFkt+O3D7XOdtjwFjAOf95Noy25AjSlO/NHbU9rxrC0v6MvCqOU59\n4EeTsyUtND7cTicgrqATY94P3HG2GxbSKvpzwMO2v7vATEXEktS/IVW2r5vvnKTvSlpj+4ikNcCz\nC3z2dGnvlKRPAu/rds9CqqLvYp5qaEQMIYM9VWvr0S5ga/V7K/DFhdxcBUMkic73uW91u6dWYJO0\nErge+PxCMhQRS9yU6229uRO4XtIB4LpqH0mjkrZPXyTp68CfAhslTUi6sTp1t6RvAt8EVgP/tluC\ntaqitk8ClyzkTSJiCDQwVtT283R6VMw+Pg68e8b+z85z/7ULTTMjDyLayp5u8SxOAltEm2V2j4go\ni/FkQ4vPNCyBLaKtMm1RRBQp0xZFREkMOCW2iCiKM9FkRBSo1MYDeQDNvZKeAxY6tdFq4GjfM7M0\nlPpuea/F87dsX9rLAyTdT+dd6zhqe1Mv6TVpIIHt5ZA0frbZA4ZZqe+W94qlaqHzsUVELHkJbBFR\nnKUU2MYWOwMDVOq75b1iSVoy39giIvplKZXYIiL6YkkENkmbJD0l6aCk+ZbmGiqSLpf0VUmPS3pM\n0m8udp76SdKIpEdmrwE57CRdKOleSU9KekLSTy92nmLhFr0qKmkE+As6M/ROAHuBd9l+fFEz1qNq\nOuM1th+WdD6wD3jHsL/XNEm/BYwCq2zftNj56RdJO4Gv295eLWD0Y7aPL3a+YmGWQontauCg7UO2\nTwP30FmHcKjZPmL74er3i8ATwNrFzVV/SFoH/AKwvdu1w0TSBcBbgbsAbJ9OUBtOSyGwrQWenrE/\nQSEBYJqkK4CrgIcWNyd98/vAbwOlDTRcDzwHfLKqZm+v1vuIIbMUAlvRJL0S+BzwXtsnFjs/vZJ0\nE/Cs7X2LnZcBWAb8FPAJ21cBJ4Eivvm2zVIIbIeBy2fsr6uODT1Jy+kEtbttl7LC11uAt0v6Dp3P\nBtdK+vTiZqlvJoAJ29Ml63vpBLoYMkshsO0FNkhaX32s3UJnHcKhVq2BeBfwhO3fW+z89Ivt222v\ns30Fnf9WX7H9K4ucrb6w/QzwtKTXVoc2AkU09rTNok9bZPuMpPcAu4ERYIftxxY5W/3wFuAW4JuS\nHq2O/Uvb9y1inqK736CzjuUK4BDwa4ucn3gZFr27R0REvy2FqmhERF8lsEVEcRLYIqI4CWwRUZwE\ntogoTgJbRBQngS0iipPAFhHF+X+lOqJj8RQE5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "s.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(5):\n",
    "    r = train_function(X_train,y_train)\n",
    "    loss_i = r[0]\n",
    "    \n",
    "    print(\"loss at iter %i:%.4f\" % (i, loss_i))\n",
    "    \n",
    "    print(\"train auc:\",roc_auc_score(y_train, predict_function(X_train)))\n",
    "    print(\"test auc:\",roc_auc_score(y_test, predict_function(X_test)))\n",
    "\n",
    "    \n",
    "print (\"resulting weights:\")\n",
    "#plt.imshow(shared_weights.get_value().reshape(8, -1))\n",
    "plt.imshow(weights.eval().reshape(8, -1))\n",
    "plt.colorbar();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: my1stNN\n",
    "Your ultimate task for this week is to build your first neural network [almost] from scratch and pure tensorflow.\n",
    "\n",
    "This time you will same digit recognition problem, but at a larger scale\n",
    "* images are now 28x28\n",
    "* 10 different digits\n",
    "* 50k samples\n",
    "\n",
    "Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) NN should already have ive you an edge over logistic regression.\n",
    "\n",
    "__[bonus score]__\n",
    "If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! The milestones would be 95%/97.5%/98.5% accuraсy on test set.\n",
    "\n",
    "__SPOILER!__\n",
    "At the end of the notebook you will find a few tips and frequently made mistakes. If you feel enough might to shoot yourself in the foot without external assistance, we encourage you to do so, but if you encounter any unsurpassable issues, please do look there before mailing us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 28, 28) (50000,)\n"
     ]
    }
   ],
   "source": [
    "from mnist import load_dataset\n",
    "\n",
    "#[down]loading the original MNIST dataset.\n",
    "#Please note that you should only train your NN on _train sample,\n",
    "# _val can be used to evaluate out-of-sample error, compare models or perform early-stopping\n",
    "# _test should be hidden under a rock untill final evaluation... But we both know it is near impossible to catch you evaluating on it.\n",
    "X_train,y_train,X_val,y_val,X_test,y_test = load_dataset()\n",
    "\n",
    "print (X_train.shape,y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Invalid dimensions for image data",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-112-b21c3ff9c032>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/pyplot.py\u001b[0m in \u001b[0;36mimshow\u001b[0;34m(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs)\u001b[0m\n\u001b[1;32m   2697\u001b[0m         \u001b[0mfilternorm\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfilternorm\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilterrad\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfilterrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mimlim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mimlim\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2698\u001b[0m         resample=resample, url=url, **({\"data\": data} if data is not\n\u001b[0;32m-> 2699\u001b[0;31m         None else {}), **kwargs)\n\u001b[0m\u001b[1;32m   2700\u001b[0m     \u001b[0msci\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__ret\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2701\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0m__ret\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/__init__.py\u001b[0m in \u001b[0;36minner\u001b[0;34m(ax, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1808\u001b[0m                         \u001b[0;34m\"the Matplotlib list!)\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlabel_namer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1809\u001b[0m                         RuntimeWarning, stacklevel=2)\n\u001b[0;32m-> 1810\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1811\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1812\u001b[0m         inner.__doc__ = _add_data_doc(inner.__doc__,\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/axes/_axes.py\u001b[0m in \u001b[0;36mimshow\u001b[0;34m(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)\u001b[0m\n\u001b[1;32m   5492\u001b[0m                               resample=resample, **kwargs)\n\u001b[1;32m   5493\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5494\u001b[0;31m         \u001b[0mim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5495\u001b[0m         \u001b[0mim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_alpha\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0malpha\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5496\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_clip_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36mset_data\u001b[0;34m(self, A)\u001b[0m\n\u001b[1;32m    636\u001b[0m         if not (self._A.ndim == 2\n\u001b[1;32m    637\u001b[0m                 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):\n\u001b[0;32m--> 638\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Invalid dimensions for image data\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    639\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    640\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_A\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndim\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: Invalid dimensions for image data"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAD8CAYAAACVSwr3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADGxJREFUeJzt23GIpHd9x/H3x1xTaRq1mBXk7jSR\nXhqvtpB0SVOEmmJaLinc/WGROwhtSsihNVJQCimWVOJfVmpBuNZeqUQFjad/lAVPArWRgHgxGxJj\n7kJkPW1zUZozpv4jGkO//WMm7WS/u5knd7Mzt/X9goV5nvntzHeH4X3PPPNcqgpJmvSKRQ8g6cJj\nGCQ1hkFSYxgkNYZBUmMYJDVTw5DkE0meTvLYJvcnyceSrCV5NMk1sx9T0jwNOWK4G9j3EvffCOwZ\n/xwG/uH8x5K0SFPDUFX3Az98iSUHgE/VyAngNUleP6sBJc3fjhk8xk7gyYntM+N931+/MMlhRkcV\nXHLJJb911VVXzeDpJW3moYce+kFVLb3c35tFGAarqqPAUYDl5eVaXV2d59NLP3eS/Pu5/N4svpV4\nCtg9sb1rvE/SNjWLMKwAfzz+duI64EdV1T5GSNo+pn6USPJZ4HrgsiRngL8GfgGgqj4OHAduAtaA\nHwN/ulXDSpqPqWGoqkNT7i/gPTObSNLCeeWjpMYwSGoMg6TGMEhqDIOkxjBIagyDpMYwSGoMg6TG\nMEhqDIOkxjBIagyDpMYwSGoMg6TGMEhqDIOkxjBIagyDpMYwSGoMg6TGMEhqDIOkxjBIagyDpMYw\nSGoMg6TGMEhqDIOkxjBIagyDpMYwSGoMg6TGMEhqDIOkZlAYkuxL8kSStSR3bHD/G5Lcl+ThJI8m\nuWn2o0qal6lhSHIRcAS4EdgLHEqyd92yvwKOVdXVwEHg72c9qKT5GXLEcC2wVlWnq+o54B7gwLo1\nBbxqfPvVwPdmN6KkeRsShp3AkxPbZ8b7Jn0QuDnJGeA48N6NHijJ4SSrSVbPnj17DuNKmodZnXw8\nBNxdVbuAm4BPJ2mPXVVHq2q5qpaXlpZm9NSSZm1IGJ4Cdk9s7xrvm3QrcAygqr4GvBK4bBYDSpq/\nIWF4ENiT5IokFzM6ubiybs1/AG8HSPJmRmHws4K0TU0NQ1U9D9wO3As8zujbh5NJ7kqyf7zs/cBt\nSb4BfBa4papqq4aWtLV2DFlUVccZnVSc3HfnxO1TwFtnO5qkRfHKR0mNYZDUGAZJjWGQ1BgGSY1h\nkNQYBkmNYZDUGAZJjWGQ1BgGSY1hkNQYBkmNYZDUGAZJjWGQ1BgGSY1hkNQYBkmNYZDUGAZJjWGQ\n1BgGSY1hkNQYBkmNYZDUGAZJjWGQ1BgGSY1hkNQYBkmNYZDUGAZJjWGQ1AwKQ5J9SZ5Ispbkjk3W\nvDPJqSQnk3xmtmNKmqcd0xYkuQg4Avw+cAZ4MMlKVZ2aWLMH+EvgrVX1bJLXbdXAkrbekCOGa4G1\nqjpdVc8B9wAH1q25DThSVc8CVNXTsx1T0jwNCcNO4MmJ7TPjfZOuBK5M8tUkJ5Ls2+iBkhxOsppk\n9ezZs+c2saQtN6uTjzuAPcD1wCHgn5K8Zv2iqjpaVctVtby0tDSjp5Y0a0PC8BSwe2J713jfpDPA\nSlX9rKq+A3yLUSgkbUNDwvAgsCfJFUkuBg4CK+vW/AujowWSXMboo8XpGc4paY6mhqGqngduB+4F\nHgeOVdXJJHcl2T9edi/wTJJTwH3AX1TVM1s1tKStlapayBMvLy/X6urqQp5b+nmR5KGqWn65v+eV\nj5IawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEM\nkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQyS\nGsMgqTEMkppBYUiyL8kTSdaS3PES696RpJIsz25ESfM2NQxJLgKOADcCe4FDSfZusO5S4M+BB2Y9\npKT5GnLEcC2wVlWnq+o54B7gwAbrPgR8GPjJDOeTtABDwrATeHJi+8x43/9Kcg2wu6q++FIPlORw\nktUkq2fPnn3Zw0qaj/M++ZjkFcBHgfdPW1tVR6tquaqWl5aWzvepJW2RIWF4Ctg9sb1rvO8FlwJv\nAb6S5LvAdcCKJyCl7WtIGB4E9iS5IsnFwEFg5YU7q+pHVXVZVV1eVZcDJ4D9VbW6JRNL2nJTw1BV\nzwO3A/cCjwPHqupkkruS7N/qASXN344hi6rqOHB83b47N1l7/fmPJWmRvPJRUmMYJDWGQVJjGCQ1\nhkFSYxgkNYZBUmMYJDWGQVJjGCQ1hkFSYxgkNYZBUmMYJDWGQVJjGCQ1hkFSYxgkNYZBUmMYJDWG\nQVJjGCQ1hkFSYxgkNYZBUmMYJDWGQVJjGCQ1hkFSYxgkNYZBUmMYJDWGQVJjGCQ1g8KQZF+SJ5Ks\nJbljg/vfl+RUkkeTfDnJG2c/qqR5mRqGJBcBR4Abgb3AoSR71y17GFiuqt8EvgD8zawHlTQ/Q44Y\nrgXWqup0VT0H3AMcmFxQVfdV1Y/HmyeAXbMdU9I8DQnDTuDJie0z432buRX40kZ3JDmcZDXJ6tmz\nZ4dPKWmuZnryMcnNwDLwkY3ur6qjVbVcVctLS0uzfGpJM7RjwJqngN0T27vG+14kyQ3AB4C3VdVP\nZzOepEUYcsTwILAnyRVJLgYOAiuTC5JcDfwjsL+qnp79mJLmaWoYqup54HbgXuBx4FhVnUxyV5L9\n42UfAX4Z+HySR5KsbPJwkraBIR8lqKrjwPF1++6cuH3DjOeStEBe+SipMQySGsMgqTEMkhrDIKkx\nDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEM\nkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkxDJIawyCpMQySGsMgqTEMkhrDIKkZFIYk+5I8kWQt\nyR0b3P+LST43vv+BJJfPelBJ8zM1DEkuAo4ANwJ7gUNJ9q5bdivwbFX9KvB3wIdnPaik+RlyxHAt\nsFZVp6vqOeAe4MC6NQeAT45vfwF4e5LMbkxJ87RjwJqdwJMT22eA395sTVU9n+RHwGuBH0wuSnIY\nODze/GmSx85l6AW5jHV/zwVsO80K22ve7TQrwK+dyy8NCcPMVNVR4ChAktWqWp7n85+P7TTvdpoV\ntte822lWGM17Lr835KPEU8Duie1d430brkmyA3g18My5DCRp8YaE4UFgT5IrklwMHARW1q1ZAf5k\nfPuPgH+rqprdmJLmaepHifE5g9uBe4GLgE9U1ckkdwGrVbUC/DPw6SRrwA8ZxWOao+cx9yJsp3m3\n06ywvebdTrPCOc4b/2GXtJ5XPkpqDIOkZsvDsJ0upx4w6/uSnEryaJIvJ3njIuacmOcl551Y944k\nlWRhX7MNmTXJO8ev78kkn5n3jOtmmfZeeEOS+5I8PH4/3LSIOcezfCLJ05tdF5SRj43/lkeTXDP1\nQatqy34Ynaz8NvAm4GLgG8DedWv+DPj4+PZB4HNbOdN5zvp7wC+Nb797UbMOnXe87lLgfuAEsHyh\nzgrsAR4GfmW8/boL+bVldFLv3ePbe4HvLnDe3wWuAR7b5P6bgC8BAa4DHpj2mFt9xLCdLqeeOmtV\n3VdVPx5vnmB0TceiDHltAT7E6P+u/GSew60zZNbbgCNV9SxAVT095xknDZm3gFeNb78a+N4c53vx\nIFX3M/o2cDMHgE/VyAngNUle/1KPudVh2Ohy6p2bramq54EXLqeetyGzTrqVUYUXZeq840PG3VX1\nxXkOtoEhr+2VwJVJvprkRJJ9c5uuGzLvB4Gbk5wBjgPvnc9o5+Tlvrfne0n0/xdJbgaWgbctepbN\nJHkF8FHglgWPMtQORh8nrmd0JHZ/kt+oqv9a6FSbOwTcXVV/m+R3GF3H85aq+u9FDzYLW33EsJ0u\npx4yK0luAD4A7K+qn85pto1Mm/dS4C3AV5J8l9Fny5UFnYAc8tqeAVaq6mdV9R3gW4xCsQhD5r0V\nOAZQVV8DXsnoP1hdiAa9t19ki0+K7ABOA1fwfydxfn3dmvfw4pOPxxZ0AmfIrFczOim1ZxEzvtx5\n163/Cos7+Tjktd0HfHJ8+zJGh76vvYDn/RJwy/j2mxmdY8gC3w+Xs/nJxz/kxScfvz718eYw8E2M\n6v9t4APjfXcx+hcXRqX9PLAGfB140wJf3Gmz/ivwn8Aj45+VRc06ZN51axcWhoGvbRh99DkFfBM4\neCG/toy+ifjqOBqPAH+wwFk/C3wf+BmjI69bgXcB75p4bY+M/5ZvDnkfeEm0pMYrHyU1hkFSYxgk\nNYZBUmMYJDWGQVJjGCQ1/wMKpFHVdp3xCwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(X_train[0,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<here you could just as well create computation graph>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<this may or may not be a good place to evaluating loss and optimizer>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<this may be a perfect cell to write a training&evaluation loop in>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<predict & evaluate on test here, right? No cheating pls.>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "# SPOILERS!\n",
    "\n",
    "Recommended pipeline\n",
    "\n",
    "* Adapt logistic regression from previous assignment to classify some number against others (e.g. zero vs nonzero)\n",
    "* Generalize it to multiclass logistic regression.\n",
    "  - Either try to remember lecture 0 or google it.\n",
    "  - Instead of weight vector you'll have to use matrix (feature_id x class_id)\n",
    "  - softmax (exp over sum of exps) can implemented manually or as T.nnet.softmax (stable)\n",
    "  - probably better to use STOCHASTIC gradient descent (minibatch)\n",
    "    - in which case sample should probably be shuffled (or use random subsamples on each iteration)\n",
    "* Add a hidden layer. Now your logistic regression uses hidden neurons instead of inputs.\n",
    "  - Hidden layer uses the same math as output layer (ex-logistic regression), but uses some nonlinearity (sigmoid) instead of softmax\n",
    "  - You need to train both layers, not just output layer :)\n",
    "  - Do not initialize layers with zeros (due to symmetry effects). A gaussian noize with small sigma will do.\n",
    "  - 50 hidden neurons and a sigmoid nonlinearity will do for a start. Many ways to improve. \n",
    "  - In ideal casae this totals to 2 .dot's, 1 softmax and 1 sigmoid\n",
    "  - __make sure this neural network works better than logistic regression__\n",
    "  \n",
    "* Now's the time to try improving the network. Consider layers (size, neuron count),  nonlinearities, optimization methods, initialization - whatever you want, but please avoid convolutions for now."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}