Part2_Music_Generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uoJsVjtCMunI"
   },
   "source": [
    "<table align=\"center\">\n",
    "  <td align=\"center\"><a target=\"_blank\" href=\"http://introtodeeplearning.com\">\n",
    "        <img src=\"https://i.ibb.co/Jr88sn2/mit.png\" style=\"padding-bottom:5px;\" />\n",
    "      Visit MIT Deep Learning</a></td>\n",
    "  <td align=\"center\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/aamini/introtodeeplearning/blob/master/lab1/Part2_Music_Generation.ipynb\">\n",
    "        <img src=\"https://i.ibb.co/2P3SLwK/colab.png\"  style=\"padding-bottom:5px;\" />Run in Google Colab</a></td>\n",
    "  <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/aamini/introtodeeplearning/blob/master/lab1/Part2_Music_Generation.ipynb\">\n",
    "        <img src=\"https://i.ibb.co/xfJbPmL/github.png\"  height=\"70px\" style=\"padding-bottom:5px;\"  />View Source on GitHub</a></td>\n",
    "</table>\n",
    "\n",
    "# Copyright Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bUik05YqMyCH"
   },
   "outputs": [],
   "source": [
    "# Copyright 2021 MIT 6.S191 Introduction to Deep Learning. All Rights Reserved.\n",
    "# \n",
    "# Licensed under the MIT License. You may not use this file except in compliance\n",
    "# with the License. Use and/or modification of this code outside of 6.S191 must\n",
    "# reference:\n",
    "#\n",
    "# © MIT 6.S191: Introduction to Deep Learning\n",
    "# http://introtodeeplearning.com\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O-97SDET3JG-"
   },
   "source": [
    "# Lab 1: Intro to TensorFlow and Music Generation with RNNs\n",
    "\n",
    "# Part 2: Music Generation with RNNs\n",
    "\n",
    "In this portion of the lab, we will explore building a Recurrent Neural Network (RNN) for music generation. We will train a model to learn the patterns in raw sheet music in [ABC notation](https://en.wikipedia.org/wiki/ABC_notation) and then use this model to generate new music. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rsvlBQYCrE4I"
   },
   "source": [
    "## 2.1 Dependencies \n",
    "First, let's download the course repository, install dependencies, and import the relevant packages we'll need for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "riVZCVK65QTH",
    "outputId": "a309314a-ba78-43c8-8a4d-ca8212149ba8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting mitdeeplearning\n",
      "  Downloading mitdeeplearning-0.2.0.tar.gz (2.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 2.1 MB 858 kB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from mitdeeplearning) (1.19.5)\n",
      "Requirement already satisfied: regex in /opt/conda/lib/python3.7/site-packages (from mitdeeplearning) (2020.11.13)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from mitdeeplearning) (4.55.1)\n",
      "Requirement already satisfied: gym in /opt/conda/lib/python3.7/site-packages (from mitdeeplearning) (0.18.0)\n",
      "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from gym->mitdeeplearning) (1.6.0)\n",
      "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from gym->mitdeeplearning) (1.5.0)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from gym->mitdeeplearning) (1.5.4)\n",
      "Requirement already satisfied: Pillow<=7.2.0 in /opt/conda/lib/python3.7/site-packages (from gym->mitdeeplearning) (7.2.0)\n",
      "Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from pyglet<=1.5.0,>=1.4.0->gym->mitdeeplearning) (0.18.2)\n",
      "Building wheels for collected packages: mitdeeplearning\n",
      "  Building wheel for mitdeeplearning (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for mitdeeplearning: filename=mitdeeplearning-0.2.0-py3-none-any.whl size=2115443 sha256=754cd540042b43525a8876ee2b8b26435716eb8aee9049b8b1ec48bac3c7aa7e\n",
      "  Stored in directory: /root/.cache/pip/wheels/9a/b9/4f/99b7c8c5c75355550b83e1fcfc02956fb40c35eb01e2262877\n",
      "Successfully built mitdeeplearning\n",
      "Installing collected packages: mitdeeplearning\n",
      "Successfully installed mitdeeplearning-0.2.0\n",
      "^C\n"
     ]
    },
    {
     "ename": "AssertionError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-976f1eabb7ca>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;31m# Check that we are using a GPU, if not switch runtimes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;31m#   using Runtime > Change Runtime Type > GPU\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlist_physical_devices\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'GPU'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Import Tensorflow 2.0\n",
    "#%tensorflow_version 2.x\n",
    "import tensorflow as tf \n",
    "\n",
    "# Download and import the MIT 6.S191 package\n",
    "!pip install mitdeeplearning\n",
    "import mitdeeplearning as mdl\n",
    "\n",
    "# Import all remaining packages\n",
    "import numpy as np\n",
    "import os\n",
    "import time\n",
    "import functools\n",
    "from IPython import display as ipythondisplay\n",
    "from tqdm import tqdm\n",
    "!apt-get install abcmidi timidity > /dev/null 2>&1\n",
    "\n",
    "# Check that we are using a GPU, if not switch runtimes\n",
    "#   using Runtime > Change Runtime Type > GPU\n",
    "assert len(tf.config.list_physical_devices('GPU')) > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ajvp0No4qDm"
   },
   "source": [
    "## 2.2 Dataset\n",
    "\n",
    "![Let's Dance!](http://33.media.tumblr.com/3d223954ad0a77f4e98a7b87136aa395/tumblr_nlct5lFVbF1qhu7oio1_500.gif)\n",
    "\n",
    "We've gathered a dataset of thousands of Irish folk songs, represented in the ABC notation. Let's download the dataset and inspect it: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "P7dFnP5q3Jve",
    "outputId": "5dd7114c-7d44-40e0-c5b9-af980521a095"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 817 songs in text\n",
      "\n",
      "Example song: \n",
      "X:1\n",
      "T:Alexander's\n",
      "Z: id:dc-hornpipe-1\n",
      "M:C|\n",
      "L:1/8\n",
      "K:D Major\n",
      "(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\n",
      "dAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\n",
      "AG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\n",
      "FAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\n"
     ]
    }
   ],
   "source": [
    "# Download the dataset\n",
    "songs = mdl.lab1.load_training_data()\n",
    "\n",
    "# Print one of the songs to inspect it in greater detail!\n",
    "example_song = songs[0]\n",
    "print(\"\\nExample song: \")\n",
    "print(example_song)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hKF3EHJlCAj2"
   },
   "source": [
    "We can easily convert a song in ABC notation to an audio waveform and play it back. Be patient for this conversion to run, it can take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "11toYzhEEKDz"
   },
   "outputs": [],
   "source": [
    "# Convert the ABC notation to audio file and listen to it\n",
    "mdl.lab1.play_song(example_song)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7vH24yyquwKQ"
   },
   "source": [
    "One important thing to think about is that this notation of music does not simply contain information on the notes being played, but additionally there is meta information such as the song title, key, and tempo. How does the number of different characters that are present in the text file impact the complexity of the learning problem? This will become important soon, when we generate a numerical representation for the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "IlCgQBRVymwR",
    "outputId": "a183ed42-6938-44a7-df51-eae0479218f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 83 unique characters in the dataset\n"
     ]
    }
   ],
   "source": [
    "# Join our list of song strings into a single string containing all songs\n",
    "songs_joined = \"\\n\\n\".join(songs) \n",
    "\n",
    "# Find all unique characters in the joined string\n",
    "vocab = sorted(set(songs_joined))\n",
    "print(\"There are\", len(vocab), \"unique characters in the dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rNnrKn_lL-IJ"
   },
   "source": [
    "## 2.3 Process the dataset for the learning task\n",
    "\n",
    "Let's take a step back and consider our prediction task. We're trying to train a RNN model to learn patterns in ABC music, and then use this model to generate (i.e., predict) a new piece of music based on this learned information. \n",
    "\n",
    "Breaking this down, what we're really asking the model is: given a character, or a sequence of characters, what is the most probable next character? We'll train the model to perform this task. \n",
    "\n",
    "To achieve this, we will input a sequence of characters to the model, and train the model to predict the output, that is, the following character at each time step. RNNs maintain an internal state that depends on previously seen elements, so information about all characters seen up until a given moment will be taken into account in generating the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LFjSVAlWzf-N"
   },
   "source": [
    "### Vectorize the text\n",
    "\n",
    "Before we begin training our RNN model, we'll need to create a numerical representation of our text-based dataset. To do this, we'll generate two lookup tables: one that maps characters to numbers, and a second that maps numbers back to characters. Recall that we just identified the unique characters present in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "IalZLbvOzf-F"
   },
   "outputs": [],
   "source": [
    "### Define numerical representation of text ###\n",
    "\n",
    "# Create a mapping from character to unique index.\n",
    "# For example, to get the index of the character \"d\", \n",
    "#   we can evaluate `char2idx[\"d\"]`.  \n",
    "char2idx = {u:i for i, u in enumerate(vocab)}\n",
    "\n",
    "# Create a mapping from indices to characters. This is\n",
    "#   the inverse of char2idx and allows us to convert back\n",
    "#   from unique index to the character in our vocabulary.\n",
    "idx2char = np.array(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tZfqhkYCymwX"
   },
   "source": [
    "This gives us an integer representation for each character. Observe that the unique characters (i.e., our vocabulary) in the text are mapped as indices from 0 to `len(unique)`. Let's take a peek at this numerical representation of our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "FYyNlCNXymwY",
    "outputId": "33fefb10-e3cc-4b38-b66e-840278920a5f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  '\\n':   0,\n",
      "  ' ' :   1,\n",
      "  '!' :   2,\n",
      "  '\"' :   3,\n",
      "  '#' :   4,\n",
      "  \"'\" :   5,\n",
      "  '(' :   6,\n",
      "  ')' :   7,\n",
      "  ',' :   8,\n",
      "  '-' :   9,\n",
      "  '.' :  10,\n",
      "  '/' :  11,\n",
      "  '0' :  12,\n",
      "  '1' :  13,\n",
      "  '2' :  14,\n",
      "  '3' :  15,\n",
      "  '4' :  16,\n",
      "  '5' :  17,\n",
      "  '6' :  18,\n",
      "  '7' :  19,\n",
      "  ...\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print('{')\n",
    "for char,_ in zip(char2idx, range(20)):\n",
    "    print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))\n",
    "print('  ...\\n}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "g-LnKyu4dczc"
   },
   "outputs": [],
   "source": [
    "### Vectorize the songs string ###\n",
    "\n",
    "'''TODO: Write a function to convert the all songs string to a vectorized\n",
    "    (i.e., numeric) representation. Use the appropriate mapping\n",
    "    above to convert from vocab characters to the corresponding indices.\n",
    "\n",
    "  NOTE: the output of the `vectorize_string` function \n",
    "  should be a np.array with `N` elements, where `N` is\n",
    "  the number of characters in the input string\n",
    "'''\n",
    "\n",
    "def vectorize_string(string):\n",
    "  vectorized_songs = np.array([char2idx[song] for song in string ])\n",
    "  return vectorized_songs\n",
    "vectorized_songs = vectorize_string(songs_joined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IqxpSuZ1w-ub"
   },
   "source": [
    "We can also look at how the first part of the text is mapped to an integer representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "l1VKcQHcymwb",
    "outputId": "bf832391-7aa6-4ac4-db96-93a3e5c5bfc9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'X:1\\nT:Alex' ---- characters mapped to int ----> [49 22 13  0 45 22 26 67 60 79]\n"
     ]
    }
   ],
   "source": [
    "print ('{} ---- characters mapped to int ----> {}'.format(repr(songs_joined[:10]), vectorized_songs[:10]))\n",
    "# check that vectorized_songs is a numpy array\n",
    "assert isinstance(vectorized_songs, np.ndarray), \"returned result should be a numpy array\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hgsVvVxnymwf"
   },
   "source": [
    "### Create training examples and targets\n",
    "\n",
    "Our next step is to actually divide the text into example sequences that we'll use during training. Each input sequence that we feed into our RNN will contain `seq_length` characters from the text. We'll also need to define a target sequence for each input sequence, which will be used in training the RNN to predict the next character. For each input, the corresponding target will contain the same length of text, except shifted one character to the right.\n",
    "\n",
    "To do this, we'll break the text into chunks of `seq_length+1`. Suppose `seq_length` is 4 and our text is \"Hello\". Then, our input sequence is \"Hell\" and the target sequence is \"ello\".\n",
    "\n",
    "The batch method will then let us convert this stream of character indices to sequences of the desired size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "LF-N8F7BoDRi",
    "outputId": "7094cde0-bcd3-4cb5-8729-00ad646230ce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[PASS] test_batch_func_types\n",
      "[PASS] test_batch_func_shapes\n",
      "[PASS] test_batch_func_next_step\n",
      "======\n",
      "[PASS] passed all tests!\n"
     ]
    }
   ],
   "source": [
    "### Batch definition to create training examples ###\n",
    "\n",
    "def get_batch(vectorized_songs, seq_length, batch_size):\n",
    "  # the length of the vectorized songs string\n",
    "  n = vectorized_songs.shape[0] - 1\n",
    "  # randomly choose the starting indices for the examples in the training batch\n",
    "  idx = np.random.choice(n-seq_length, batch_size)\n",
    "\n",
    "  '''TODO: construct a list of input sequences for the training batch'''\n",
    "  input_batch = [vectorized_songs[i:i+seq_length] for i in idx]\n",
    "  '''TODO: construct a list of output sequences for the training batch'''\n",
    "  output_batch = [vectorized_songs[i+1:i+seq_length+1] for i in idx]\n",
    "\n",
    "  # x_batch, y_batch provide the true inputs and targets for network training\n",
    "  x_batch = np.reshape(input_batch, [batch_size, seq_length])\n",
    "  y_batch = np.reshape(output_batch, [batch_size, seq_length])\n",
    "  return x_batch, y_batch\n",
    "\n",
    "\n",
    "# Perform some simple tests to make sure your batch function is working properly! \n",
    "test_args = (vectorized_songs, 10, 2)\n",
    "if not mdl.lab1.test_batch_func_types(get_batch, test_args) or \\\n",
    "   not mdl.lab1.test_batch_func_shapes(get_batch, test_args) or \\\n",
    "   not mdl.lab1.test_batch_func_next_step(get_batch, test_args): \n",
    "   print(\"======\\n[FAIL] could not pass tests\")\n",
    "else: \n",
    "   print(\"======\\n[PASS] passed all tests!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_33OHL3b84i0"
   },
   "source": [
    "For each of these vectors, each index is processed at a single time step. So, for the input at time step 0, the model receives the index for the first character in the sequence, and tries to predict the index of the next character. At the next timestep, it does the same thing, but the RNN considers the information from the previous step, i.e., its updated state, in addition to the current input.\n",
    "\n",
    "We can make this concrete by taking a look at how this works over the first several characters in our text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "0eBu9WZG84i0",
    "outputId": "30814ec8-b569-4025-cda7-5c213b18c197"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step   0\n",
      "  input: 27 ('B')\n",
      "  expected output: 58 ('c')\n",
      "Step   1\n",
      "  input: 58 ('c')\n",
      "  expected output: 26 ('A')\n",
      "Step   2\n",
      "  input: 26 ('A')\n",
      "  expected output: 82 ('|')\n",
      "Step   3\n",
      "  input: 82 ('|')\n",
      "  expected output: 31 ('F')\n",
      "Step   4\n",
      "  input: 31 ('F')\n",
      "  expected output: 14 ('2')\n"
     ]
    }
   ],
   "source": [
    "x_batch, y_batch = get_batch(vectorized_songs, seq_length=5, batch_size=1)\n",
    "\n",
    "for i, (input_idx, target_idx) in enumerate(zip(np.squeeze(x_batch), np.squeeze(y_batch))):\n",
    "    print(\"Step {:3d}\".format(i))\n",
    "    print(\"  input: {} ({:s})\".format(input_idx, repr(idx2char[input_idx])))\n",
    "    print(\"  expected output: {} ({:s})\".format(target_idx, repr(idx2char[target_idx])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r6oUuElIMgVx"
   },
   "source": [
    "## 2.4 The Recurrent Neural Network (RNN) model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m8gPwEjRzf-Z"
   },
   "source": [
    "Now we're ready to define and train a RNN model on our ABC music dataset, and then use that trained model to generate a new song. We'll train our RNN using batches of song snippets from our dataset, which we generated in the previous section.\n",
    "\n",
    "The model is based off the LSTM architecture, where we use a state vector to maintain information about the temporal relationships between consecutive characters. The final output of the LSTM is then fed into a fully connected [`Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layer where we'll output a softmax over each character in the vocabulary, and then sample from this distribution to predict the next character. \n",
    "\n",
    "As we introduced in the first portion of this lab, we'll be using the Keras API, specifically, [`tf.keras.Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential), to define the model. Three layers are used to define the model:\n",
    "\n",
    "* [`tf.keras.layers.Embedding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding): This is the input layer, consisting of a trainable lookup table that maps the numbers of each character to a vector with `embedding_dim` dimensions.\n",
    "* [`tf.keras.layers.LSTM`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM): Our LSTM network, with size `units=rnn_units`. \n",
    "* [`tf.keras.layers.Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): The output layer, with `vocab_size` outputs.\n",
    "\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/aamini/introtodeeplearning/2019/lab1/img/lstm_unrolled-01-01.png\" alt=\"Drawing\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rlaOqndqBmJo"
   },
   "source": [
    "### Define the RNN model\n",
    "\n",
    "Now, we will define a function that we will use to actually build the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "8DsWzojvkbc7"
   },
   "outputs": [],
   "source": [
    "def LSTM(rnn_units): \n",
    "  return tf.keras.layers.LSTM(\n",
    "    rnn_units, \n",
    "    return_sequences=True, \n",
    "    recurrent_initializer='glorot_uniform',\n",
    "    recurrent_activation='sigmoid',\n",
    "    stateful=True,\n",
    "  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IbWU4dMJmMvq"
   },
   "source": [
    "The time has come! Fill in the `TODOs` to define the RNN model within the `build_model` function, and then call the function you just defined to instantiate the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "MtCrdfzEI2N0"
   },
   "outputs": [],
   "source": [
    "### Defining the RNN Model ###\n",
    "\n",
    "'''TODO: Add LSTM and Dense layers to define the RNN model using the Sequential API.'''\n",
    "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
    "  model = tf.keras.Sequential([\n",
    "    # Layer 1: Embedding layer to transform indices into dense vectors \n",
    "    #   of a fixed embedding size\n",
    "    tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[batch_size, None]),\n",
    "\n",
    "    # Layer 2: LSTM with `rnn_units` number of units. \n",
    "    # TODO: Call the LSTM function defined above to add this layer.\n",
    "    LSTM(rnn_units),\n",
    "\n",
    "    # Layer 3: Dense (fully-connected) layer that transforms the LSTM output\n",
    "    #   into the vocabulary size. \n",
    "    # TODO: Add the Dense layer.\n",
    "    tf.keras.layers.Dense(units = vocab_size)\n",
    "  ])\n",
    "\n",
    "  return model\n",
    "\n",
    "# Build a simple model with default hyperparameters. You will get the \n",
    "#   chance to change these later.\n",
    "model = build_model(len(vocab), embedding_dim=256, rnn_units=1024, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-ubPo0_9Prjb"
   },
   "source": [
    "### Test out the RNN model\n",
    "\n",
    "It's always a good idea to run a few simple checks on our model to see that it behaves as expected.  \n",
    "\n",
    "First, we can use the `Model.summary` function to print out a summary of our model's internal workings. Here we can check the layers in the model, the shape of the output of each of the layers, the batch size, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "RwG1DD6rDrRM",
    "outputId": "7d699a73-e0a3-40d6-fc7d-a1f0430c2854"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (32, None, 256)           21248     \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (32, None, 1024)          5246976   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (32, None, 83)            85075     \n",
      "=================================================================\n",
      "Total params: 5,353,299\n",
      "Trainable params: 5,353,299\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8xeDn5nZD0LX"
   },
   "source": [
    "We can also quickly check the dimensionality of our output, using a sequence length of 100. Note that the model can be run on inputs of any length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "C-_70kKAPrPU",
    "outputId": "c018cb10-2820-448e-b642-2e507bba7edc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input shape:       (32, 100)  # (batch_size, sequence_length)\n",
      "Prediction shape:  (32, 100, 83) # (batch_size, sequence_length, vocab_size)\n"
     ]
    }
   ],
   "source": [
    "x, y = get_batch(vectorized_songs, seq_length=100, batch_size=32)\n",
    "pred = model(x)\n",
    "print(\"Input shape:      \", x.shape, \" # (batch_size, sequence_length)\")\n",
    "print(\"Prediction shape: \", pred.shape, \"# (batch_size, sequence_length, vocab_size)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mT1HvFVUGpoE"
   },
   "source": [
    "### Predictions from the untrained model\n",
    "\n",
    "Let's take a look at what our untrained model is predicting.\n",
    "\n",
    "To get actual predictions from the model, we sample from the output distribution, which is defined by a `softmax` over our character vocabulary. This will give us actual character indices. This means we are using a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) to sample over the example prediction. This gives a prediction of the next character (specifically its index) at each timestep.\n",
    "\n",
    "Note here that we sample from this probability distribution, as opposed to simply taking the `argmax`, which can cause the model to get stuck in a loop.\n",
    "\n",
    "Let's try this sampling out for the first example in the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "4V4MfFg0RQJg",
    "outputId": "be47da50-ab03-4564-b74f-bf9c783c8712"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1, 48, 61, 57, 39, 18, 13, 19, 11, 22, 42, 58, 65, 21, 37, 13, 59,\n",
       "       70, 35, 70, 56,  5, 64, 26,  9, 33, 40, 82, 20, 81, 58, 49, 46, 75,\n",
       "       43, 63, 53,  2, 72, 78, 42, 79, 52, 15, 80, 46, 11, 59, 64, 75, 33,\n",
       "       73, 77, 70, 75, 38, 17, 73, 44, 35, 68, 61,  5, 61, 43, 73,  6, 73,\n",
       "       51, 44,  9, 12, 58, 47, 16, 54, 23, 61, 66, 75, 12, 42, 66, 38,  1,\n",
       "       54, 12, 19, 27, 45, 67, 67, 61, 39, 63, 49, 63, 12, 17, 38])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices = tf.random.categorical(pred[0], num_samples=1)\n",
    "sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()\n",
    "sampled_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LfLtsP3mUhCG"
   },
   "source": [
    "We can now decode these to see the text predicted by the untrained model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "xWcFwPwLSo05",
    "outputId": "1c382838-b088-4f8e-b265-db41f6113c2b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: \n",
      " '\\nK:D Major\\nfe|d2Bd A2FA|BAFA D2ED|B,DA,D DFBF|AFDF E2fe|!\\nd2Bd A2FA|BAFA D2ED|B,DA,D DFBF|AFEF D2:|!'\n",
      "\n",
      "Next Char Predictions: \n",
      " \" WfbN617/:Qcj9L1doJoa'iA-HO|8zcXUtRh]!qwQx[3yU/ditHrvotM5rSJmf'fRr(rZS-0cV4^<fkt0QkM ^07BTllfNhXh05M\"\n"
     ]
    }
   ],
   "source": [
    "print(\"Input: \\n\", repr(\"\".join(idx2char[x[0]])))\n",
    "print()\n",
    "print(\"Next Char Predictions: \\n\", repr(\"\".join(idx2char[sampled_indices])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HEHHcRasIDm9"
   },
   "source": [
    "As you can see, the text predicted by the untrained model is pretty nonsensical! How can we do better? We can train the network!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LJL0Q0YPY6Ee"
   },
   "source": [
    "## 2.5 Training the model: loss and training operations\n",
    "\n",
    "Now it's time to train the model!\n",
    "\n",
    "At this point, we can think of our next character prediction problem as a standard classification problem. Given the previous state of the RNN, as well as the input at a given time step, we want to predict the class of the next character -- that is, to actually predict the next character. \n",
    "\n",
    "To train our model on this classification task, we can use a form of the `crossentropy` loss (negative log likelihood loss). Specifically, we will use the [`sparse_categorical_crossentropy`](https://www.tensorflow.org/api_docs/python/tf/keras/losses/sparse_categorical_crossentropy) loss, as it utilizes integer targets for categorical classification tasks. We will want to compute the loss using the true targets -- the `labels` -- and the predicted targets -- the `logits`.\n",
    "\n",
    "Let's first compute the loss using our example predictions from the untrained model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "4HrXTACTdzY-",
    "outputId": "17e8a589-e864-4489-aaf1-2787fe01133b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction shape:  (32, 100, 83)  # (batch_size, sequence_length, vocab_size)\n",
      "scalar_loss:       4.4191628\n"
     ]
    }
   ],
   "source": [
    "### Defining the loss function ###\n",
    "\n",
    "'''TODO: define the loss function to compute and return the loss between\n",
    "    the true labels and predictions (logits). Set the argument from_logits=True.'''\n",
    "def compute_loss(labels, logits):\n",
    "  loss = tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True) # TODO\n",
    "  return loss\n",
    "\n",
    "'''TODO: compute the loss using the true next characters from the example batch \n",
    "    and the predictions from the untrained model several cells above'''\n",
    "example_batch_loss = compute_loss(y, pred) # TODO\n",
    "\n",
    "print(\"Prediction shape: \", pred.shape, \" # (batch_size, sequence_length, vocab_size)\") \n",
    "print(\"scalar_loss:      \", example_batch_loss.numpy().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0Seh7e6eRqd7"
   },
   "source": [
    "Let's start by defining some hyperparameters for training the model. To start, we have provided some reasonable values for some of the parameters. It is up to you to use what we've learned in class to help optimize the parameter selection here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "JQWUUhKotkAY"
   },
   "outputs": [],
   "source": [
    "### Hyperparameter setting and optimization ###\n",
    "\n",
    "# Optimization parameters:\n",
    "num_training_iterations = 2000  # Increase this to train longer\n",
    "batch_size = 4  # Experiment between 1 and 64\n",
    "seq_length = 100  # Experiment between 50 and 500\n",
    "learning_rate = 5e-3  # Experiment between 1e-5 and 1e-1\n",
    "\n",
    "# Model parameters: \n",
    "vocab_size = len(vocab)\n",
    "embedding_dim = 256 \n",
    "rnn_units = 1024  # Experiment between 1 and 2048\n",
    "\n",
    "# Checkpoint location: \n",
    "checkpoint_dir = './training_checkpoints'\n",
    "checkpoint_prefix = os.path.join(checkpoint_dir, \"my_ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5cu11p1MKYZd"
   },
   "source": [
    "Now, we are ready to define our training operation -- the optimizer and duration of training -- and use this function to train the model. You will experiment with the choice of optimizer and the duration for which you train your models, and see how these changes affect the network's output. Some optimizers you may like to try are [`Adam`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam?version=stable) and [`Adagrad`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adagrad?version=stable).\n",
    "\n",
    "First, we will instantiate a new model and an optimizer. Then, we will use the [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape) method to perform the backpropagation operations. \n",
    "\n",
    "We will also generate a print-out of the model's progress through training, which will help us easily visualize whether or not we are minimizing the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "F31vzJ_u66cb",
    "outputId": "650f2255-c9b3-4ad3-f1b2-8cbb91aa2d7d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAv7UlEQVR4nO3dd5wTZf4H8M832cYCS9ull6VXBZQqRRCkqYhesZ1ydv15Ylc4PeupKOp5iOWwYC93h+UURVDBVVABKav0Kp1d6i5l+/P7Y2aSSTLJJtlMssx+3q8XL5LJTObZZPc7zzzP93keUUqBiIicx5XoAhARkT0Y4ImIHIoBnojIoRjgiYgcigGeiMihkhJdALPMzEyVnZ2d6GIQEZ00fv755/1KqSyr16pVgM/OzsayZcsSXQwiopOGiPwW7DU20RARORQDPBGRQzHAExE5FAM8EZFDMcATETmUrVk0IrINQCGAcgBlSqk+dp6PiIi84pEmOVwptT8O5yEiIhNHNNHsKyjC/DX7El0MIqJqxe4ArwDME5GfReQ6qx1E5DoRWSYiy/Lz86M6ycUzf8S1by5DRQXnticiMtgd4AcppU4DMBbATSIy1H8HpdRMpVQfpVSfrCzL0baV2nnoOACgpLyiKmUlInIUWwO8Umq3/n8egI8A9LPjPMlu7ccoLmOAJyIy2BbgRaS2iNQ1HgMYBeBXO87lDfDldrw9EdFJyc4smiYAPhIR4zzvKqXm2nEiI8CXsAZPRORhW4BXSm0B0NOu9zdLdgsABngiIjNHpEl6avDsZCUi8nBIgNdq8MWlDPBERAaHBHjW4ImI/DkrwLMNnojIwyEBXm+iYZokEZGHIwK826UF+LJyTlVARGRwRIAXaAGeU9EQEXk5IsDr8R1KMcITERkcEeD1+M4aPBGRiSMCvKGCNXgiIg9HBHjRq/AM8EREXo4I8AbGdyIiL0cEeG8WDSM8EZHBEQHewE5WIiIvRwR4tsETEQVyRIA3MA+eiMjLEQHeW4NPbDmIiKoTZwR4drISEQVwRIA3sAZPROTliADvaaJhhCci8nBEgP9u434AbKIhIjJzRIA3sAJPROTlqADPNEkiIi9HBXg20RAReTkswCe6BERE1YfDAjwjPBGRwVEBnvGdiMjLUQGeefBERF6OCPBvX90fANvgiYjMHBHgB3VoBIBt8EREZo4I8CICEebBExGZOSLAA4BLhE00REQmDgrwbKIhIjJzTIAXEZQzwBMReTgmwLuEefBERGaOCfBuEebBExGZOCbAs5OViMiXYwK8sJOViMiH7QFeRNwiskJEPrPzPC6XMA+eiMgkHjX4WwCstfskbKIhIvJla4AXkZYAzgHwip3nAZgHT0Tkz+4a/LMA7gZQEWwHEblORJaJyLL8/PyoT7T/aAl+3XUk6uOJiJzGtgAvIucCyFNK/RxqP6XUTKVUH6VUn6ysrCqdc9VOBngiIkOSje89CMB4ERkHIA1Ahoi8rZT6kx0nS0lyoV6tZDvemojopGRbDV4pNUUp1VIplQ3gYgDf2BXcAWBAu0ZoUb+WXW9PRHTScUwePDtZiYh82dlE46GUWghgoZ3ncIugnHmSREQezqnBu5gHT0Rk5pwAL1x0m4jIzDEB3u0StsETEZk4JsBzwQ8iIl+OCfCcD56IyJdjArxLgN2Hi3DwWEmii0JEVC04J8C7BCXlFTjtkfmJLgoRUbXgnAAvkugiEBFVK44J8G5TgN+UV5jAkhARVQ+OCfAulzfAj3wmJ4ElISKqHhwT4FPcbKIhIjJzTIBv1TA90UUgIqpWHBPgV+w4nOgiEBFVK44J8BzkRETkyzEB3tzJSkRETgrwzIMnIvLhmADPJBoiIl+OCfCswRMR+XJMgBe/AK84dTAR1XCOCfBuv5+ESTVEVNM5KMD71uC5uhMR1XSOCfD+bfDlrMITUQ3n2ADPGjwR1XSOCfD+TTSswRNRTeeYAJ9RK9nneUVFggpCRFRNOCbA3zS8vc/zcjbREFEN55gAn5rkRpemdT3Pi8vKE1gaIqLEc0yAB4D2WXU8jx/5bE0CS0JElHiOCvDmGvySrYcSWBIiosRzVIC/cZi3HX7/0WI89/XGBJaGiCixHBXgk/zmK3h6/oYElYSIKPEcFeCJiMiLAZ6IyKEY4ImIHIoBnojIoRjgiYgcigGeiMihGOCJiBwqrAAvIrVFxKU/7iQi40UkubLjiIgoccKtwecASBORFgC+BnAlgNdDHSAiaSKyRERWichqEXmoakUNz/RLesfjNERE1V64AV6UUscBXAjgOaXUBQC6VXJMMYCzlFI9AfQCMEZEBkRd0jCN79nc8zjJbxEQIqKaJOwALyIDAVwGYI6+LSnUAUpzVH+arP+L6yTtZRUKx4rL4nlKIqJqI9wAfyuAKQA+UkqtFpF2ABZUdpCIuEVkJYA8APOVUj9Z7HOdiCwTkWX5+fnhlzxM89fsi/l7EhGdDMIK8Eqpb5VS45VST+idrfuVUpPCOK5cKdULQEsA/USkh8U+M5VSfZRSfbKysiItv6Xhnb3vI2ylIaIaKtwsmndFJENEagNYA2C9iNwV7kmUUocBLAQwJppCRmrGpafF4zRERNVauE003ZRSBQAmAPgcQGsAl4c6QESyRKS+/rgWgJEA1kVd0gjUTvV2Dwir8ERUQ4XsKDVJ1vPeJwCYoZQqFZHKOkybAXhDRNzQLiT/Vkp9Fn1RiYgoEuEG+H8B2AZgFYAcEWkDoCDUAUqpXABMSiciSpBwO1mnK6VaKKXG6emPvwEYbnPZquSWER0BMBeeiGqucDtZ64nIM0Y6o4g8DaC2zWWrkvN6NgMAlFfENfWeiKjaCLeT9TUAhQD+qP8rADDLrkLFQpJL+9Fufm9FgktCRJQY4bbBt1dK/c70/CF9AFO1lZzkvXYdOFqMRnVSE1gaIqL4C7cGf0JEBhtPRGQQgBP2FCk2kk1t71e9vjSBJSEiSoxwa/A3AHhTROrpzw8BmGhPkWIj2e29dm0/eDyBJSEiSoywArxSahWAniKSoT8vEJFbAeTaWLYqSXJ7a/AuDnYiohooohWdlFIF+ohWALjdhvLEjLkG72KqJBHVQFVZsq9aR81UUydrtS4oEZFNqhLgq3WCuYhgSMfMRBeDiChhQrbBi0ghrAO5AKhlS4liiIOciKgmq2xVprrxKogdyvQAzzBPRDVRVZpoqj3W4ImoJqsRAT6/sDjBJSEiir8aEeABoIK1eSKqYRwd4Gsluz2PX/5uSwJLQkQUf44O8NMv8a438tGKXQksCRFR/Dk6wDetl4YB7RoCAApOlLKZhohqFEcHeABw69MU7D5ShGnz1ie4NERE8eP4AG+eaOyNxdsSVxAiojhzfIB3myYaKytnEw0R1RzOD/CmGnxJeUUCS0JEFF+OD/AdGtdJdBGIiBLC8QH+ztGdE10EIqKEcHyANy/8AQBrdhcE2ZOIyFkcH+D9XfvmskQXgYgoLmpEgO/S1Dvr8a7DJxJYEiKi+KkRAX5g+0aJLgIRUdzViAAvXJWViGqgGhHgh3byXZtVKYWSMubEE5Gz1YgAP6xzY6SYsmmmzl2HTvd9geKy8gSWiojIXjUiwANA3TTv8rP/+labG76ohLV4InKuGhPgU5MsflQ2zRORg9WYAJ9iFeD1ucdeWLgJK3ccjmt5iIjsVmMCfOOMtIBtFUqL8E/OXY8Jzy+Kd5GIiGxVYwL8oxN6BGwrV5w+mIicq8YE+NqpSQHbKhjgicjBakyAT3IH9qhWVGg58URETmRbgBeRViKyQETWishqEbnFrnOFo3Fd6zb4ctNC3Az2ROQkdtbgywDcoZTqCmAAgJtEpJuN56tUsl8tvrxC+bTDV9gc3xdv2o+jxWX2noSISGdbgFdK7VFKLdcfFwJYC6CFXecLR6cmdX2eVyiFigrf53bYe6QIuTsP49JXfsKt76+w5RxERP4Cex5tICLZAHoD+MnitesAXAcArVu3trUcr07siwGPf+15fua0hUhL9l7jVu8uQK9W9bFgfR66N8+wbNaJhvmc6/cVxuQ9iYgqY3snq4jUATAbwK1KqYDllJRSM5VSfZRSfbKysmwtS1bd1IBtRaXeKvyE5xfhybnrcOWspbh45o+2loWIyG62BngRSYYW3N9RSn1o57nC4XZVPjfBCws3AwC25B+zuzhERLayM4tGALwKYK1S6hm7zmOn9XsL8cvOI4kuBhFRVOyswQ8CcDmAs0Rkpf5vnI3ni7nRz+bgvBnfAwBeX7QVV85agm835Ce4VERE4bGtk1Up9T0cNF/jg5+uAQAsWJ+P16/si2GdG2NTXiH2Hy3BgHbhLwnIVHsiipcaM5LVkHPX8IiPqfBLkP/zrKXYnH8UI5/JiaozdtfhEzhmyoc/dKwEhUWlEb8PEVEoNS7At26UHvExS7YdjNn5lQIGTf0Gv3/pB8+23o/Mx8DHv4nZOYiIgBoY4KNRVBq4tJ9bqtb6tHZPgc+dAUe4ElGsMcCH4c7/5AZsi8VUw8ctLhxE4VJK4eu1+wKaEIkMNTLA109Pxg1ntg97//1HiwO2jXj6W8/j7Mlzgh67aNP+oK9xumKqio9W7MLVbyzDO0u2J7ooVE3VyAC/8v5RmDy2iy3vfaKkHOc+9x0ue0XrfH3luy1B91Vhrvm9ZOtBvLF4WwxKR06y50gRAGD34RMJLglVV3GZi6Ym6Xr/XJ/nEqKtvrQivAj/x39pHbITz8j22b5i+yF0b17Per1ZIqrxGBkSaM/hIsxfsy+qYzfnH8UFLyzGI5+tiXGpqq6otBwnSti/QJRoNTrAP37hKTF7r+zJc7Byx2GfbV/8sgcHLNrvDefN+B7XvrksqvMdOFoCAFizJ2D+toTr+/evAu5kiCj+anQTzSX9WmPDvkLMWrQtJu834flFPs9vfGd5wD6xWjXKWIkqVLqmUipkE5FdCpnySVQt1OgaPADcO64rPrlpUNzOt68weI0+EkYGjkv/BldsP4TsyXOw4+BxAMDGfYVoO+XzqJuAThZKqRq/1KJj5gOhmKvxAT7J7ULrhpGPbo1WeYic5ZvfW4HjJeHVfo0A/+OWg/h+43789+edAICF+mRoRnPRF7/uqUJpq7++j36FM6Z6RwGv3HEYPR+ah0PHShJYqtg4UVJuOciOKFw1PsADsRm0FAufrtqND5fvwrb9weei/8f8DcjdedjnQnHL+yuQ7Na+ytIyLTPHpTfNfLh8V9gXjVj4YGl8c7L3Hy3xpAsCwAsLNuHIiVL8tDV200skStf756LXw/MSXQw6iTHAA6iTWn26It7+8TcMe2ohnl+wCav8Om1f+nYz/vn1RoyfschnkFSFUp4FxUvLK7Bk60Hc8Z9Vntf3mgJguPYfLY6q9njP7F8iPoaCM684RhQpBngAaclubJt6DqbYNPgpEuv2amu2TvtyPc7367Sd+sU6z+OycnOAhycXvrS8Avd/8qvPcVb3J9mT5yB78hwUl5VbtmH3+ftXuPCFxfjPsh3R/ihks5re90CVY4A3GXdKMwBAi/q1ElySyplr8EopJOm9rSXlgZkzxq6d7/sCD326GkdOeKcm7nzfXDw1b73P/s/oz9fsKcBd/83FriqOlNyUV+jJiy8pS0yN9HhJGR76dLUj2ub9JSBRKipl5RX486wlWBqD2VmLy8pxz39zkVcQ+d1pTcIAb2IEQhEgNcTo0AV3DotPgUIoN8XJgqIy/PPrjQC0Grz/3/vMnM3YcfA4issqMGvRNvzhpcU+r/972U6f59O/2eTzvMx0srLyioBaf2l5RdClDY+cKMXIZ3LQ9f652JR3FJ3u+wL/W7U7YL+nvlyPC1/Q7ljyCouQPXkO/r10h/7zlXouDBUVCk/PWx9Ws9Ouwyc8w/hnLdqGWYu2ofcj87Ep72ilx1LVVFQo5PtljO0tKMLC9fm45b0VVX7/eav34YNlO/BQNRzoV50wwFsQATo0rhP09baZtbH18XF46+p+tpcl2G34st+sa0EvLtwcMPjp38t2+gyo2rAvsgD34fJdnsfDn16IzvfN9bxfYVEpJjy/COfN+B6b8wPft9R0cVi9W7sIzFu9FwCw/cBxz883Y8EmLN9+GADw2wEt1fMDvXno1Afn4U+v/gQAyN11BM99swm3fuAbJNbtDRzwNWjqN54MG/OMi1b7VsWBo8XI3Xk4rH1vend5yMnpnGLGgk3o++hXnCcnwRjgTYxb3fTkJLx5lTd4t8usjRcuO81vX0HXZhm2l+nNH36z3B7p4Cxzs4y/yu7wjbsDANhxUPuD/WptHgAttXP1bi1gHj4e2PxhDqzmpqP/rdqNodMWYMiTCwJq40YGkDlTaImeFWNs8+98PO+570P+DC6Xfe0Y42cswvgZiyrfEcCc3PDSViu7Q9l+4Hi1zhT6Zp32+7Enig7+cJxsvQ9zf93rqdjEEwO8ScsGtXDnqE54ZWIfNKqT6tnucgnG9mgasL8rDo2fX62NzUClyv7QcncexoDHvg46tcLMnM2W29ea7haszmFOQTVqc5/l7sECPQDsPHQCAx7/2rNPQVGpZ2HzY8VlAXOdGx/5yh2HfZY9LNU7nQ8ft76QuW0M8FXto7By/ds/ex73fngexjyb4/P60GkL8N3GwKmof/7tEDbsK4x5eSLl/dOwvsDH7Dwxf0d73PD2z7jurZ8r3zHGGOBNRAR/OasjWukDny7q0wqANh2AiCAt2YU/m2Z0NH656tVKtq1MoQZGxUqF0mqhewuK8NS8DZb7PPb5uoD8/O837se+Au8Fwaqs5m3mLKCPVuwK2BfQmmOm63cMG/OOYppfB7D5D7r7A18GHG8sr3jQ1Jm6ZndBlVfgMhw6VoJPVlqX3Wz17iP4YfOBqM+z65D3onHoeKknu6oyv3txMUb9I6fS/VZsP+TTrHTkeGlM1wU2Pm27En2Mpr3Pcvec1NlEP2w+gOzJc3x+X2OJAT6Eywe2AeC9vV/3yFg8OL675/UkPfe8dcN09G/b0JYyxCPAmxc0eS/E4hHDnlro89xoFzes2R3Ytl3V8n9suhB8vGJXwNiAYP76kTcf/+l5632aaKKJB+8v2Y7syXMwcdYS3PL+Ss+UEGZ5hUWeppVzpn+PS17W1gQ4crwUI55eiPVhBul4uOCFxRg/Y5HnO+v58Dyc8mDlg6qMqSF+/u1gwOR6ZkZt3fxRz8ndHbAtFtbuif/nuvdIEW7/YGWVRxq/9K12Zxzu73WkGOBDMAKBO8inVDctGS/96XTMurIvnrmoly1lqM7trP7+leO7uMma3QU4EMOaya0frMSDnwbPmjCybvx9vS4PVWmhWbO7AM9+pd1V5OrZQics/rD7Pfq1T3MTACxcn4dvN+Zjc/4xTP9mY8AxwURywyF6fbnAVAM3X+BC8Z8gr6SsIqC/Ztv+Y7jslR+x/2gx2k75HDO+2YTfvfgDJjy/CFe8tgTFZYGfhVUN/rHPvXdwE55fhImvLfE8v/+TX/G9RZNTOMrCXFfBysFjJRjy5DchL1ZWHv18LT5cscvnrjQadrfyMsCHYEzk1SA9Jeg+Y3o0RWadVLSoXwv/vLgXvrt7OH7664g4lbB6Gzf9O1z4wuLKdwwhkmUN754duHauwdwGf/N7K5A9eQ4mPL8o4A5jU14hftqiNa0cOVGKnA35GDf9O+z1y7d+aeFmjP5HTqUDwf48aym+1vtRtuYHn4IC0Jp1qtLccKqpBv7uT9Z3Ymt2F/jUOkvKfYPjbR+sRM+HfGvyT365Dos2HcAMPX32rR+9Hf85G/It70yMwGX8PMpn3IbWh2L0tQBaMoH/HWG4yqpwl/jA/1Zjx8ETngvdbweOBaR3WjFGjr++eBv2HNGa0z5dtTvqphZlU7cxA3wI3Zpl4G/ndsOzYdbOz+/VAq0apqNJRppn2zd3nIl+Fs03fbMbxKqYjmZu468KqwFWK3ccxoUv+l6ARj6Tg4tmak0rf3l3Oa4w1TIBb+D6cMUurN9XiLv+G3hR8U+D3KinpYaau/+nLQdwzvTv8VqI7Ki8wiLLJoFtB6wvHPmFxdiUpwXfY8VlyCsswrjp3+HmEHnoc37RsnzMAdlIJnhdXzYyzy8AWl2TjLsK46WqNNW9v2Q7jpg6z//0yk+45f2VnufmUd2GcBecKfG7+zhz2kL0ffSrSo8zj5M5WlSGPUdO4Ob3VuDGtwM7Ur/4xZs59eYP25BXGL/BWQzwIYgIrh7c1iejJlxzJg3GqxP7oF1WHdwyomPA6y/96fRYFJHC9Pc5ay23B2v7LCmrsMxSiaaC7V9LBrSA9/kvezyB6De9TX/N7gK8+cM2y1pkv0e/xlWvLw3Y/lnuHuSYasOGM6ctwMhncnDDWz+j+wNfot+jWvOR/xTSVqmb5nhcWQbS0eIyLN120LfMnhq89r+5lm2urR4vCcyUMvt11xFM/vAX3D3bO7fS934L2fs30XyWuxtd75/rk+EVTDiZcA9/ugYv+zU/ppjabc/+Rw4e+GQ1AGD3kcCMKvO6EPd/shp/ebfqA73CVX1m2XKY7s3roXvzegACf4lmXn56VBcNAKibmsQFNWKsqLQcl778Iy7t38az7W8f/xriiMhYNTO1/+vnnsfbpp7jqTHPXr4Ts5fvDNjfsHjzAcsOXv87DQA4rl885laSf33Tu4EL05RXKE9gryzAX/aK1rTSumE6cu4eDsDUBq8H82A1+NJyhSSX9Ws3vbvcc/EJ1fTh3wS/YJ12sft11xE0q5eGXg/Px8iuTfDYBT3QWL+7Li2vQLLb5fO36Z+Kanht0VYAwLVD23m2Jft1zM3TL5rGV739wHE0zki1HBFfoPdxlFco5MXoDjUY1uDjwP8PZGD7RkH3rSzlskOT4CNsKTorth/G8u2HcadpBs4PYjjJWmW1/n0FRRHdGQx5ckHVChQGc0AON8V0u+nCYxxy5ayl2HukCItMtW5zs1tFhQoa/M13FsbfkNU4DXMN/vkFmzwXSAUt1RbQxpNMnKXd/SzbdhAd7/0CP2w+4NPJGW4qKoCgC92XVyjkFRZh6LQFeGb+Bs/5zYyf5Ym56zzNdnZlejLAx4F/Fk6tZDcA4LaRnQL2DTUHDgCcd2rzmJUrnoyfuToqiGH+t5WtIeb3B7QF1KtbJrd5gJqRDhyOV7/f6lMzLS6rwIDHvw46yGfyh7k+AXrxZutMGmMyvdP/Htg+bv7spn3pHTcxeXauT2bOXr35ZLE+PmHRpv2W7ffhCBbg9xwp8nSmz8zZYjkmYfXuAlRUKM9oXzsxwMeBfxNNkh7xbxkZ2DbfPit0Dd3/dn/VA6PC7gRO1KyDDWunWKYVkubSl3/yGZVbHZSXKxwvKUNJWQXeWxL+3cwjn63BPbNzsaWSi5rhy9X7fGrwl778k2WNfu2egoinrq5QvtNslFUo5GzIx7NfaYP5RCpvvjKbk7vHM3+SfxONmdFJH8rRkjKfOyPW4E9ixnfXskEtfHX70KD7zbi0N578/akB29tm1vY89g/w9WolY0LvFqif7tu0M6Z74NQKiRrWnWTjNAGxEI/BZJUJ1gmcKDsOHUe3+79Ep/u+iPhYY/nIcPmnOV775rKA2UkPHCuxzFgCEPbIqcKiMlzx2hJPB/JzfrOmVuamd5dj6LQFeOnbLZ40yWi5RGydPsNzHtvPQJ4OtKy6qejQuK7lPtumnoNzT22OVg3Tse6RMdj46FhPe/yLfzoNbRqlY+blp8MiIQNAYDupyxXYnp8UotYBAPXTkzH1wlPC+ZEiEo9fZMONw9pHfMwWi1kwa7rcINM/22G13wjob9bl4bwZoSePM7Mjh3z93sKgGVZPzF3nM2grWua0WbuqGAzwcWDUGMLtrEpLdiPZ7UKjOtoAqySXC9/eNRyjujf1qcGba8b3n9fN5z0EgiX3+g64Stb3n3vrEM+26Zf09jxe8bezcXG/1jEPyNG8X7Q1pG5RzPAZbP6dmizckbCxMNEiAygSq3cVYP6afRhkWny9qkY/mxOwolos+c+8akzjEGsM8HFgNAFY5dz2y45sDhtzJ6w5cJ7fqwXm3eZt/vnLWR2QmuTt2Pz39QNRN02r0Ter512xyuj8PKtLY8/8IUZ57zunKwDgVou+gkhEE+CN1bUiVZ3W16X4eHr+Blz75jJbZvW0SiGNhY/9Jtv7eKU9AZ5/DXFg1LpdFpfTt67pF9HCypcPbIODx0rwwsLNGGMxhTEAdGxcJ2Cu+n5tG+Lta/rji1/2ICMtCY9M6IH2mbXRqanWZDTRNEvmqS3rIXfnEVw1qC2uGaLl/hpzsUQjmpkcBUDjuqkBoyYBoHOTulgfZErc2gzwFEPhzt8fqXjdNfKvIQ6MVhWrGnxqktunpu17YOCm1CQ37h7TBZcPbINGtX0HS1XWE9+hcR3crI+qvXyAd1DPtqnn+Oz35lX9kFdYHLNFMtwuQWadVOw/WowG6ck4FGTOdrO0ZDcu6tvKsiNsUIfMoAG+sjTTaLhEa2Y7v1dzfGJTTYvIDmyiiYPuzTMgAvzfsA5RHW9VAW5Wr1ZALq7RZn9W18ZRncdQPz0FnZpYdwYb5kwaHPS1u8d09nnudgla1NdGEL72576WmUJmrRrWwpSxXXH72Z3Qp43vnD1zJg3GDcPaBTkSyKybisEdMvH21f1xvT7ycEjHTLw6sU/Ic4ZiXJjvGdMFd43uHHLfqwe3xZmdsqI+F1EsMcDHQf30FGx9/BwM7pgZ0XEPn98D7bJqo2WDWpXvDCCzTiqW3DsCd4/u4tm2aPJZPm3z0fr0L4MDguRnN3uD/KSzvBcv/wuZ2yWeJqA2jWrj/F7NcfmANljxt7Ox5bFxnv22TT0Hc28dgu/uPgv10pMhIkhN9v6Kfj5pCLo3r4eMNG920NBOWT53ICluF96+pj8Gd8zEyG5NAABNMtIwomsTzz6hUlWtGAE+yS24aXgHPHZB8EwjAdAuy5vWekaIUcuRuLR/65i8z8nIPwWYwscmmmpscMdMfHPHsIiOaVw3zed5i/rhXRwqc0rLeigq9Q7CUgro0aIeHrvgFIgAF/dthelB8ordLsGFp7XEhae19Gx7ZEIPy327NPXtOzAGOXZpWhfdmmuvpSW7sfXxccgrLPakgtZKduNEablPZlGfNg0w7fenBnTYBktVDca4gzJGU17av3XQLJNaKW7PHDCA1h9yUd9WntkPrx/aLmDefMOUsV3wuMX84t/fMxwtG6SjYXoKZiyILHfbCaIdbXoymR+DSpgV22rwIvKaiOSJSOxmbaKEsmoqurR/a1zSr3XAepv/vWEgrh3SFkDlWTRL7x2JRZPPsnzNyHG+/1y/NFARNMlIQ5rfFAhuU3qliOAPfVp5Ol4fu+AU/PPiXj775z44KmTZAN8afGVqpbh9FhcZ3aMpzu/VAh/+3xl479oBmDKua9Bjrz8zMId/UIdGaNlAW0LyjlGBU1sAWhOglexG6ZbbzctO2uGy/q1juiB9JFMlnKw6VtIkGi07m2heBzDGxvenOEupZKCUWZ/shhilj6atLIsmq25q0DsNo+O4sgWbjZdDTf96af/WOL9XCwDAvy4/HXNvHYKMtGR0aBx8eoibhrf3BGzzOzfNSLPcv3ZKEtJTvDfGZ7TXmuVOa90g5CRz5iYuw68PjcasP/fzPBcRXNy3lc8+vVvXt+xcv+HM9ri4n3WzzjX6hdcuVw7KxguXnRZyn6ER9FOYv9MuTe0JhIn0+aQhle8UJdsCvFIqB8DJs94cVUpE8PmkIRjVrQk6W/yh9WpVH43rejN7jFvrqgycenB8dwxo1xC9W9cPud8zf+yJTk3qhD2p2ejuTT3NQf+5fiAev/AUn5rtpBEdsebh0bjj7M547tLeOL1NA9Q2Be7sTOvace3UJM/UEsY4An9pyYF/duN7NQ84pk5qUkBHuvlitPmxcZh9wxmWoyDvHt0Z1w1ph/E9AyenS7LK142hzDqpPtM/jPVL5/180hC8cWXfsPsVzL89k/QssCYZqXjr6n7WB5ic3a0JplXSqR9rI039PeHoFuQOLBYS3skqIteJyDIRWZafH7hoAVUv3ZpnYOYVfSwnW/r4pkFYcu9Iz3Pjj7wqAb5rswy8f93AgKYYf2N6NMO8286M6lwNaqfgkn6tfRZUv/3sTkhPSYLLJTirSxPMvvEMn7TRFy47HS9cdlpAsO6b3QDjTmmGf1zU02dsgdkPk0cEHGP0C1w5KHTt2vyebpcETWV16a+ZRyobmtZLw/RLeuPda/v7bP/96S0D9g3XX8d1wTN/7Iklfx2B+ukpPiOubxreAQ+YRlp3a54BEcElfUMHeOMCYK7tGz9tz5b1MaRj5XcB7bPq4A99WsW1k/rlK073mT8qkRIe4JVSM5VSfZRSfbKymF7mJP3bNcRFfVrhid/FtwYVDw1rp2DcKc0Cmq3aNKqNlCQXLujdMuiMgw1qp2DD38diRBctnbXYtJxgZRcoq4nbhnXW/m7euaZ/wGvBjO/ZHP2yG+I8vYbfo0UGnvpDT7x37QDPPj1b1gv7/a4b2h4XntbSs6CGcSc1tFMWerSohysHtcUbV/XDxzcN8hzTOCP0ojcD2zXCtqnn4LELTsGY7k3x2c2DPU11xuXj9Sv74pYRHdEkyHsZX8E9o7vgkiBNVtFo1bAW1j48xmc6EJdoP7eI2DIeIxrVoxTkSMluF574/alo1dC6OaM6uuPsTnj0AusMHyvRTqmQkuTCy1f0wZ/PyMY//KZ7XnbfSKx92Lr7yghwVwz0DlS7c1Rn/DDlLAzqoLX3W9Uerdq8k9wuPHdJb6x+aDQ+vFELvG30jtmmGWlhdfxdP7QdXr+yb8D2Vg3T8c41/fHSn7xt8Wd2ykKvVvU9z5tkpAXMl2RmdB7XSnHjpctPR48W9Tz9IcYEfsM6N8ZtZ3dCzt3D0bB2Ct69pj/qpnmb0prrfTv10pPxuGkivXvGeFOJzfz7OIJpWT8dtVLcPllrj15wCtY+on1v5u/AmFfpg+sG+PQ1vX11f3x1+5lhnS9aTJMkMrnZYv3cUB6Z0AO3j+rkWe80Ei6X+DQLGTIrWc5x6+PjfDqd3S7xzC/03xsGBgT4VQ+MQq1kN176drNPH4nBPL2D8Xhkt8aeKTTuO6erZzrjM9o3wuLNB/D5pCHYceg4RltMS20wLjih+Kf1GgZ3yEQ7i7URsvTyt/frGE9NcmP5384GoPVTrNh+GM9e1MuyD6JTkzq4cVh7FBSV4sWFmzG4Q6ZnnderBrfF+0sD551f8/BodLv/S8/z5y4NbPoyN0tN+0NP5GzIx7GScsyZNAQ/bjmA/u0a4dyezfCvb7U02UjHxUTDtgAvIu8BGAYgU0R2AnhAKfWqXecjSoRktwuN66bhnFOaYXSQuYFiLVRGUR+LyeuMsQKTwrh41auVjB+mnIWsOqm4Z7aW659hmnb6zav6oaxCIS3ZbWvnYLAO1N6tG+Dda/ujb4hJ+l6b2Bcrdx7G8M6BI7p/nDLCU8M3PsUB7RriztGdkZ7itlwz4fUr+yI9JQm3jOjoWUDE6iJsnta+TmoSVtw/CkdOlCKrbqpnZHh5nHP6bQvwSqlL7Hpvourm+UrSAk8mxt2AMQZBoLXvL912EEluF4JNnRStVg1r4WhRGa4d2g5dm2Wgf9uGIS9iRuppMA1qp1gGd0DrYDace2pzvLBwM0Z1b+oJwFYzUvZvq6W33nZ2JxSVlePb9b7JIF2a1sW6vYWo8Fu4JCXJ5bnjMEwa2RGvfL81ZPljiU00RGTNNEneoA6ZYTW5RCPnruEAKh/rEGvdmmcETLTXon4tvHV1P/Ru3QCDpn6DIydKfTq+p4ztiiljfdNf37t2AP4+Zy3+0KfyLCTzNBvxwABPRJaMHHy7R5LGO7BXxki/nH3jGVi4Pi/oAtuGBrVT8PQfe0bw/pnobepsthMDPBFZmjK2K+qnp0SdKXSy69C4TshRztF66+rw01mrigGeiCzVS0/G5LHW6YR0cmAePBGRQzHAExE5FAM8EZFDMcATETkUAzwRkUMxwBMRORQDPBGRQzHAExE5lCirBR0TRETyAfwW5eGZAPbHsDixwnJFhuWKDMsVGSeWq41SynK1pGoV4KtCRJYppfokuhz+WK7IsFyRYbkiU9PKxSYaIiKHYoAnInIoJwX4mYkuQBAsV2RYrsiwXJGpUeVyTBs8ERH5clINnoiITBjgiYgc6qQP8CIyRkTWi8gmEZkc53O3EpEFIrJWRFaLyC369gdFZJeIrNT/jTMdM0Uv63oRGW1j2baJyC/6+Zfp2xqKyHwR2aj/3yCe5RKRzqbPZKWIFIjIrYn6vETkNRHJE5FfTdsi/oxE5HT9s94kItOlimvQBSnXNBFZJyK5IvKRiNTXt2eLyAnTZ/dSnMsV8XcXp3J9YCrTNhFZqW+Py+cVIjbE9/dLKXXS/gPgBrAZQDsAKQBWAegWx/M3A3Ca/rgugA0AugF4EMCdFvt308uYCqCtXna3TWXbBiDTb9uTACbrjycDeCLe5fL77vYCaJOozwvAUACnAfi1Kp8RgCUABgIQAF8AGGtDuUYBSNIfP2EqV7Z5P7/3iUe5Iv7u4lEuv9efBnB/PD8vBI8Ncf39Otlr8P0AbFJKbVFKlQB4H8D58Tq5UmqPUmq5/rgQwFoALUIccj6A95VSxUqprQA2QfsZ4uV8AG/oj98AMCGB5RoBYLNSKtTIZVvLpZTKAXDQ4pxhf0Yi0gxAhlLqB6X9Nb5pOiZm5VJKzVNKlelPfwTQMtR7xKtcIST08zLotd0/Angv1HvEulwhYkNcf79O9gDfAsAO0/OdCB1gbSMi2QB6A/hJ3/QX/Xb6NdNtWDzLqwDME5GfReQ6fVsTpdQeQPsFBNA4AeUyXAzfP7pEf16GSD+jFvrjeJbxKmg1OUNbEVkhIt+KyBB9WzzLFcl3F+/PawiAfUqpjaZtcf28/GJDXH+/TvYAb9UWFfe8TxGpA2A2gFuVUgUAXgTQHkAvAHug3SIC8S3vIKXUaQDGArhJRIaG2Deun6OIpAAYD+A/+qbq8HlVJlhZ4v3Z3QugDMA7+qY9AForpXoDuB3AuyKSEcdyRfrdxfs7vQS+FYm4fl4WsSHorkHOX6VynewBfieAVqbnLQHsjmcBRCQZ2hf4jlLqQwBQSu1TSpUrpSoAvAxvs0LcyquU2q3/nwfgI70M+/RbPuOWNC/e5dKNBbBcKbVPL2PCPy+TSD+jnfBtLrGtjCIyEcC5AC7Tb9eh39If0B//DK3ttlO8yhXFdxfPzysJwIUAPjCVN26fl1VsQJx/v072AL8UQEcRaavXCi8G8L94nVxv33sVwFql1DOm7c1Mu10AwOjd/x+Ai0UkVUTaAugIrQMl1uWqLSJ1jcfQOuh+1c8/Ud9tIoBP4lkuE59aVaI/Lz8RfUb6bXahiAzQfx+uMB0TMyIyBsA9AMYrpY6btmeJiFt/3E4v15Y4liui7y5e5dKNBLBOKeVp4ojX5xUsNiDev1/R9hJXl38AxkHrod4M4N44n3swtNulXAAr9X/jALwF4Bd9+/8ANDMdc69e1vWoYvZAiHK1g9YjvwrAauNzAdAIwNcANur/N4xnufTzpAM4AKCeaVtCPi9oF5k9AEqh1ZSujuYzAtAHWmDbDGAG9BHiMS7XJmhttMbv2Uv6vr/Tv+NVAJYDOC/O5Yr4u4tHufTtrwO4wW/fuHxeCB4b4vr7xakKiIgc6mRvoiEioiAY4ImIHIoBnojIoRjgiYgcigGeiMihGODJMUTkqP5/tohcGuP3/qvf88WxfH8iOzDAkxNlA4gowBuDX0LwCfBKqTMiLBNR3DHAkxNNBTBEn+/7NhFxizaf+lJ9UqzrAUBEhulzdr8LbbAORORjfYK21cYkbSIyFUAt/f3e0bcZdwuiv/evos3ZfZHpvReKyH9Fm8f9HX0kIkRkqois0cvyVNw/HaoxkhJdACIbTIY2R/m5AKAH6iNKqb4ikgpgkYjM0/ftB6CH0qZoBYCrlFIHRaQWgKUiMlspNVlE/qKU6mVxrguhTbTVE0CmfkyO/lpvAN2hzR2yCMAgEVkDbUh/F6WUEn3hDiI7sAZPNcEoAFeItqrPT9CGi3fUX1tiCu4AMElEVkGbc72Vab9gBgN4T2kTbu0D8C2Avqb33qm0ibhWQms6KgBQBOAVEbkQwPHAtySKDQZ4qgkEwM1KqV76v7ZKKaMGf8yzk8gwaBNUDVRK9QSwAkBaGO8dTLHpcTm0FZnKoN01zIa2cMPcCH4OoogwwJMTFUJbJs3wJYAb9elbISKd9Fk2/dUDcEgpdVxEugAYYHqt1DjeTw6Ai/R2/ixoy8cFnfFSnx+8nlLqcwC3QmveIbIF2+DJiXIBlOlNLa8D+Ce05pHlekdnPqyXPZsL4AYRyYU2o9+PptdmAsgVkeVKqctM2z+Ctl7mKmizB96tlNqrXyCs1AXwiYikQav93xbVT0gUBs4mSUTkUGyiISJyKAZ4IiKHYoAnInIoBngiIodigCcicigGeCIih2KAJyJyqP8HKdebIvc9lrcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [1:06:33<00:00,  2.00s/it]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAv7UlEQVR4nO3dd5wTZf4H8M832cYCS9ull6VXBZQqRRCkqYhesZ1ydv15Ylc4PeupKOp5iOWwYC93h+UURVDBVVABKav0Kp1d6i5l+/P7Y2aSSTLJJtlMssx+3q8XL5LJTObZZPc7zzzP93keUUqBiIicx5XoAhARkT0Y4ImIHIoBnojIoRjgiYgcigGeiMihkhJdALPMzEyVnZ2d6GIQEZ00fv755/1KqSyr16pVgM/OzsayZcsSXQwiopOGiPwW7DU20RARORQDPBGRQzHAExE5FAM8EZFDMcATETmUrVk0IrINQCGAcgBlSqk+dp6PiIi84pEmOVwptT8O5yEiIhNHNNHsKyjC/DX7El0MIqJqxe4ArwDME5GfReQ6qx1E5DoRWSYiy/Lz86M6ycUzf8S1by5DRQXnticiMtgd4AcppU4DMBbATSIy1H8HpdRMpVQfpVSfrCzL0baV2nnoOACgpLyiKmUlInIUWwO8Umq3/n8egI8A9LPjPMlu7ccoLmOAJyIy2BbgRaS2iNQ1HgMYBeBXO87lDfDldrw9EdFJyc4smiYAPhIR4zzvKqXm2nEiI8CXsAZPRORhW4BXSm0B0NOu9zdLdgsABngiIjNHpEl6avDsZCUi8nBIgNdq8MWlDPBERAaHBHjW4ImI/DkrwLMNnojIwyEBXm+iYZokEZGHIwK826UF+LJyTlVARGRwRIAXaAGeU9EQEXk5IsDr8R1KMcITERkcEeD1+M4aPBGRiSMCvKGCNXgiIg9HBHjRq/AM8EREXo4I8AbGdyIiL0cEeG8WDSM8EZHBEQHewE5WIiIvRwR4tsETEQVyRIA3MA+eiMjLEQHeW4NPbDmIiKoTZwR4drISEQVwRIA3sAZPROTliADvaaJhhCci8nBEgP9u434AbKIhIjJzRIA3sAJPROTlqADPNEkiIi9HBXg20RAReTkswCe6BERE1YfDAjwjPBGRwVEBnvGdiMjLUQGeefBERF6OCPBvX90fANvgiYjMHBHgB3VoBIBt8EREZo4I8CICEebBExGZOSLAA4BLhE00REQmDgrwbKIhIjJzTIAXEZQzwBMReTgmwLuEefBERGaOCfBuEebBExGZOCbAs5OViMiXYwK8sJOViMiH7QFeRNwiskJEPrPzPC6XMA+eiMgkHjX4WwCstfskbKIhIvJla4AXkZYAzgHwip3nAZgHT0Tkz+4a/LMA7gZQEWwHEblORJaJyLL8/PyoT7T/aAl+3XUk6uOJiJzGtgAvIucCyFNK/RxqP6XUTKVUH6VUn6ysrCqdc9VOBngiIkOSje89CMB4ERkHIA1Ahoi8rZT6kx0nS0lyoV6tZDvemojopGRbDV4pNUUp1VIplQ3gYgDf2BXcAWBAu0ZoUb+WXW9PRHTScUwePDtZiYh82dlE46GUWghgoZ3ncIugnHmSREQezqnBu5gHT0Rk5pwAL1x0m4jIzDEB3u0StsETEZk4JsBzwQ8iIl+OCfCcD56IyJdjArxLgN2Hi3DwWEmii0JEVC04J8C7BCXlFTjtkfmJLgoRUbXgnAAvkugiEBFVK44J8G5TgN+UV5jAkhARVQ+OCfAulzfAj3wmJ4ElISKqHhwT4FPcbKIhIjJzTIBv1TA90UUgIqpWHBPgV+w4nOgiEBFVK44J8BzkRETkyzEB3tzJSkRETgrwzIMnIvLhmADPJBoiIl+OCfCswRMR+XJMgBe/AK84dTAR1XCOCfBuv5+ESTVEVNM5KMD71uC5uhMR1XSOCfD+bfDlrMITUQ3n2ADPGjwR1XSOCfD+TTSswRNRTeeYAJ9RK9nneUVFggpCRFRNOCbA3zS8vc/zcjbREFEN55gAn5rkRpemdT3Pi8vKE1gaIqLEc0yAB4D2WXU8jx/5bE0CS0JElHiOCvDmGvySrYcSWBIiosRzVIC/cZi3HX7/0WI89/XGBJaGiCixHBXgk/zmK3h6/oYElYSIKPEcFeCJiMiLAZ6IyKEY4ImIHIoBnojIoRjgiYgcigGeiMihGOCJiBwqrAAvIrVFxKU/7iQi40UkubLjiIgoccKtwecASBORFgC+BnAlgNdDHSAiaSKyRERWichqEXmoakUNz/RLesfjNERE1V64AV6UUscBXAjgOaXUBQC6VXJMMYCzlFI9AfQCMEZEBkRd0jCN79nc8zjJbxEQIqKaJOwALyIDAVwGYI6+LSnUAUpzVH+arP+L6yTtZRUKx4rL4nlKIqJqI9wAfyuAKQA+UkqtFpF2ABZUdpCIuEVkJYA8APOVUj9Z7HOdiCwTkWX5+fnhlzxM89fsi/l7EhGdDMIK8Eqpb5VS45VST+idrfuVUpPCOK5cKdULQEsA/USkh8U+M5VSfZRSfbKysiItv6Xhnb3vI2ylIaIaKtwsmndFJENEagNYA2C9iNwV7kmUUocBLAQwJppCRmrGpafF4zRERNVauE003ZRSBQAmAPgcQGsAl4c6QESyRKS+/rgWgJEA1kVd0gjUTvV2Dwir8ERUQ4XsKDVJ1vPeJwCYoZQqFZHKOkybAXhDRNzQLiT/Vkp9Fn1RiYgoEuEG+H8B2AZgFYAcEWkDoCDUAUqpXABMSiciSpBwO1mnK6VaKKXG6emPvwEYbnPZquSWER0BMBeeiGqucDtZ64nIM0Y6o4g8DaC2zWWrkvN6NgMAlFfENfWeiKjaCLeT9TUAhQD+qP8rADDLrkLFQpJL+9Fufm9FgktCRJQY4bbBt1dK/c70/CF9AFO1lZzkvXYdOFqMRnVSE1gaIqL4C7cGf0JEBhtPRGQQgBP2FCk2kk1t71e9vjSBJSEiSoxwa/A3AHhTROrpzw8BmGhPkWIj2e29dm0/eDyBJSEiSoywArxSahWAniKSoT8vEJFbAeTaWLYqSXJ7a/AuDnYiohooohWdlFIF+ohWALjdhvLEjLkG72KqJBHVQFVZsq9aR81UUydrtS4oEZFNqhLgq3WCuYhgSMfMRBeDiChhQrbBi0ghrAO5AKhlS4liiIOciKgmq2xVprrxKogdyvQAzzBPRDVRVZpoqj3W4ImoJqsRAT6/sDjBJSEiir8aEeABoIK1eSKqYRwd4Gsluz2PX/5uSwJLQkQUf44O8NMv8a438tGKXQksCRFR/Dk6wDetl4YB7RoCAApOlLKZhohqFEcHeABw69MU7D5ShGnz1ie4NERE8eP4AG+eaOyNxdsSVxAiojhzfIB3myYaKytnEw0R1RzOD/CmGnxJeUUCS0JEFF+OD/AdGtdJdBGIiBLC8QH+ztGdE10EIqKEcHyANy/8AQBrdhcE2ZOIyFkcH+D9XfvmskQXgYgoLmpEgO/S1Dvr8a7DJxJYEiKi+KkRAX5g+0aJLgIRUdzViAAvXJWViGqgGhHgh3byXZtVKYWSMubEE5Gz1YgAP6xzY6SYsmmmzl2HTvd9geKy8gSWiojIXjUiwANA3TTv8rP/+labG76ohLV4InKuGhPgU5MsflQ2zRORg9WYAJ9iFeD1ucdeWLgJK3ccjmt5iIjsVmMCfOOMtIBtFUqL8E/OXY8Jzy+Kd5GIiGxVYwL8oxN6BGwrV5w+mIicq8YE+NqpSQHbKhjgicjBakyAT3IH9qhWVGg58URETmRbgBeRViKyQETWishqEbnFrnOFo3Fd6zb4ctNC3Az2ROQkdtbgywDcoZTqCmAAgJtEpJuN56tUsl8tvrxC+bTDV9gc3xdv2o+jxWX2noSISGdbgFdK7VFKLdcfFwJYC6CFXecLR6cmdX2eVyiFigrf53bYe6QIuTsP49JXfsKt76+w5RxERP4Cex5tICLZAHoD+MnitesAXAcArVu3trUcr07siwGPf+15fua0hUhL9l7jVu8uQK9W9bFgfR66N8+wbNaJhvmc6/cVxuQ9iYgqY3snq4jUATAbwK1KqYDllJRSM5VSfZRSfbKysmwtS1bd1IBtRaXeKvyE5xfhybnrcOWspbh45o+2loWIyG62BngRSYYW3N9RSn1o57nC4XZVPjfBCws3AwC25B+zuzhERLayM4tGALwKYK1S6hm7zmOn9XsL8cvOI4kuBhFRVOyswQ8CcDmAs0Rkpf5vnI3ni7nRz+bgvBnfAwBeX7QVV85agm835Ce4VERE4bGtk1Up9T0cNF/jg5+uAQAsWJ+P16/si2GdG2NTXiH2Hy3BgHbhLwnIVHsiipcaM5LVkHPX8IiPqfBLkP/zrKXYnH8UI5/JiaozdtfhEzhmyoc/dKwEhUWlEb8PEVEoNS7At26UHvExS7YdjNn5lQIGTf0Gv3/pB8+23o/Mx8DHv4nZOYiIgBoY4KNRVBq4tJ9bqtb6tHZPgc+dAUe4ElGsMcCH4c7/5AZsi8VUw8ctLhxE4VJK4eu1+wKaEIkMNTLA109Pxg1ntg97//1HiwO2jXj6W8/j7Mlzgh67aNP+oK9xumKqio9W7MLVbyzDO0u2J7ooVE3VyAC/8v5RmDy2iy3vfaKkHOc+9x0ue0XrfH3luy1B91Vhrvm9ZOtBvLF4WwxKR06y50gRAGD34RMJLglVV3GZi6Ym6Xr/XJ/nEqKtvrQivAj/x39pHbITz8j22b5i+yF0b17Per1ZIqrxGBkSaM/hIsxfsy+qYzfnH8UFLyzGI5+tiXGpqq6otBwnSti/QJRoNTrAP37hKTF7r+zJc7Byx2GfbV/8sgcHLNrvDefN+B7XvrksqvMdOFoCAFizJ2D+toTr+/evAu5kiCj+anQTzSX9WmPDvkLMWrQtJu834flFPs9vfGd5wD6xWjXKWIkqVLqmUipkE5FdCpnySVQt1OgaPADcO64rPrlpUNzOt68weI0+EkYGjkv/BldsP4TsyXOw4+BxAMDGfYVoO+XzqJuAThZKqRq/1KJj5gOhmKvxAT7J7ULrhpGPbo1WeYic5ZvfW4HjJeHVfo0A/+OWg/h+43789+edAICF+mRoRnPRF7/uqUJpq7++j36FM6Z6RwGv3HEYPR+ah0PHShJYqtg4UVJuOciOKFw1PsADsRm0FAufrtqND5fvwrb9weei/8f8DcjdedjnQnHL+yuQ7Na+ytIyLTPHpTfNfLh8V9gXjVj4YGl8c7L3Hy3xpAsCwAsLNuHIiVL8tDV200skStf756LXw/MSXQw6iTHAA6iTWn26It7+8TcMe2ohnl+wCav8Om1f+nYz/vn1RoyfschnkFSFUp4FxUvLK7Bk60Hc8Z9Vntf3mgJguPYfLY6q9njP7F8iPoaCM684RhQpBngAaclubJt6DqbYNPgpEuv2amu2TvtyPc7367Sd+sU6z+OycnOAhycXvrS8Avd/8qvPcVb3J9mT5yB78hwUl5VbtmH3+ftXuPCFxfjPsh3R/ihks5re90CVY4A3GXdKMwBAi/q1ElySyplr8EopJOm9rSXlgZkzxq6d7/sCD326GkdOeKcm7nzfXDw1b73P/s/oz9fsKcBd/83FriqOlNyUV+jJiy8pS0yN9HhJGR76dLUj2ub9JSBRKipl5RX486wlWBqD2VmLy8pxz39zkVcQ+d1pTcIAb2IEQhEgNcTo0AV3DotPgUIoN8XJgqIy/PPrjQC0Grz/3/vMnM3YcfA4issqMGvRNvzhpcU+r/972U6f59O/2eTzvMx0srLyioBaf2l5RdClDY+cKMXIZ3LQ9f652JR3FJ3u+wL/W7U7YL+nvlyPC1/Q7ljyCouQPXkO/r10h/7zlXouDBUVCk/PWx9Ws9Ouwyc8w/hnLdqGWYu2ofcj87Ep72ilx1LVVFQo5PtljO0tKMLC9fm45b0VVX7/eav34YNlO/BQNRzoV50wwFsQATo0rhP09baZtbH18XF46+p+tpcl2G34st+sa0EvLtwcMPjp38t2+gyo2rAvsgD34fJdnsfDn16IzvfN9bxfYVEpJjy/COfN+B6b8wPft9R0cVi9W7sIzFu9FwCw/cBxz883Y8EmLN9+GADw2wEt1fMDvXno1Afn4U+v/gQAyN11BM99swm3fuAbJNbtDRzwNWjqN54MG/OMi1b7VsWBo8XI3Xk4rH1vend5yMnpnGLGgk3o++hXnCcnwRjgTYxb3fTkJLx5lTd4t8usjRcuO81vX0HXZhm2l+nNH36z3B7p4Cxzs4y/yu7wjbsDANhxUPuD/WptHgAttXP1bi1gHj4e2PxhDqzmpqP/rdqNodMWYMiTCwJq40YGkDlTaImeFWNs8+98PO+570P+DC6Xfe0Y42cswvgZiyrfEcCc3PDSViu7Q9l+4Hi1zhT6Zp32+7Enig7+cJxsvQ9zf93rqdjEEwO8ScsGtXDnqE54ZWIfNKqT6tnucgnG9mgasL8rDo2fX62NzUClyv7QcncexoDHvg46tcLMnM2W29ea7haszmFOQTVqc5/l7sECPQDsPHQCAx7/2rNPQVGpZ2HzY8VlAXOdGx/5yh2HfZY9LNU7nQ8ft76QuW0M8FXto7By/ds/ex73fngexjyb4/P60GkL8N3GwKmof/7tEDbsK4x5eSLl/dOwvsDH7Dwxf0d73PD2z7jurZ8r3zHGGOBNRAR/OasjWukDny7q0wqANh2AiCAt2YU/m2Z0NH656tVKtq1MoQZGxUqF0mqhewuK8NS8DZb7PPb5uoD8/O837se+Au8Fwaqs5m3mLKCPVuwK2BfQmmOm63cMG/OOYppfB7D5D7r7A18GHG8sr3jQ1Jm6ZndBlVfgMhw6VoJPVlqX3Wz17iP4YfOBqM+z65D3onHoeKknu6oyv3txMUb9I6fS/VZsP+TTrHTkeGlM1wU2Pm27En2Mpr3Pcvec1NlEP2w+gOzJc3x+X2OJAT6Eywe2AeC9vV/3yFg8OL675/UkPfe8dcN09G/b0JYyxCPAmxc0eS/E4hHDnlro89xoFzes2R3Ytl3V8n9suhB8vGJXwNiAYP76kTcf/+l5632aaKKJB+8v2Y7syXMwcdYS3PL+Ss+UEGZ5hUWeppVzpn+PS17W1gQ4crwUI55eiPVhBul4uOCFxRg/Y5HnO+v58Dyc8mDlg6qMqSF+/u1gwOR6ZkZt3fxRz8ndHbAtFtbuif/nuvdIEW7/YGWVRxq/9K12Zxzu73WkGOBDMAKBO8inVDctGS/96XTMurIvnrmoly1lqM7trP7+leO7uMma3QU4EMOaya0frMSDnwbPmjCybvx9vS4PVWmhWbO7AM9+pd1V5OrZQics/rD7Pfq1T3MTACxcn4dvN+Zjc/4xTP9mY8AxwURywyF6fbnAVAM3X+BC8Z8gr6SsIqC/Ztv+Y7jslR+x/2gx2k75HDO+2YTfvfgDJjy/CFe8tgTFZYGfhVUN/rHPvXdwE55fhImvLfE8v/+TX/G9RZNTOMrCXFfBysFjJRjy5DchL1ZWHv18LT5cscvnrjQadrfyMsCHYEzk1SA9Jeg+Y3o0RWadVLSoXwv/vLgXvrt7OH7664g4lbB6Gzf9O1z4wuLKdwwhkmUN754duHauwdwGf/N7K5A9eQ4mPL8o4A5jU14hftqiNa0cOVGKnA35GDf9O+z1y7d+aeFmjP5HTqUDwf48aym+1vtRtuYHn4IC0Jp1qtLccKqpBv7uT9Z3Ymt2F/jUOkvKfYPjbR+sRM+HfGvyT365Dos2HcAMPX32rR+9Hf85G/It70yMwGX8PMpn3IbWh2L0tQBaMoH/HWG4yqpwl/jA/1Zjx8ETngvdbweOBaR3WjFGjr++eBv2HNGa0z5dtTvqphZlU7cxA3wI3Zpl4G/ndsOzYdbOz+/VAq0apqNJRppn2zd3nIl+Fs03fbMbxKqYjmZu468KqwFWK3ccxoUv+l6ARj6Tg4tmak0rf3l3Oa4w1TIBb+D6cMUurN9XiLv+G3hR8U+D3KinpYaau/+nLQdwzvTv8VqI7Ki8wiLLJoFtB6wvHPmFxdiUpwXfY8VlyCsswrjp3+HmEHnoc37RsnzMAdlIJnhdXzYyzy8AWl2TjLsK46WqNNW9v2Q7jpg6z//0yk+45f2VnufmUd2GcBecKfG7+zhz2kL0ffSrSo8zj5M5WlSGPUdO4Ob3VuDGtwM7Ur/4xZs59eYP25BXGL/BWQzwIYgIrh7c1iejJlxzJg3GqxP7oF1WHdwyomPA6y/96fRYFJHC9Pc5ay23B2v7LCmrsMxSiaaC7V9LBrSA9/kvezyB6De9TX/N7gK8+cM2y1pkv0e/xlWvLw3Y/lnuHuSYasOGM6ctwMhncnDDWz+j+wNfot+jWvOR/xTSVqmb5nhcWQbS0eIyLN120LfMnhq89r+5lm2urR4vCcyUMvt11xFM/vAX3D3bO7fS934L2fs30XyWuxtd75/rk+EVTDiZcA9/ugYv+zU/ppjabc/+Rw4e+GQ1AGD3kcCMKvO6EPd/shp/ebfqA73CVX1m2XKY7s3roXvzegACf4lmXn56VBcNAKibmsQFNWKsqLQcl778Iy7t38az7W8f/xriiMhYNTO1/+vnnsfbpp7jqTHPXr4Ts5fvDNjfsHjzAcsOXv87DQA4rl885laSf33Tu4EL05RXKE9gryzAX/aK1rTSumE6cu4eDsDUBq8H82A1+NJyhSSX9Ws3vbvcc/EJ1fTh3wS/YJ12sft11xE0q5eGXg/Px8iuTfDYBT3QWL+7Li2vQLLb5fO36Z+Kanht0VYAwLVD23m2Jft1zM3TL5rGV739wHE0zki1HBFfoPdxlFco5MXoDjUY1uDjwP8PZGD7RkH3rSzlskOT4CNsKTorth/G8u2HcadpBs4PYjjJWmW1/n0FRRHdGQx5ckHVChQGc0AON8V0u+nCYxxy5ayl2HukCItMtW5zs1tFhQoa/M13FsbfkNU4DXMN/vkFmzwXSAUt1RbQxpNMnKXd/SzbdhAd7/0CP2w+4NPJGW4qKoCgC92XVyjkFRZh6LQFeGb+Bs/5zYyf5Ym56zzNdnZlejLAx4F/Fk6tZDcA4LaRnQL2DTUHDgCcd2rzmJUrnoyfuToqiGH+t5WtIeb3B7QF1KtbJrd5gJqRDhyOV7/f6lMzLS6rwIDHvw46yGfyh7k+AXrxZutMGmMyvdP/Htg+bv7spn3pHTcxeXauT2bOXr35ZLE+PmHRpv2W7ffhCBbg9xwp8nSmz8zZYjkmYfXuAlRUKM9oXzsxwMeBfxNNkh7xbxkZ2DbfPit0Dd3/dn/VA6PC7gRO1KyDDWunWKYVkubSl3/yGZVbHZSXKxwvKUNJWQXeWxL+3cwjn63BPbNzsaWSi5rhy9X7fGrwl778k2WNfu2egoinrq5QvtNslFUo5GzIx7NfaYP5RCpvvjKbk7vHM3+SfxONmdFJH8rRkjKfOyPW4E9ixnfXskEtfHX70KD7zbi0N578/akB29tm1vY89g/w9WolY0LvFqif7tu0M6Z74NQKiRrWnWTjNAGxEI/BZJUJ1gmcKDsOHUe3+79Ep/u+iPhYY/nIcPmnOV775rKA2UkPHCuxzFgCEPbIqcKiMlzx2hJPB/JzfrOmVuamd5dj6LQFeOnbLZ40yWi5RGydPsNzHtvPQJ4OtKy6qejQuK7lPtumnoNzT22OVg3Tse6RMdj46FhPe/yLfzoNbRqlY+blp8MiIQNAYDupyxXYnp8UotYBAPXTkzH1wlPC+ZEiEo9fZMONw9pHfMwWi1kwa7rcINM/22G13wjob9bl4bwZoSePM7Mjh3z93sKgGVZPzF3nM2grWua0WbuqGAzwcWDUGMLtrEpLdiPZ7UKjOtoAqySXC9/eNRyjujf1qcGba8b3n9fN5z0EgiX3+g64Stb3n3vrEM+26Zf09jxe8bezcXG/1jEPyNG8X7Q1pG5RzPAZbP6dmizckbCxMNEiAygSq3cVYP6afRhkWny9qkY/mxOwolos+c+8akzjEGsM8HFgNAFY5dz2y45sDhtzJ6w5cJ7fqwXm3eZt/vnLWR2QmuTt2Pz39QNRN02r0Ter512xyuj8PKtLY8/8IUZ57zunKwDgVou+gkhEE+CN1bUiVZ3W16X4eHr+Blz75jJbZvW0SiGNhY/9Jtv7eKU9AZ5/DXFg1LpdFpfTt67pF9HCypcPbIODx0rwwsLNGGMxhTEAdGxcJ2Cu+n5tG+Lta/rji1/2ICMtCY9M6IH2mbXRqanWZDTRNEvmqS3rIXfnEVw1qC2uGaLl/hpzsUQjmpkcBUDjuqkBoyYBoHOTulgfZErc2gzwFEPhzt8fqXjdNfKvIQ6MVhWrGnxqktunpu17YOCm1CQ37h7TBZcPbINGtX0HS1XWE9+hcR3crI+qvXyAd1DPtqnn+Oz35lX9kFdYHLNFMtwuQWadVOw/WowG6ck4FGTOdrO0ZDcu6tvKsiNsUIfMoAG+sjTTaLhEa2Y7v1dzfGJTTYvIDmyiiYPuzTMgAvzfsA5RHW9VAW5Wr1ZALq7RZn9W18ZRncdQPz0FnZpYdwYb5kwaHPS1u8d09nnudgla1NdGEL72576WmUJmrRrWwpSxXXH72Z3Qp43vnD1zJg3GDcPaBTkSyKybisEdMvH21f1xvT7ycEjHTLw6sU/Ic4ZiXJjvGdMFd43uHHLfqwe3xZmdsqI+F1EsMcDHQf30FGx9/BwM7pgZ0XEPn98D7bJqo2WDWpXvDCCzTiqW3DsCd4/u4tm2aPJZPm3z0fr0L4MDguRnN3uD/KSzvBcv/wuZ2yWeJqA2jWrj/F7NcfmANljxt7Ox5bFxnv22TT0Hc28dgu/uPgv10pMhIkhN9v6Kfj5pCLo3r4eMNG920NBOWT53ICluF96+pj8Gd8zEyG5NAABNMtIwomsTzz6hUlWtGAE+yS24aXgHPHZB8EwjAdAuy5vWekaIUcuRuLR/65i8z8nIPwWYwscmmmpscMdMfHPHsIiOaVw3zed5i/rhXRwqc0rLeigq9Q7CUgro0aIeHrvgFIgAF/dthelB8ordLsGFp7XEhae19Gx7ZEIPy327NPXtOzAGOXZpWhfdmmuvpSW7sfXxccgrLPakgtZKduNEablPZlGfNg0w7fenBnTYBktVDca4gzJGU17av3XQLJNaKW7PHDCA1h9yUd9WntkPrx/aLmDefMOUsV3wuMX84t/fMxwtG6SjYXoKZiyILHfbCaIdbXoymR+DSpgV22rwIvKaiOSJSOxmbaKEsmoqurR/a1zSr3XAepv/vWEgrh3SFkDlWTRL7x2JRZPPsnzNyHG+/1y/NFARNMlIQ5rfFAhuU3qliOAPfVp5Ol4fu+AU/PPiXj775z44KmTZAN8afGVqpbh9FhcZ3aMpzu/VAh/+3xl479oBmDKua9Bjrz8zMId/UIdGaNlAW0LyjlGBU1sAWhOglexG6ZbbzctO2uGy/q1juiB9JFMlnKw6VtIkGi07m2heBzDGxvenOEupZKCUWZ/shhilj6atLIsmq25q0DsNo+O4sgWbjZdDTf96af/WOL9XCwDAvy4/HXNvHYKMtGR0aBx8eoibhrf3BGzzOzfNSLPcv3ZKEtJTvDfGZ7TXmuVOa90g5CRz5iYuw68PjcasP/fzPBcRXNy3lc8+vVvXt+xcv+HM9ri4n3WzzjX6hdcuVw7KxguXnRZyn6ER9FOYv9MuTe0JhIn0+aQhle8UJdsCvFIqB8DJs94cVUpE8PmkIRjVrQk6W/yh9WpVH43rejN7jFvrqgycenB8dwxo1xC9W9cPud8zf+yJTk3qhD2p2ejuTT3NQf+5fiAev/AUn5rtpBEdsebh0bjj7M547tLeOL1NA9Q2Be7sTOvace3UJM/UEsY4An9pyYF/duN7NQ84pk5qUkBHuvlitPmxcZh9wxmWoyDvHt0Z1w1ph/E9AyenS7LK142hzDqpPtM/jPVL5/180hC8cWXfsPsVzL89k/QssCYZqXjr6n7WB5ic3a0JplXSqR9rI039PeHoFuQOLBYS3skqIteJyDIRWZafH7hoAVUv3ZpnYOYVfSwnW/r4pkFYcu9Iz3Pjj7wqAb5rswy8f93AgKYYf2N6NMO8286M6lwNaqfgkn6tfRZUv/3sTkhPSYLLJTirSxPMvvEMn7TRFy47HS9cdlpAsO6b3QDjTmmGf1zU02dsgdkPk0cEHGP0C1w5KHTt2vyebpcETWV16a+ZRyobmtZLw/RLeuPda/v7bP/96S0D9g3XX8d1wTN/7Iklfx2B+ukpPiOubxreAQ+YRlp3a54BEcElfUMHeOMCYK7tGz9tz5b1MaRj5XcB7bPq4A99WsW1k/rlK073mT8qkRIe4JVSM5VSfZRSfbKymF7mJP3bNcRFfVrhid/FtwYVDw1rp2DcKc0Cmq3aNKqNlCQXLujdMuiMgw1qp2DD38diRBctnbXYtJxgZRcoq4nbhnXW/m7euaZ/wGvBjO/ZHP2yG+I8vYbfo0UGnvpDT7x37QDPPj1b1gv7/a4b2h4XntbSs6CGcSc1tFMWerSohysHtcUbV/XDxzcN8hzTOCP0ojcD2zXCtqnn4LELTsGY7k3x2c2DPU11xuXj9Sv74pYRHdEkyHsZX8E9o7vgkiBNVtFo1bAW1j48xmc6EJdoP7eI2DIeIxrVoxTkSMluF574/alo1dC6OaM6uuPsTnj0AusMHyvRTqmQkuTCy1f0wZ/PyMY//KZ7XnbfSKx92Lr7yghwVwz0DlS7c1Rn/DDlLAzqoLX3W9Uerdq8k9wuPHdJb6x+aDQ+vFELvG30jtmmGWlhdfxdP7QdXr+yb8D2Vg3T8c41/fHSn7xt8Wd2ykKvVvU9z5tkpAXMl2RmdB7XSnHjpctPR48W9Tz9IcYEfsM6N8ZtZ3dCzt3D0bB2Ct69pj/qpnmb0prrfTv10pPxuGkivXvGeFOJzfz7OIJpWT8dtVLcPllrj15wCtY+on1v5u/AmFfpg+sG+PQ1vX11f3x1+5lhnS9aTJMkMrnZYv3cUB6Z0AO3j+rkWe80Ei6X+DQLGTIrWc5x6+PjfDqd3S7xzC/03xsGBgT4VQ+MQq1kN176drNPH4nBPL2D8Xhkt8aeKTTuO6erZzrjM9o3wuLNB/D5pCHYceg4RltMS20wLjih+Kf1GgZ3yEQ7i7URsvTyt/frGE9NcmP5384GoPVTrNh+GM9e1MuyD6JTkzq4cVh7FBSV4sWFmzG4Q6ZnnderBrfF+0sD551f8/BodLv/S8/z5y4NbPoyN0tN+0NP5GzIx7GScsyZNAQ/bjmA/u0a4dyezfCvb7U02UjHxUTDtgAvIu8BGAYgU0R2AnhAKfWqXecjSoRktwuN66bhnFOaYXSQuYFiLVRGUR+LyeuMsQKTwrh41auVjB+mnIWsOqm4Z7aW659hmnb6zav6oaxCIS3ZbWvnYLAO1N6tG+Dda/ujb4hJ+l6b2Bcrdx7G8M6BI7p/nDLCU8M3PsUB7RriztGdkZ7itlwz4fUr+yI9JQm3jOjoWUDE6iJsnta+TmoSVtw/CkdOlCKrbqpnZHh5nHP6bQvwSqlL7Hpvourm+UrSAk8mxt2AMQZBoLXvL912EEluF4JNnRStVg1r4WhRGa4d2g5dm2Wgf9uGIS9iRuppMA1qp1gGd0DrYDace2pzvLBwM0Z1b+oJwFYzUvZvq6W33nZ2JxSVlePb9b7JIF2a1sW6vYWo8Fu4JCXJ5bnjMEwa2RGvfL81ZPljiU00RGTNNEneoA6ZYTW5RCPnruEAKh/rEGvdmmcETLTXon4tvHV1P/Ru3QCDpn6DIydKfTq+p4ztiiljfdNf37t2AP4+Zy3+0KfyLCTzNBvxwABPRJaMHHy7R5LGO7BXxki/nH3jGVi4Pi/oAtuGBrVT8PQfe0bw/pnobepsthMDPBFZmjK2K+qnp0SdKXSy69C4TshRztF66+rw01mrigGeiCzVS0/G5LHW6YR0cmAePBGRQzHAExE5FAM8EZFDMcATETkUAzwRkUMxwBMRORQDPBGRQzHAExE5lCirBR0TRETyAfwW5eGZAPbHsDixwnJFhuWKDMsVGSeWq41SynK1pGoV4KtCRJYppfokuhz+WK7IsFyRYbkiU9PKxSYaIiKHYoAnInIoJwX4mYkuQBAsV2RYrsiwXJGpUeVyTBs8ERH5clINnoiITBjgiYgc6qQP8CIyRkTWi8gmEZkc53O3EpEFIrJWRFaLyC369gdFZJeIrNT/jTMdM0Uv63oRGW1j2baJyC/6+Zfp2xqKyHwR2aj/3yCe5RKRzqbPZKWIFIjIrYn6vETkNRHJE5FfTdsi/oxE5HT9s94kItOlimvQBSnXNBFZJyK5IvKRiNTXt2eLyAnTZ/dSnMsV8XcXp3J9YCrTNhFZqW+Py+cVIjbE9/dLKXXS/gPgBrAZQDsAKQBWAegWx/M3A3Ca/rgugA0AugF4EMCdFvt308uYCqCtXna3TWXbBiDTb9uTACbrjycDeCLe5fL77vYCaJOozwvAUACnAfi1Kp8RgCUABgIQAF8AGGtDuUYBSNIfP2EqV7Z5P7/3iUe5Iv7u4lEuv9efBnB/PD8vBI8Ncf39Otlr8P0AbFJKbVFKlQB4H8D58Tq5UmqPUmq5/rgQwFoALUIccj6A95VSxUqprQA2QfsZ4uV8AG/oj98AMCGB5RoBYLNSKtTIZVvLpZTKAXDQ4pxhf0Yi0gxAhlLqB6X9Nb5pOiZm5VJKzVNKlelPfwTQMtR7xKtcIST08zLotd0/Angv1HvEulwhYkNcf79O9gDfAsAO0/OdCB1gbSMi2QB6A/hJ3/QX/Xb6NdNtWDzLqwDME5GfReQ6fVsTpdQeQPsFBNA4AeUyXAzfP7pEf16GSD+jFvrjeJbxKmg1OUNbEVkhIt+KyBB9WzzLFcl3F+/PawiAfUqpjaZtcf28/GJDXH+/TvYAb9UWFfe8TxGpA2A2gFuVUgUAXgTQHkAvAHug3SIC8S3vIKXUaQDGArhJRIaG2Deun6OIpAAYD+A/+qbq8HlVJlhZ4v3Z3QugDMA7+qY9AForpXoDuB3AuyKSEcdyRfrdxfs7vQS+FYm4fl4WsSHorkHOX6VynewBfieAVqbnLQHsjmcBRCQZ2hf4jlLqQwBQSu1TSpUrpSoAvAxvs0LcyquU2q3/nwfgI70M+/RbPuOWNC/e5dKNBbBcKbVPL2PCPy+TSD+jnfBtLrGtjCIyEcC5AC7Tb9eh39If0B//DK3ttlO8yhXFdxfPzysJwIUAPjCVN26fl1VsQJx/v072AL8UQEcRaavXCi8G8L94nVxv33sVwFql1DOm7c1Mu10AwOjd/x+Ai0UkVUTaAugIrQMl1uWqLSJ1jcfQOuh+1c8/Ud9tIoBP4lkuE59aVaI/Lz8RfUb6bXahiAzQfx+uMB0TMyIyBsA9AMYrpY6btmeJiFt/3E4v15Y4liui7y5e5dKNBLBOKeVp4ojX5xUsNiDev1/R9hJXl38AxkHrod4M4N44n3swtNulXAAr9X/jALwF4Bd9+/8ANDMdc69e1vWoYvZAiHK1g9YjvwrAauNzAdAIwNcANur/N4xnufTzpAM4AKCeaVtCPi9oF5k9AEqh1ZSujuYzAtAHWmDbDGAG9BHiMS7XJmhttMbv2Uv6vr/Tv+NVAJYDOC/O5Yr4u4tHufTtrwO4wW/fuHxeCB4b4vr7xakKiIgc6mRvoiEioiAY4ImIHIoBnojIoRjgiYgcigGeiMihGODJMUTkqP5/tohcGuP3/qvf88WxfH8iOzDAkxNlA4gowBuDX0LwCfBKqTMiLBNR3DHAkxNNBTBEn+/7NhFxizaf+lJ9UqzrAUBEhulzdr8LbbAORORjfYK21cYkbSIyFUAt/f3e0bcZdwuiv/evos3ZfZHpvReKyH9Fm8f9HX0kIkRkqois0cvyVNw/HaoxkhJdACIbTIY2R/m5AKAH6iNKqb4ikgpgkYjM0/ftB6CH0qZoBYCrlFIHRaQWgKUiMlspNVlE/qKU6mVxrguhTbTVE0CmfkyO/lpvAN2hzR2yCMAgEVkDbUh/F6WUEn3hDiI7sAZPNcEoAFeItqrPT9CGi3fUX1tiCu4AMElEVkGbc72Vab9gBgN4T2kTbu0D8C2Avqb33qm0ibhWQms6KgBQBOAVEbkQwPHAtySKDQZ4qgkEwM1KqV76v7ZKKaMGf8yzk8gwaBNUDVRK9QSwAkBaGO8dTLHpcTm0FZnKoN01zIa2cMPcCH4OoogwwJMTFUJbJs3wJYAb9elbISKd9Fk2/dUDcEgpdVxEugAYYHqt1DjeTw6Ai/R2/ixoy8cFnfFSnx+8nlLqcwC3QmveIbIF2+DJiXIBlOlNLa8D+Ce05pHlekdnPqyXPZsL4AYRyYU2o9+PptdmAsgVkeVKqctM2z+Ctl7mKmizB96tlNqrXyCs1AXwiYikQav93xbVT0gUBs4mSUTkUGyiISJyKAZ4IiKHYoAnInIoBngiIodigCcicigGeCIih2KAJyJyqP8HKdebIvc9lrcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### Define optimizer and training operation ###\n",
    "\n",
    "'''TODO: instantiate a new model for training using the `build_model`\n",
    "  function and the hyperparameters created above.'''\n",
    "model = build_model(vocab_size, embedding_dim, rnn_units, batch_size)\n",
    "\n",
    "'''TODO: instantiate an optimizer with its learning rate.\n",
    "  Checkout the tensorflow website for a list of supported optimizers.\n",
    "  https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/\n",
    "  Try using the Adam optimizer to start.'''\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate)\n",
    "\n",
    "@tf.function\n",
    "def train_step(x, y): \n",
    "  # Use tf.GradientTape()\n",
    "  with tf.GradientTape() as tape:\n",
    "  \n",
    "    '''TODO: feed the current input into the model and generate predictions'''\n",
    "    y_hat = model(x)\n",
    "  \n",
    "    '''TODO: compute the loss!'''\n",
    "    loss = compute_loss(y, y_hat)\n",
    "\n",
    "  # Now, compute the gradients \n",
    "  '''TODO: complete the function call for gradient computation. \n",
    "      Remember that we want the gradient of the loss with respect all \n",
    "      of the model parameters. \n",
    "      HINT: use `model.trainable_variables` to get a list of all model\n",
    "      parameters.'''\n",
    "  grads = tape.gradient(loss, model.trainable_variables)\n",
    "  \n",
    "  # Apply the gradients to the optimizer so it can update the model accordingly\n",
    "  optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "  return loss\n",
    "\n",
    "##################\n",
    "# Begin training!#\n",
    "##################\n",
    "\n",
    "history = []\n",
    "plotter = mdl.util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss')\n",
    "if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists\n",
    "\n",
    "for iter in tqdm(range(num_training_iterations)):\n",
    "\n",
    "  # Grab a batch and propagate it through the network\n",
    "  x_batch, y_batch = get_batch(vectorized_songs, seq_length, batch_size)\n",
    "  loss = train_step(x_batch, y_batch)\n",
    "\n",
    "  # Update the progress bar\n",
    "  history.append(loss.numpy().mean())\n",
    "  plotter.plot(history)\n",
    "\n",
    "  # Update the model with the changed weights!\n",
    "  if iter % 100 == 0:     \n",
    "    model.save_weights(checkpoint_prefix)\n",
    "    \n",
    "# Save the trained model and the weights\n",
    "model.save_weights(checkpoint_prefix)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kKkD5M6eoSiN"
   },
   "source": [
    "## 2.6 Generate music using the RNN model\n",
    "\n",
    "Now, we can use our trained RNN model to generate some music! When generating music, we'll have to feed the model some sort of seed to get it started (because it can't predict anything without something to start with!).\n",
    "\n",
    "Once we have a generated seed, we can then iteratively predict each successive character (remember, we are using the ABC representation for our music) using our trained RNN. More specifically, recall that our RNN outputs a `softmax` over possible successive characters. For inference, we iteratively sample from these distributions, and then use our samples to encode a generated song in the ABC format.\n",
    "\n",
    "Then, all we have to do is write it to a file and listen!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JIPcXllKjkdr"
   },
   "source": [
    "### Restore the latest checkpoint\n",
    "\n",
    "To keep this inference step simple, we will use a batch size of 1. Because of how the RNN state is passed from timestep to timestep, the model will only be able to accept a fixed batch size once it is built. \n",
    "\n",
    "To run the model with a different `batch_size`, we'll need to rebuild the model and restore the weights from the latest checkpoint, i.e., the weights after the last checkpoint during training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "LycQ-ot_jjyu"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (1, None, 256)            21248     \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (1, None, 1024)           5246976   \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (1, None, 83)             85075     \n",
      "=================================================================\n",
      "Total params: 5,353,299\n",
      "Trainable params: 5,353,299\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "'''TODO: Rebuild the model using a batch_size=1'''\n",
    "model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
    "\n",
    "# Restore the model weights for the last checkpoint after training\n",
    "model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n",
    "model.build(tf.TensorShape([1, None]))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I9b4V2C8N62l"
   },
   "source": [
    "Notice that we have fed in a fixed `batch_size` of 1 for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DjGz1tDkzf-u"
   },
   "source": [
    "### The prediction procedure\n",
    "\n",
    "Now, we're ready to write the code to generate text in the ABC music format:\n",
    "\n",
    "* Initialize a \"seed\" start string and the RNN state, and set the number of characters we want to generate.\n",
    "\n",
    "* Use the start string and the RNN state to obtain the probability distribution over the next predicted character.\n",
    "\n",
    "* Sample from multinomial distribution to calculate the index of the predicted character. This predicted character is then used as the next input to the model.\n",
    "\n",
    "* At each time step, the updated RNN state is fed back into the model, so that it now has more context in making the next prediction. After predicting the next character, the updated RNN states are again fed back into the model, which is how it learns sequence dependencies in the data, as it gets more information from the previous predictions.\n",
    "\n",
    "![LSTM inference](https://raw.githubusercontent.com/aamini/introtodeeplearning/2019/lab1/img/lstm_inference.png)\n",
    "\n",
    "Complete and experiment with this code block (as well as some of the aspects of network definition and training!), and see how the model performs. How do songs generated after training with a small number of epochs compare to those generated after a longer duration of training?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "WvuwZBX5Ogfd"
   },
   "outputs": [],
   "source": [
    "### Prediction of a generated song ###\n",
    "\n",
    "def generate_text(model, start_string, generation_length=1000):\n",
    "  # Evaluation step (generating ABC text using the learned RNN model)\n",
    "\n",
    "  '''TODO: convert the start string to numbers (vectorize)'''\n",
    "  input_eval = [char2idx[num] for num in start_string]\n",
    "  input_eval = tf.expand_dims(input_eval, 0)\n",
    "\n",
    "  # Empty string to store our results\n",
    "  text_generated = []\n",
    "\n",
    "  # Here batch size == 1\n",
    "  model.reset_states()\n",
    "  tqdm._instances.clear()\n",
    "\n",
    "  for i in tqdm(range(generation_length)):\n",
    "      '''TODO: evaluate the inputs and generate the next character predictions'''\n",
    "      predictions = model(input_eval)\n",
    "      \n",
    "      # Remove the batch dimension\n",
    "      predictions = tf.squeeze(predictions, 0)\n",
    "      \n",
    "      '''TODO: use a multinomial distribution to sample'''\n",
    "      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
    "      \n",
    "      # Pass the prediction along with the previous hidden state\n",
    "      #   as the next inputs to the model\n",
    "      input_eval = tf.expand_dims([predicted_id], 0)\n",
    "      \n",
    "      '''TODO: add the predicted character to the generated text!'''\n",
    "      # Hint: consider what format the prediction is in vs. the output\n",
    "      text_generated.append(idx2char[predicted_id])\n",
    "    \n",
    "  return (start_string + ''.join(text_generated))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ktovv0RFhrkn"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 25%|██▍       | 2461/10000 [00:21<01:11, 105.44it/s]"
     ]
    }
   ],
   "source": [
    "'''TODO: Use the model and the function defined above to generate ABC format text of length 1000!\n",
    "    As you may notice, ABC files start with \"X\" - this may be a good start string.'''\n",
    "generated_text = generate_text(model, start_string=\"X\", generation_length=30000) # TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AM2Uma_-yVIq"
   },
   "source": [
    "### Play back the generated music!\n",
    "\n",
    "We can now call a function to convert the ABC format text to an audio file, and then play that back to check out our generated music! Try training longer if the resulting song is not long enough, or re-generating the song!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "LrOtG64bfLto"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 43 songs in text\n",
      "X:184\n",
      "T:Connishelland\n",
      "Z: id:dc-reel-14\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "BcB B2:|!\n",
      "d|edB def|gfg agf|g2b agf gf gf|e3 c B2|!\n",
      "A2 ^cA A2:|!\n",
      "cd|fa a3 g3|bag fed|Bcd ecA|afd ecA A2:|!\n",
      "a|bag faa|afd fdd|Adf adB|GBB BAG|d2e f2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:19\n",
      "T:Beahe Willedel-8\n",
      "M: Aufg abaf|gfed edga|bggb agbg|!\n",
      "eaa^g a2|]!\n",
      "\n",
      "\n",
      "\n",
      "X:85\n",
      "T:Hig Tindlandell\n",
      "Z: id:dc-slipjig-54\n",
      "M:9/8\n",
      "L:1/8\n",
      "K:D Major\n",
      "zE|D2B, A,B,D|A,DD DFG|ABc d^c|d3 f3 AGF|G3 G2:|!\n",
      "F|AFE Bed|cAG AGE|D3|]!\n",
      "cd|Aff eg2|dcA dcA|]!\n",
      "\n",
      "\n",
      "\n",
      "X:18\n",
      "T:Brian O'Lynn\n",
      "Z: id:dc-slipjig-6\n",
      "M:9/8\n",
      "L:1/8\n",
      "K:G Major\n",
      "D|G2G cGB|cAB cde|cAA Bcd|Adf d2\n",
      "L:1/8\n",
      "K:G Major\n",
      "d>c|BA G>A|Bd B|!\n",
      "A2 Bc|de a>f|g2 f/g/ ag|e f2|A2 B2 de|d2 c2 A2|de dBGB cAFA|!\n",
      "GAFA G2EF|EFGE D2A|BEE E2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:11\n",
      "T:Fishef dy Buscha Casin\n",
      "Z: id:dc-reel-140\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "B|ADF AFA|def a2f|gfe d2c|B3 AEA|!\n",
      "F2A AFA|BEE id:dc-ocarolan-31\n",
      "M:3/4\n",
      "L:1/8\n",
      "K:G Major\n",
      "\n",
      "\n",
      "\n",
      "L:1/8\n",
      "K:A Major\n",
      "D||GAB ecA|BAG Bcd|edB gfg|d2B BAG Bde|gdB BAG|EGE DB,c|\n",
      "M:C\n",
      "L:1/8\n",
      "K:A Major\n",
      "A|F3A BdcA|d2fd fdak Croin\n",
      "Z: id:dc-slipjig-93\n",
      "M: ig:dc-reel-82\n",
      "M:C\n",
      "L:1/8\n",
      "K:A Dorian\n",
      "D>E|DE G>A|B2 G2 DB,|DG2 G2|D>D FA c2|!\n",
      "G>E F/G/E/D/|Ad f/f/ gf|ge d>g|!\n",
      "g>a ba a^ga|fef afd|ede fd|FA dA|^cA A>A|!\n",
      "AF EC|D2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:24\n",
      "T:Road to Reee's cay\n",
      "Z: id:dc-jig-95\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:D Majig-9\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:A Mixolydian\n",
      "ef/e/ L:6/8\n",
      "L:1/8\n",
      "K:D Major\n",
      "ed|cAA A2B|cAd cAG|AGA FDE|DED GFD|!\n",
      "D3 Bcd|faf edA|f3 fec|B2G B2:|!\n",
      "A|BAG ded|ded d3|GAB c2d|e2d e3|!\n",
      "g2G BAG|dBG ADE|FGF EDB,|DFA GEC|Dd,D FDD:|!\n",
      "E3 B3|AGF D2|A3 A de|f2 f2|A2 d2 f2|A2 f2 f2|A2 f2 A2=g|!\n",
      "d2 fd Bc|BA ec|Bc A2|!\n",
      "G3 A G3|A a^g g2|dg ed|d2 de Bd|A2 f2 f2|]!\n",
      "\n",
      "\n",
      "\n",
      "X:28\n",
      "T:Connaig Pischa\n",
      "Z: id:dc-reel-382\n",
      "M:C\n",
      "L:1/8\n",
      "K:A Major\n",
      "dB|c2A cAF GFE|DFA ecA|Ace fed|cAe f3:|!\n",
      "aga agf g3|g3 gab|age pomelle Honder Pubewle\n",
      "Z: id:dc-jig-19\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:D Major\n",
      "f|g3 f3|fed cAF GFE|DB, DB,|D2c =c E2:|!\n",
      "GB AD|FA GA BA|fA fe|dA (3ABc B2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:10\n",
      "T:Denor's Days\n",
      "Z: id:dc-reel-148\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "c|d2ef a2ea|edge abag|afge dBAc|BEE2 G2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:148\n",
      "T:Plaf Dashlebar\n",
      "Z: id:dc-jig-4\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:A Major\n",
      "eae ced|cAA cAeA|defg afdf afgf|dgfe dcAF|EGce c3:|!\n",
      "gage d^cde|f2ef a2ba|gedB G2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:13\n",
      "T:Conlelland Orovin\n",
      "Z: id:dc-slipjig-22\n",
      "M:9/8\n",
      "L:1/8\n",
      "K:G Major\n",
      "GA|B2G BAG|EGG AGE|EDC D2:|!\n",
      "de|f3 f3|gfe gfe|dBA agf gag|fed cAG|E3 Bef|!\n",
      "g>a ge|fe d^c|d2 d3 A|!\n",
      "e>f ge|de ge|dB A3:|!\n",
      "f2 f2|fd df|fg e2|d4 g2|c2 d2 d2|Bc ec|!\n",
      "d3 f g^fd|efe dcA|AGE DEB|!\n",
      "G3 A G2|c2 B2 c2 A2|G2 G2 D2|GF A2 B2|A4 C2|D2 C2 D2|]!\n",
      "B,|A,2B, D2|BAFG ABcA|B2AB G3:|!\n",
      "K:G Major\n",
      "dAF Beef|gedB BAAG|F2FE DFAF|ABdA FAdA|!\n",
      "Beed ADFA|defe defe|d2fd ddBd|eBB2 bgaf|!\n",
      "gfeg befg|a3 g3:|!\n",
      "g2ee g3f|efge fdBd|ABce agag|fdd2 fed/c/B/A/|B2 A2 D2|FE GB AF|E2 E2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:8\n",
      "T:Brian O'Lynn\n",
      "Z: id:dc-hornpipe-76\n",
      "M:C|\n",
      "L:1/8\n",
      "K:A Dorian\n",
      "ed|^cABG AGED G2AG|A2BG ADB,A,|D2B,A, D2Bd|cedB A2GA|!\n",
      "B3A B2BA|B3 cBA|EAc ABc|!\n",
      "d2e fdB|AFA dBG|ABA dAF|GEC|D2B,D D2B|!\n",
      "Add eaa|gfg ag^g|bgg agf g2e fga|ced cAA:|!\n",
      "BcB BAF|ded ded dBd|fed ^cAA|!\n",
      "A2 d>e|fed adc|cAG FD|!\n",
      "D2 F/G/A/B/c/|dG e/B/A/G/ F/E/|D>D FGA|Bd cA/B/|cA A/c/B/A/|G2 G2 E2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:14\n",
      "T:Paddman's Ramblerury\n",
      "Z: id:dc-jig-7\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:G Major\n",
      "G>A GA BG|GB AG/A/|de f/g/f/e/ de|!\n",
      "g2 fd ed|ce Bc|!\n",
      "ecA Aef|gf gagec d^cAG|FAdB A2FA|!\n",
      "dABd edBd|e2d2 BAFA|!\n",
      "FGAB cB=f|edef gfed|BAFA defg|fedB ADFA|]!\n",
      "\n",
      "\n",
      "\n",
      "X:336\n",
      "T:Ord's Dannafgaf|edBA BAFA|!\n",
      "D2FA DFAF|GFGA BGE2|AGEG AFGE|EDB,A, DEFG|!\n",
      "EFEE cBBA|B2EB ABcA|defg afge|dcBA G2e2|!\n",
      "cBAe f3e|faf edcA|dfaf gfec:|!\n",
      "HighlE:C\n",
      "L:1/8\n",
      "K:A Major\n",
      "g|e2ef g2bg baga|bgge e2fe|d2cd A2AB|!\n",
      "aeae aged|cAA2 df2e|f2fe dcA|dcA AGE|D3 A2F|!\n",
      "D2B BAG D2:|!\n",
      "e|f3 a3|baf gee|AGE DB,D|ECC Dorian\n",
      "A3f|g2dB cefg|aeae aged|cAAc BEEe|!\n",
      "f2f afd|afe efg|fad fAf|gfe def gfe|d3 fd:|!\n",
      "8\n",
      "L:1/8\n",
      "K:B Dorian\n",
      "ed|efga bgag|fecA dAA2|agfe dcBA|!\n",
      "defg afdf|bgaf gefe|efge fedB|ABAB c2:|!\n",
      " X:14\n",
      "T:Firding of Tan\n",
      "Z: id:dc-reel-362\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "A|d2Bd gdBd|gdBc dB|(3Bed edB|!\n",
      "EAA BAG|A2B cBA|!\n",
      "c2A A3|BAG FDE DFG|F2G FAG|EA=F G2|]!\n",
      "\n",
      "\n",
      "\n",
      "X:24\n",
      "T:Nick Raf in the Hour\n",
      "Z: id:dc-polka-30\n",
      "M:C|\n",
      "L:1/8\n",
      "K:D Major\n",
      "zABd fedB|ADFD A,DFA|E2FD ABAe|f2df efge:|!\n",
      "agfd efg|dce a3|gfe gfg|f2d gfg|!\n",
      "BcB B2c def|g3 gag|fd cA|FA G3:|!\n",
      "dfedAG2 DGBd|e2AB cAEG (3faf gfed|cA (3BcA (3FEF|DEGA BAFA|BcdB A2:|!\n",
      "AB|cBe^c O=oan\n",
      "Z: id:dc-polka-30\n",
      "M:2/4\n",
      "L:1/8\n",
      "K:A Major\n",
      "BAdB AFA|BEE cBc|ded fdB|EBd cAF|!\n",
      "FE|EDE cAF|GAB c3|ABc d3|]!\n",
      "\n",
      "\n",
      "\n",
      "X:74\n",
      "T:Orandy Ra\n",
      "X:237\n",
      "T:Hug erise\n",
      "Z: id:dc-hornpipe-48\n",
      "M:C|\n",
      "L:1/8\n",
      "K:G Major\n",
      "DGBG|c2B2 A2ce|dedB ABGA|efgd BAA2|!\n",
      "B,CE Bced|eddB AFDF|bBdB ADFA|BFAF E2:|!\n",
      "fe|f2df ec|B2B>A Bc|de d>e/f/g/|fe f>g|af gefd|!\n",
      "dfaf gbeg|faed cAA|AGF EDE|!\n",
      "DED DEA|BcB BBA|Bcd A3:|!\n",
      "g2e fdB|AGE DB,D|E2cA BdAG|AGBd gdBd|gdBd gdgd|!\n",
      "B2GB AGED|E2BA G2FG|ABAG FDEF|!\n",
      "Addd f3e|faa^g a2=fa|baf gedB|!\n",
      "ceef afeg|f2df b2gb|afce dcBA|!\n",
      "c2Bc AGFA|BABc A3B|BAGF G2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:18\n",
      "T:Bish Hert\n",
      "Z: id:dc-slipjig-7\n",
      "M:3/4\n",
      "L:1/8\n",
      "K:G Major\n",
      "g>f|ed dB Ac|d2 d:|!\n",
      "z|g2f edB|def ged|BAF E2:|!\n",
      "e|a^gab age|fdd gag|fed cAA2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:78\n",
      "T:DA, DFA,A,|B,A,B,E DB,B,C D3B, A,F,D|!\n",
      "E2eA fded|cABc d2cB|d2cA BAFc|ABcA d2|]!\n",
      "\n",
      "\n",
      "\n",
      "X:12\n",
      "T:Nhen\n",
      "Z: id:dc-reel-258\n",
      "M:C\n",
      "L:1/8\n",
      "K:A Major\n",
      "AG|AddB AGFG|EB,B, D,B,A,|FD D2:|!\n",
      "z|afd afe|dce gfg|aba age|dBB BAF|!\n",
      "GBd gA|BdB BAG Bcd|ged dBA|f2e dAG|!\n",
      "BeB B2A|GEB cBc|dcA AGA|!\n",
      "EGB BEB|cBc A3|AGE DB,A,|DA,A, D3:|!\n",
      "\n",
      "\n",
      "\n",
      "X:3\n",
      "T:Carolamours\n",
      "Z: id:dc-waliz-3\n",
      "M:C|\n",
      "L:1/8\n",
      "K:D Major\n",
      "dA|A3B c2BG|AcBA eBe2|e2 (3ABA (3AB^c|dc-reel-143\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "D3 d3|G3 dGB|EAG FED|dDD BGB|!\n",
      "A3 dAG|EDE G2E DB,C|EGE DEG|DFA Bcd|!\n",
      "g3 bag|a3 g3|gf edB|d2B Bcd|!\n",
      "dcA d2B|cBc cBc|dge f2|!\n",
      "a^g afd|faf bag|afg fga b3|dgb gef|gfe d^cd|cee ecA Bcd|!\n",
      "dgg aba|bgg a2f g3|f3 gab|faf gfe:A|!\n",
      "cde^c dfeg|afdf g2dg|bgag bgag|eaa2 bagf|a2bg agef|!\n",
      "afdf gfed|c2ec AB|^cABc defg|fece of Da^f d=f|!\n",
      "gfgb aged|cdef gbag|e2d2 c2ef|gece dedg|f2 b2 a3f|!\n",
      "b3f afde|fgaf gfed|c2BA B2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:88\n",
      "T:Eilly Drind's\n",
      "Z: id:dc-reel-112\n",
      "M:C\n",
      "L:1/8\n",
      "K:G Major\n",
      "ge|f2gf bfaf|eBcA defg|afdf afge|dBcA dfg2:|]!\n",
      "\n",
      "\n",
      "\n",
      "X:3\n",
      "T:Garnan Hayfefed:|!\n",
      "e|faaf gfec|dcAG FEGB|cBcd eAec|dcBG AGED|!\n",
      "B,DDB, G,A,B,|CDFD FEDC|B,DA, D4|]!\n",
      "\n",
      "\n",
      "\n",
      "X:110\n",
      "T:Mus of No. 2\n",
      "Z: id:dc-reel-256\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "A3B AFF2|FEDF DB,A,2|BAG2 ABcA|d2dc d2|]!\n",
      "\n",
      "\n",
      "\n",
      "X:370\n",
      "T:Wild ar Mur\n",
      "Z: id:dc-reel-50\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "B|A3B cAFG|A2Bd ABcA|def/g/f/ec d2:|!\n",
      "cd|ecAc BAA2|BcBA cBcd|eaa^g a3f|ed^ce a3=g Ra-onan\n",
      "M:C\n",
      "L:1/8\n",
      "K:A Major\n",
      "d>e/f/g/a/e/ d/c/B/|c/A/B/ AF DC|F3 EDC|D2F D2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:14\n",
      "T:Nolling3A f3a|bgba gfe|ded d2B|A2F dEF|!\n",
      "ECD D2E DEF|GAA E2G|cAG FED|FAA fed|cA GE|DA D>E|!\n",
      "E>F EC|ED DC|!\n",
      "D2|AD F/G/|Ad d2|A2 A2 G2|GF D>a|ge g>e|!\n",
      "def age|aea edB|cBA Bcd|!\n",
      "eaa age|fed cde|fdB BAF Bc|def afd|!\n",
      "eef fec|eAA fBd|agf gfe|ded d2:|!\n",
      "e|f2ag efge|dBGB ADFA|GABG AGEF|G3A BGG2|B2B2 A3 B3|AG2 cBc|!\n",
      "d2e fed|e3 B3 dAB|cBc ABc|d3 d2:|!\n",
      "f|afd ecA|fed cAF|G\n",
      "L:1/8\n",
      "K:A Dorian\n",
      "A|DA,A, E3|AFD E2g|fed cAG|F2D D2C in Liskellan's\n",
      "Z: id:dc-ocarolan-5\n",
      "M:C 4 D2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:58\n",
      "T:Crowley's of Bance\n",
      "Z: id:dc-reel-152\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "Bc|d2 dec edc d^cA|dAd d2e|fed dcB|AGE DB,D|!\n",
      "EDE ADD|E2d cAG EFG|A2d fdB|A2c BAG|!\n",
      "FAd fdd|edB cBA|def gdB|!\n",
      "AF EF|GE E>f|ed Bc|d2 BA A2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:188\n",
      "T:Prip O'Lynn\n",
      "Z: id:dc-polka-2\n",
      "M:9/8\n",
      "L:1/8\n",
      "K:G Major\n",
      "G2|DG BAG|BAG AGF|E3 E2:|!\n",
      "A|B3 B<g|agf edB|ded edB|!\n",
      "d>e fd|ed dA/B/|c/B/A/:|!\n",
      "\n",
      "\n",
      "\n",
      "X:148\n",
      "T:Brian\n",
      "B|A2BA BAFA|DFFA d2de|f2dB ADFA|BABc ABcd|fdcA AFDA|!\n",
      "d2AB c=cAB|cABG ABAF|E3F EFAd|eaa2 bgf|!\n",
      "dcAG EDD:|!\n",
      "ed|a2af bagb|agef gfef|gfed cAA2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:267\n",
      "T:Woml-49\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "fg|edBd ADFA|defg af^gf|defd edfd|edBc AB^cd|!\n",
      "EAcA BcAB|cABG A2BA|Bed2 edBd|!\n",
      "eAling\n",
      "Z: id:dc-reel-244\n",
      "M:C\n",
      "L:1/8\n",
      "K:D Major\n",
      "dA|G3 A3|B3 B AG|EB cd|ed de/f/|!\n",
      "ed A>c|Bd de|fd e^c|de fd AF DF|!\n",
      "D>A de|fd de/|de fdB|dBG cGE|!\n",
      "\n",
      "\n",
      "\n",
      "X:18\n",
      "T:Conn O'Raffer\n",
      "Z: id:dc-slipjig-2\n",
      "M:2/4\n",
      "L:1/8\n",
      "K:G Major\n",
      "Ad|cdec dege|!\n",
      "defg a^gab|e2f edB|ded def|gfe def|!\n",
      "g2b agf|gfe fed|cAF Gge|dcB AGA|B2G BAG|!\n",
      "ABc d^cA|AGE DB,|DG BG|G2 C2|A2 GA|]!\n",
      "\n",
      "\n",
      "\n",
      "X:84\n",
      "T:Bunf of Ballinis\n",
      "Z: id:dc-slipjig-25\n",
      "M:2/4\n",
      "L:1/8\n",
      "K:E Dorian\n",
      "E>D|AD EA|!\n",
      "FA AB G>A|BA Be|fe ef|!\n",
      "fd e/f/g/f/g/|ae f/A/B/B/|!\n",
      "d|g2 f2 fg|a2 ba a2|f2 f2|f2 f2|af ec|d2 d>e|!\n",
      "e>f ec|d/e/f/ gd|^cA A>e|g/f/e/d/ d2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:31\n",
      "T:Bhee Se Islarelgagfe bgec|!\n",
      "Gefg a2fe|dAA2 defe|dBAF E2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:170\n",
      "T:Pron\n",
      "Z: id:dc-jig-14\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:G Major\n",
      "GE|E3 ABd|e3 ABc|d2E E2:|!\n",
      "fg|agb agf g3|dBG B2c|dcd ecA|BdB BAF:|!\n",
      "\n",
      "\n",
      "\n",
      "X:14\n",
      "T:F/Major\n",
      "\n",
      "\n",
      "\n",
      "X:23\n",
      "T:Paddy O'Ryaney\n",
      "Z: id:dc-reel-242\n",
      "M:C\n",
      "L:1/8\n",
      "K:G Major\n",
      "D2|GGG2 FDED|GFGB AGFD|GABc dfga|bged cAA2|BAFA c3e|fafd cAK:|!\n",
      "\n",
      "\n",
      "\n",
      "X:18\n",
      "T:Conning Big\n",
      "Z: id:dc-reel-241\n",
      "M:C\n",
      "L:1/8\n",
      "K:G Major\n",
      "Ad|gedg edBA|dgbg agge|defe defa|bgef g2dg|!\n",
      "agfe defe|dcBA gfge|fdcA AGFG|]!\n",
      "\n",
      "\n",
      "\n",
      "X:14\n",
      "T:Fareen of Larked cAA2 GBGB|!\n",
      "A2cA BABc|dBB2 cBAF|GEBE AGEF GABc|!\n",
      "dBBA BAFA|dAfA d2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:111\n",
      "T:Od B ADFA|BABc d2:|!\n",
      "de|fefa gfed|cdef eBdg|eagf ecAc|!\n",
      "d2eg f2fe|fedf bgbg|afge fafg|!\n",
      "a^gaf gfed|Bdgf efge|fdcA BcAG|!\n",
      "A2fA fAaA|Bee2:|!\n",
      "gfed cAGc|Bdgd dag2|edBd gfg2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:8\n",
      "T:Boys ic Bay\n",
      "Z: id:dc-jig-164\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:D Major\n",
      "AG|FDD|Add BAG c2A|dcA Bce|dcB A2d|!\n",
      "fgf ged dBG|ABe gag|f2d A2c|B2E E2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:24\n",
      "T:Proving RundGE|!\n",
      "D3A BcdB|A2FA BAFD|E2eA eAeg|eAA2 edBd|ADFG A2FA|f2df efge:|!\n",
      "agef g2fg|afge fde2|fdad defa|!\n",
      "befa gbag|fedB A3 AFD|D3 EFG|Ad^c d^c|Bc Ag/f/g/|!\n",
      "fd ^cA A2|BA Bc|BB AB|dc A:|!\n",
      "\n",
      "\n",
      "\n",
      "X:29\n",
      "T:Pip to Rian\n",
      "Z: id:dc-setdancett\n",
      "Z: id:dc-slipjig-2\n",
      "M:C\n",
      "L:1/8\n",
      "K:G Major\n",
      "GE|D2B, D2 B2|A2 dc d|ecA AB=c|dec dcB|A3 B2A|!\n",
      "B2B efg|a3 g2:|!\n",
      "e|age a3|bag age|dBd g3|bag a2f fai|fe f/e|fdfg edAF:|!\n",
      "agfe d2Bd|c2A2 AcGA|B2GB BAA2|Defe dBB2|!\n",
      "aff2 bfge|^fde^c dedB|defd efgB|edef defg|afge dB:|!\n",
      "ed|e2ef ecAe|faaf bfaf|gefd gfga|bgag fdgf|B2^ce gfed|!\n",
      "c2B2 cdeg|f2df afdf|gefg afed|fdec d2:|!\n",
      "K:A Dorian\n",
      "B|cAA2 BGEE|D2d2 AGA|BABc dcBc|A2GA BAGA|!\n",
      "BABc B2AG|EdcA BEE2|c2BB AGFE|DGBG DGBF|EBBA BAGA|!\n",
      "B2BA BAGB|ABce fedB|ADFA dAFA|GBB2 |GABd c2Ac|B2E2 GABc|!\n",
      "dBdB gBAB|Add2 B2AF DB,B,D|B2BE ABAG|E2B2 B2dB|!\n",
      "cAdA eAfg|afge defg a2fg|ae^e2 f2:|!\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:11\n",
      "T:Planxty B2:|!\n",
      "\n",
      "\n",
      "\n",
      "X:15\n",
      "T:Fabf\n",
      "Z: id:dc-setdance-15\n",
      "M:6/8\n",
      "L:1/8\n",
      "K:D Major\n",
      "G>A Bd|ed dB|BA BA|G>A BA|!\n",
      "A2 B2 GE|D/E/F/G/ AF|E2 E2 c2|cA B/c/B/|AG A|]!\n",
      "e>e|fd df|ed Bd|cB Ae|!\n",
      "f2 ed|!\n",
      "cA Ac|BA cA|EA cA FA|!\n",
      "GB A/E/E/D/|D2 D2:|!\n",
      "eA A/B/c/|ae a/b/ ae|=cdef g2fg|!\n",
      "(3fga cege faaf|ecAc fAfA|f3e dcBA|!\n",
      "B2BA FAA2|fdec A3B|!\n",
      "d3f edef|gfga ge (3Bcd (3Afaf gfef|!\n",
      "g3e fag|fd ede|fed fed|cAB A:|!\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### Play back generated songs ###\n",
    "\n",
    "generated_songs = mdl.lab1.extract_song_snippet(generated_text)\n",
    "for i, song in enumerate(generated_songs): \n",
    "  # Synthesize the waveform from a song\n",
    "  mdl.lab1.play_song(song)\n",
    "  print(song, end=\"\\n\\n\\n\\n\")\n",
    "  #print(song)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HgVvcrYmSKGG"
   },
   "source": [
    "## 2.7 Experiment and **get awarded for the best songs**!\n",
    "\n",
    "Congrats on making your first sequence model in TensorFlow! It's a pretty big accomplishment, and hopefully you have some sweet tunes to show for it.\n",
    "\n",
    "Consider how you may improve your model and what seems to be most important in terms of performance. Here are some ideas to get you started:\n",
    "\n",
    "*  How does the number of training epochs affect the performance?\n",
    "*  What if you alter or augment the dataset? \n",
    "*  Does the choice of start string significantly affect the result? \n",
    "\n",
    "Try to optimize your model and submit your best song! **MIT students and affiliates will be eligible for prizes during the IAP offering**. To enter the competition, MIT students and affiliates should upload the following to the course Canvas:\n",
    "\n",
    "* a recording of your song;\n",
    "* iPython notebook with the code you used to generate the song;\n",
    "* a description and/or diagram of the architecture and hyperparameters you used -- if there are any additional or interesting modifications you made to the template code, please include these in your description.\n",
    "\n",
    "You can also tweet us at [@MITDeepLearning](https://twitter.com/MITDeepLearning) a copy of the song! See this example song generated by a previous 6.S191 student (credit Ana Heart): <a href=\"https://twitter.com/AnaWhatever16/status/1263092914680410112?s=20\">song from May 20, 2020.</a>\n",
    "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n",
    "\n",
    "Have fun and happy listening!\n",
    "\n",
    "![Let's Dance!](http://33.media.tumblr.com/3d223954ad0a77f4e98a7b87136aa395/tumblr_nlct5lFVbF1qhu7oio1_500.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}