20_scheduled_batch_inference_lab_workset.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "ESEkRyOvXNp7"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Install cron, twint, & update fastai\n",
    "!apt install  -qq cron\n",
    "!pip install -Uqq fastai\n",
    "!git clone --depth=1 https://github.com/twintproject/twint.git\n",
    "!cd /content/twint && pip install -qq . -r requirements.txt\n",
    "!pip install aiohttp==3.7.0\n",
    "!pip instll nest_asyncio\n",
    "!rm -r twint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "oKI1rZrJz25s"
   },
   "outputs": [],
   "source": [
    "# Write the directory we will store the CSVs\n",
    "!mkdir csvs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5T5FOBjavzyx",
    "outputId": "f27fa912-0db1-43fe-e4b4-c73f7e1fa410"
   },
   "outputs": [],
   "source": [
    "%%writefile scrape_tweets.py\n",
    "#!/usr/bin/env python\n",
    "import twint\n",
    "import nest_asyncio\n",
    "from datetime import datetime\n",
    "import pandas as pd\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "c = twint.Config()\n",
    "c.Limit = 10\n",
    "c.Lang = \"en\"\n",
    "c.Store_csv = True\n",
    "c.Search = \"apple\"\n",
    "c.Output = f\"/content/csvs/en_apple-{datetime.now()}.csv\"\n",
    "twint.run.Search(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sndEHC-gY-v-",
    "outputId": "0861171d-f76a-45a9-d1d3-708fd39873fd"
   },
   "outputs": [],
   "source": [
    "# Start crontab\n",
    "!service cron start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cr1cryiGy1CO",
    "outputId": "c14fbc75-4ead-4647-e645-77030f45d2db"
   },
   "outputs": [],
   "source": [
    "%%writefile crontab.e\n",
    "* * * * * python3 /content/scrape_tweets.py >> /content/cron.log 2>&1\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "ETxFwwgezIhX"
   },
   "outputs": [],
   "source": [
    "!crontab crontab.e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0zS7s2Kfz-av",
    "outputId": "c58da782-8a5d-42db-d0b5-18f6ab6053d8"
   },
   "outputs": [],
   "source": [
    "!crontab -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kCOaQhJ2sLXS",
    "outputId": "fbcc15ad-7fca-4ecd-ca3d-02a9375c10b7"
   },
   "outputs": [],
   "source": [
    "!sleep 120\n",
    "!ls csvs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ihj49vrnSciD",
    "outputId": "a5df8c4d-403d-48d3-bbf9-0bf5091077ef"
   },
   "outputs": [],
   "source": [
    "!tail -n 10 /content/cron.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ndvFecDOdHmQ"
   },
   "source": [
    "# Exercise 20.1\n",
    "\n",
    "### Train and save your classifier here\n",
    "\n",
    "Use this section to train a RNN to classify text by sentiment using a dataset of your choice.\n",
    "We don't particularly care about the predictive performance since accuracy isn't the goal of this lesson, but it should perform better than random chance.\n",
    "\n",
    "<!-- startquestion -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F-UvB1_1bRez"
   },
   "outputs": [],
   "source": [
    "# Create datasets/dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H9ZvgGxkPdSe"
   },
   "outputs": [],
   "source": [
    "# Train or fine-tune a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qLHHs2uJPe9t"
   },
   "outputs": [],
   "source": [
    "# Save the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4YLdwotspIFM"
   },
   "source": [
    "# Exercise 20.2\n",
    "\n",
    "### Write your inference Python file in the cell below\n",
    "\n",
    "The cell below writes out a python file that should be executed every minute after tweets are collected.\n",
    "This script should:\n",
    "\n",
    "* Load the traine model from Exercise 20.1\n",
    "* Load the most recent CSV of tweets into a DataFrame\n",
    "* Make inferences on the tweets in the data from the previous step, storing results in a `\"sentiment\"` column\n",
    "* Count the negative and positive tweets for the log\n",
    "* Write the DataFrame with the additional column back to the original CSV\n",
    "\n",
    "Once this .py file is complete, we will update our crontab so that this script runs every minute right after tweets are scraped.\n",
    "\n",
    "<!-- startquestion -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7pOnDMNdpMxN",
    "outputId": "59f05f46-d85c-4ebc-a2c7-6cc87e643a74"
   },
   "outputs": [],
   "source": [
    "%%writefile inference.py\n",
    "# Imports\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "\n",
    "def load_model():\n",
    "    \"\"\"\n",
    "    Loads and returns a trained model.\n",
    "    \"\"\"\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def load_most_recent_csv():\n",
    "    \"\"\"\n",
    "    Loads and returns the most recent CSV of tweet data for inference, \n",
    "    and the path to that CSV.\n",
    "    \"\"\"\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def add_sentiment_inference_column(data, model):\n",
    "    \"\"\"\n",
    "    Uses a model inferences to add a 'sentiment' column to the data\n",
    "    \"\"\"\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def count_neg_pos_tweets(data):\n",
    "    \"\"\"\n",
    "    Returns the number of negative and number of positive tweets\n",
    "    \"\"\"\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def write_results(data, path):\n",
    "    \"\"\"\n",
    "    Overwrites the CSV file from the dataframe \n",
    "    that includes the inference results.\n",
    "    \"\"\"\n",
    "\n",
    "def main():\n",
    "    # Load the trained model\n",
    "    print(f'{datetime.now()}: Loading model...')\n",
    "    model = load_model()\n",
    "\n",
    "    # Load the most recent CSV\n",
    "    print(f'{datetime.now()}: Loading CSV...')\n",
    "    data, path = load_most_recent_csv()\n",
    "\n",
    "    # Write predictions to a new column called \"sentiment\"\n",
    "    print(f'{datetime.now()}: Making inferences...')\n",
    "    data_with_inferences = add_sentiment_inference_column(data, model)\n",
    "\n",
    "    # Print a statement with the number of negative tweets found for our logs\n",
    "    n_negative_tweets, n_positive_tweets = count_neg_pos_tweets(data_with_inferences)\n",
    "    print(f'{datetime.now()}: We recorded {n_negative_tweets} negative tweets out of {n_total_tweets} tweets ')\n",
    "\n",
    "    # Write out the CSV to back to the original file\n",
    "    write_results(data, path)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rhawZETLqt7k",
    "outputId": "43d52a92-4ab9-41c4-abb5-104d454eac90"
   },
   "outputs": [],
   "source": [
    "%%writefile crontab.e\n",
    "* * * * * python3 /content/scrape_tweets.py >> /content/cron.log 2>&1\n",
    "* * * * * python3 /content/inference.py >> /content/inference.log 2>&1\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Vwq-tP30Qc3Q"
   },
   "outputs": [],
   "source": [
    "!crontab crontab.e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E6bya5oorkcp",
    "outputId": "e6527066-d898-4d16-e63c-45ae3df96dd2"
   },
   "outputs": [],
   "source": [
    "!crontab -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OPlLdWkIyXla"
   },
   "source": [
    "# Exercise 20.3\n",
    "\n",
    "In this exercise, you will check the outputs of at least one CSV to understand if it's working as expected.\n",
    "In the cells below, please:\n",
    "\n",
    "* use the `!head` and `!tail` bash commands to inspect the most recent CSV. Does it contain the `sentiment` column?\n",
    "* load the most recent CSV with `pandas`. Take a look at a few of the inferences. Did your model perform OK in a qualitative sense? Note that we shouldn't expect perfect performance if the training set wasn't tweets. \n",
    "\n",
    "<!-- startquestion -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KZCZOd74yntB"
   },
   "outputs": [],
   "source": [
    "# Your work here"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "20_scheduled_batch_inference_lab.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}