Assignment 1

02_activities/assignments/.ipynb_checkpoints/assignment_1-checkpoint.ipynb

@@ -0,0 +1,240 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Working with parquet files\n",
+    "\n",
+    "## Objective\n",
+    "\n",
+    "+ In this assignment, we will use the data downloaded with the module `data_manager` to create features.\n",
+    "\n",
+    "(11 pts total)\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "+ This notebook assumes that price data is available to you in the environment variable `PRICE_DATA`. If you have not done so, then execute the notebook `production_2_data_engineering.ipynb` to create this data set.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "+ Load the environment variables using dotenv. (1 pt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext dotenv\n",
+    "%dotenv \n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "C:\\Users\\ibast\\anaconda3\\Lib\\site-packages\\pandas\\core\\arrays\\masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).\n",
+      "  from pandas.core import (\n"
+     ]
+    }
+   ],
+   "source": [
+    "import dask\n",
+    "dask.config.set({'dataframe.query-planning': True})\n",
+    "import dask.dataframe as dd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "TypeError",
+     "evalue": "expected str, bytes or os.PathLike object, not NoneType",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[3], line 6\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mglob\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m glob\n\u001b[0;32m      4\u001b[0m price_data_dir \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39mgetenv(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mPRICE_DATA\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m----> 6\u001b[0m parquet_files \u001b[38;5;241m=\u001b[39m glob(os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mjoin(price_data_dir, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m**/*.parquet/*.parquet\u001b[39m\u001b[38;5;124m\"\u001b[39m))\n\u001b[0;32m      8\u001b[0m \u001b[38;5;66;03m#  I had to limit number of files to 10, because it was taking too long to run.\u001b[39;00m\n\u001b[0;32m      9\u001b[0m parquet_files \u001b[38;5;241m=\u001b[39m parquet_files[:\u001b[38;5;241m10\u001b[39m]\n",
+      "File \u001b[1;32m<frozen ntpath>:108\u001b[0m, in \u001b[0;36mjoin\u001b[1;34m(path, *paths)\u001b[0m\n",
+      "\u001b[1;31mTypeError\u001b[0m: expected str, bytes or os.PathLike object, not NoneType"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "from glob import glob\n",
+    "\n",
+    "price_data_dir = os.getenv('PRICE_DATA')\n",
+    "\n",
+    "parquet_files = glob(os.path.join(price_data_dir, \"**/*.parquet/*.parquet\"))\n",
+    "\n",
+    "#  I had to limit number of files to 10, because it was taking too long to run.\n",
+    "parquet_files = parquet_files[:10]\n",
+    "parquet_files"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import dask.dataframe as dd"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "+ Load the environment variable `PRICE_DATA`.\n",
+    "+ Use [glob](https://docs.python.org/3/library/glob.html) to find the path of all parquet files in the directory `PRICE_DATA`.\n",
+    "\n",
+    "(1pt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from glob import glob\n",
+    "\n",
+    "# Write your code below.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For each ticker and using Dask, do the following:\n",
+    "\n",
+    "+ Add lags for variables Close and Adj_Close.\n",
+    "+ Add returns based on Adjusted Close:\n",
+    "    \n",
+    "    - `returns`: (Adj Close / Adj Close_lag) - 1\n",
+    "\n",
+    "+ Add the following range: \n",
+    "\n",
+    "    - `hi_lo_range`: this is the day's High minus Low.\n",
+    "\n",
+    "+ Assign the result to `dd_feat`.\n",
+    "\n",
+    "(4 pt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Write your code below.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "+ Convert the Dask data frame to a pandas data frame. \n",
+    "+ Add a rolling average return calculation with a window of 10 days.\n",
+    "+ *Tip*: Consider using `.rolling(10).mean()`.\n",
+    "\n",
+    "(3 pt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Write your code below.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please comment:\n",
+    "\n",
+    "+ Was it necessary to convert to pandas to calculate the moving average return?\n",
+    "+ Would it have been better to do it in Dask? Why?\n",
+    "\n",
+    "(1 pt)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Criteria\n",
+    "\n",
+    "The [rubric](./assignment_1_rubric_clean.xlsx) contains the criteria for grading."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Submission Information\n",
+    "\n",
+    "🚨 **Please review our [Assignment Submission Guide](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md)** 🚨 for detailed instructions on how to format, branch, and submit your work. Following these guidelines is crucial for your submissions to be evaluated correctly.\n",
+    "\n",
+    "### Submission Parameters:\n",
+    "* Submission Due Date: `HH:MM AM/PM - DD/MM/YYYY`\n",
+    "* The branch name for your repo should be: `assignment-1`\n",
+    "* What to submit for this assignment:\n",
+    "    * This Jupyter Notebook (assignment_1.ipynb) should be populated and should be the only change in your pull request.\n",
+    "* What the pull request link should look like for this assignment: `https://github.com/<your_github_username>/production/pull/<pr_id>`\n",
+    "    * Open a private window in your browser. Copy and paste the link to your pull request into the address bar. Make sure you can see your pull request properly. This helps the technical facilitator and learning support staff review your submission easily.\n",
+    "\n",
+    "Checklist:\n",
+    "- [ ] Created a branch with the correct naming convention.\n",
+    "- [ ] Ensured that the repository is public.\n",
+    "- [ ] Reviewed the PR description guidelines and adhered to them.\n",
+    "- [ ] Verify that the link is accessible in a private browser window.\n",
+    "\n",
+    "If you encounter any difficulties or have questions, please don't hesitate to reach out to our team via our Slack at `#cohort-3-help`. Our Technical Facilitators and Learning Support staff are here to help you navigate any challenges."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}