\n",
- " \n",
- " 
\n",
- " \n",
- "
\n",
- "\n",
- " \n",
- " \n",
- " \n",
- " NYC taxi subset\n",
- " \n",
- "
Feature Layer Collection by api_data_owner\n",
- "
Last Modified: May 17, 2019\n", - "
0 comments, 462 views\n", - "
\n",
- " Feature Layer Collection by api_data_owner\n",
- " Last Modified: May 17, 2019\n", - "
0 comments, 462 views\n", - "
\n",
- " 
Big Data File Share by arcgis_python\n",
- " 
Feature Layer Collection by arcgis_python\n",
- " 
\n",
- " 
Feature Layer Collection by api_data_owner\n",
- " 
\n",
- " 
Feature Layer Collection by admin\n",
- " 
Table Layer by admin\n",
- " 
\n", " 
\n", " 
Feature Layer Collection by admin\n", " 
Table Layer by admin\n", " 
\n", " 
Dashboard by api_data_owner\n", " 
"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Since the recent past, agencies such as the [National Hurricane Center](https://www.nhc.noaa.gov/aboutintro.shtml) have been collecting quantitative data about hurricanes. In this study we use meteorological data of hurricanes recorded in the past `169` years to analyze their location, intensity and investigate if there are any statistically significant trends. We also analyze the places most affected by hurricanes and what their demographic make up is. We conclude by citing relevant articles that draw similar conclusions."]}, {"cell_type": "markdown", "metadata": {}, "source": ["This notebook covers part 1 of this study. In this notebook, we:\n", "\n", " - download data from NCEI portal\n", " - do extensive pre-processing in the form of clearing headers, merging redundant columns\n", " - aggregate the observations into hurricane tracks."]}, {"cell_type": "markdown", "metadata": {}, "source": ["**Note**: To run this sample, you need a few extra libraries in your conda environment. If you don't have the libraries, install them by running the following commands from cmd.exe or your shell."]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["pip install dask==2.14.0\n", "pip install toolz\n", "pip install fsspec==0.3.1"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Download hurricane data from NCEI FTP portal\n", "The [National Centers for Environmental Information](https://www.ncdc.noaa.gov/), formerly [National Climatic Data Center](https://www.ncdc.noaa.gov/) shares the historic hurricane track datasets at [ftp://eclipse.ncdc.noaa.gov/pub/ibtracs/v03r09/all/csv/](ftp://eclipse.ncdc.noaa.gov/pub/ibtracs/v03r09/all/csv/). We use the `ftplib` Python library to login in and download these datasets."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": ["# imports for downloading data from FTP site\n", "import os\n", "from ftplib import FTP\n", "\n", "# imports to process data using DASK\n", "from dask import delayed\n", "import dask.dataframe as ddf\n", "\n", "# imports for data analysis and visualization\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "# imports to perform spatial aggregation using ArcGIS GeoAnalytics server\n", "from arcgis.gis import GIS\n", "from arcgis.geoanalytics import get_datastores\n", "from arcgis.geoanalytics.summarize_data import reconstruct_tracks\n", "import arcgis\n", "\n", "# miscellaneous imports\n", "from pprint import pprint\n", "from copy import deepcopy"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Establish an anonymous connection to FTP site."]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [{"data": {"text/plain": ["'230 Anonymous access granted, restrictions apply'"]}, "execution_count": 4, "metadata": {}, "output_type": "execute_result"}], "source": ["conn = FTP(host='eclipse.ncdc.noaa.gov')\n", "conn.login()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Change directory to folder containing the hurricane files. List the files."]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [{"data": {"text/plain": ["176"]}, "execution_count": 5, "metadata": {}, "output_type": "execute_result"}], "source": ["conn.cwd('/pub/ibtracs/v03r10/all/csv/year/')\n", "file_list = conn.nlst()\n", "len(file_list)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Print the top 10 items."]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [{"data": {"text/plain": ["['Year.1842.ibtracs_all.v03r10.csv',\n", " 'Year.1843.ibtracs_all.v03r10.csv',\n", " 'Year.1844.ibtracs_all.v03r10.csv',\n", " 'Year.1845.ibtracs_all.v03r10.csv',\n", " 'Year.1846.ibtracs_all.v03r10.csv',\n", " 'Year.1847.ibtracs_all.v03r10.csv',\n", " 'Year.1848.ibtracs_all.v03r10.csv',\n", " 'Year.1849.ibtracs_all.v03r10.csv',\n", " 'Year.1850.ibtracs_all.v03r10.csv',\n", " 'Year.1851.ibtracs_all.v03r10.csv']"]}, "execution_count": 6, "metadata": {}, "output_type": "execute_result"}], "source": ["file_list[:10]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Download each file into the `hurricanes_raw` directory"]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": ["data_dir = r'data/hurricanes_data/'"]}, {"cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [{"data": {"text/plain": ["['Allstorms.ibtracs_all.v03r09.csv',\n", " 'hurricanes_raw',\n", " '.nb_auth_file',\n", " 'Allstorms.ibtracs_all.v03r09.csv.gz']"]}, "execution_count": 31, "metadata": {}, "output_type": "execute_result"}], "source": ["if 'hurricanes_raw' not in os.listdir(data_dir):\n", " os.mkdir(os.path.join(data_dir,'hurricanes_raw'))\n", "\n", "hurricane_raw_dir = os.path.join(data_dir,'hurricanes_raw')\n", "os.listdir(data_dir)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Now we are going to download data from 1842-2017, whih might take around 15 mins."]}, {"cell_type": "code", "execution_count": 11, "metadata": {"scrolled": true}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloaded Year.1842.ibtracs_all.v03r10.csv\n", "Downloaded Year.1843.ibtracs_all.v03r10.csv\n", "Downloaded Year.1844.ibtracs_all.v03r10.csv\n", "Downloaded Year.1845.ibtracs_all.v03r10.csv\n", "....Downloaded Year.2015.ibtracs_all.v03r10.csv\n", "Downloaded Year.2016.ibtracs_all.v03r10.csv\n", "Downloaded Year.2017.ibtracs_all.v03r10.csv\n", "CPU times: user 8.63 s, sys: 12.1 s, total: 20.8 s\n", "Wall time: 12min 5s\n"]}], "source": ["file_path = hurricane_raw_dir\n", "for file in file_list:\n", " with open(os.path.join(file_path, file), 'wb') as file_handle:\n", " try:\n", " conn.retrbinary('RETR ' + file, file_handle.write, 1024)\n", " print(f'Downloaded {file}')\n", " \n", " except Exception as download_ex:\n", " print(f'Error downloading {file} + {str(download_ex)}')"]}, {"cell_type": "markdown", "metadata": {"heading_collapsed": true}, "source": ["## Process CSV files by removing header rows\n", "The CSV files have multiple header rows. Let us start by processing one of the files as an example."]}, {"cell_type": "code", "execution_count": 2, "metadata": {"hidden": true}, "outputs": [], "source": ["csv_path = os.path.join(hurricane_raw_dir,'Year.2017.ibtracs_all.v03r10.csv')"]}, {"cell_type": "code", "execution_count": 18, "metadata": {"hidden": true}, "outputs": [{"data": {"text/html": ["