wineEnthusiastAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis and visualization of WineEnthusiast wine reviews\n",
    "Author: Manuele Nolli, student BSc Computer Science SUPSI \n",
    "\n",
    "Date: 28.11.2022\n",
    "\n",
    "Mail: manuele.nolli@student.supsi.ch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "This document is an analysis of a public dataset found on __[Kaggle.com](https://www.kaggle.com/datasets/manyregression/updated-wine-enthusiast-review)__\n",
    "\n",
    "The dataset contains 80k wine reviews with variety, location, winery, price, points, taster nam and description.\n",
    "\n",
    "My analysis will focus on the following questions:\n",
    "- Where are the wines produced?\n",
    "- What is the distribution of the points?\n",
    "- What is the distribution of the prices, and is it related to the points?\n",
    "- What is the distribution of the variety of wines?\n",
    "- How much tasters are there and how much reviews each of them has done?\n",
    "  - Are there tasters that are more reliable than others?\n",
    "  - Have the tasters a preference for a specific continent/country?\n",
    "- What are the most common words in the description of the wines?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import plotly.express as px\n",
    "import plotly.graph_objs as go\n",
    "import plotly.io as pio\n",
    "import pandas as pd\n",
    "from plotly.subplots import make_subplots\n",
    "pio.renderers.default = \"plotly_mimetype+notebook\"\n",
    "df=pd.read_csv(\"data/winemag-data-2017-2020.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datset details\n",
    "\n",
    "Whit the following code we can see the details of the dataset and how it is structured and the type of the columns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(f\"---Dataset Info---\")\n",
    "\n",
    "#printing column names\n",
    "print(f\"Total columns: {len(df.columns)}\")\n",
    "print(\"Columns names:\", end=\" \")\n",
    "for col in df:\n",
    "    if col == 'winery':\n",
    "        print(col, end=\".\")\n",
    "    else: \n",
    "        print(col, end=\", \")\n",
    "print()\n",
    "\n",
    "#columns types\n",
    "print(f\"Columns type:\")\n",
    "\n",
    "#creating temp array\n",
    "columnData = []\n",
    "dfIndexType = []\n",
    "\n",
    "for col in df.columns:\n",
    "    temp = []\n",
    "    dfIndexType.append(col)\n",
    "    temp.append(df[col].apply(type).unique())\n",
    "    temp.append(df[col].isnull().sum())\n",
    "    columnData.append(temp)\n",
    "\n",
    "#create new Dataframe\n",
    "dfColumnsType = pd.DataFrame(columnData, columns=['Types','NaN Count'])\n",
    "dfColumnsType.index = dfIndexType\n",
    "#print columns type\n",
    "display(dfColumnsType)\n",
    "\n",
    "#df size\n",
    "print(f\"Dataframe rows: {len(df)}\")\n",
    "\n",
    "#df sample\n",
    "print(\"Dataset samples:\")\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to see that the dataset contains 80k rows and 15 columns. The columns are:\n",
    "- __country__: the country of origin of wine\n",
    "- __description__: a few sentences from a sommelier describing the wine's taste, smell, look, feel, etc.\n",
    "- __designation__: the vineyard within the winery where the grapes that made the wine are from\n",
    "- __points__: the number of points WineEnthusiast rated the wine on a scale of 1-100 (though they say they only post reviews for wines that score >=80)\n",
    "- __price__: the cost for a bottle of the wine\n",
    "- __province__: the province or state that the wine is from\n",
    "- __region_1__: the wine growing area in a province or state (ie Napa)\n",
    "- __region_2__: sometimes there are more specific regions specified within a wine growing area (ie Rutherford inside the Napa Valley), but this value can sometimes be blank\n",
    "- __taster_name__: name of the person who tasted and reviewed the wine\n",
    "- __taster_photo__: url of the taster's photo\n",
    "- __taster_twitter_handle__: Twitter handle for the person who tasted and reviewed the wine\n",
    "- __title__: the title of the wine review\n",
    "- __variety__: the type of grapes used to make the wine (ie Pinot Noir)\n",
    "- __vintage__: the vintage of the wine\n",
    "- __winery__: the winery that made the wine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution of wines across continents\n",
    "\n",
    "In this section it is possible see the distribution of the wines across the continents. I used the __country__ column to see the distribution of the wines across the continents. \n",
    "I decided to create a new column called __continent__ that contains the continent of the country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Continent list\n",
    "europe = ['Austria', 'Bosnia and Herzegovina','Bulgaria','Croatia','Cyprus','Czech Republic','England', 'France','Germany','Greece','Italy','Luxembourg','Portugal','Hungary', 'Macedonia', 'Moldova', 'Romania', 'Serbia', 'Slovakia', 'Slovenia', 'Spain', 'Switzerland', 'Turkey', 'Ukraine']\n",
    "asia = ['Armenia', 'China','India','Israel','Lebanon' ]\n",
    "northAmerica = ['Canada','US','Mexico']\n",
    "sudAmerica = ['Argentina',',Brazil','Chile','Peru','Uruguay'] \n",
    "oceania = ['Australia','New Zealand'] \n",
    "africa = ['South Africa','Morocco']\n",
    "other = ['Egypt', 'Georgia']\n",
    "\n",
    "#Chose to set as 'Other' all the continent with a small amout of reviews \n",
    "def continentDispacher(row):\n",
    "    if row['country'] in europe:\n",
    "        val = 'Europe'\n",
    "    elif row['country'] in asia:\n",
    "        #val = 'Asia'\n",
    "        val = 'Other'\n",
    "    elif row['country'] in northAmerica:\n",
    "        val = 'North America'\n",
    "    elif row['country'] in sudAmerica:\n",
    "        #val = 'Sud America'\n",
    "        val = 'Other'\n",
    "    elif row['country'] in oceania:\n",
    "        #val = 'Oceania'\n",
    "        val = 'Other'\n",
    "    elif row['country'] in africa:\n",
    "        #val = 'Africa'\n",
    "        val = 'Other'\n",
    "    else:\n",
    "        val = 'Other'\n",
    "\n",
    "    return val\n",
    "\n",
    "df['continent'] = df.apply(continentDispacher,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows the distribution of the wines across the continents trough a pie chart.\n",
    "It is possible to see that the majority of the wines are produced in Europe, followed by North America."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Ditrubution of the wines by continent\n",
    "pieContinent = px.pie(df, names='continent', title='Distribution of wines across continents')\n",
    "pieContinent.update_traces(textposition='inside', textinfo='percent+label')\n",
    "pieContinent.update(layout_showlegend=False)\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "pieContinent.update_layout(\n",
    "    title={\n",
    "        'x':0.5,\n",
    "        'xanchor': 'center',\n",
    "        'yanchor': 'top'},\n",
    "        font=dict(\n",
    "        size=18),\n",
    "        height=1000,\n",
    "        width=1000)\n",
    "\"\"\"\n",
    "\n",
    "pieContinent.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#groupby country for have count\n",
    "dfCountry = df.groupby('country').count().reset_index()\n",
    "dfCountry = dfCountry[['country','continent']]\n",
    "dfCountry.columns = ['country','count']\n",
    "\n",
    "#display dfCountry in a maps\n",
    "fig = px.choropleth(dfCountry, locations=\"country\", locationmode='country names', color=\"count\", hover_name=\"country\", color_continuous_scale=px.colors.sequential.RdPu)\n",
    "\n",
    "#more realistic map\n",
    "fig.update_geos(projection_type=\"natural earth\")\n",
    "\n",
    "#update layout for enlarge the map\n",
    "fig.update_layout(margin={\"r\":0,\"t\":50,\"l\":0,\"b\":0},title = 'Wine distribution across countries')\n",
    "\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "fig.update_layout(\n",
    "    title={\n",
    "        'x':0.5,\n",
    "        'xanchor': 'center',\n",
    "        'yanchor': 'top'},\n",
    "        font=dict(\n",
    "        size=18),\n",
    "        height=1000,\n",
    "        width=2000)\n",
    "\"\"\"\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#groupby continent, country, region1 and region2 for have count\n",
    "dfRegion = df.groupby(['continent','country','region_1','region_2'], dropna=False).count().reset_index()\n",
    "dfRegion = dfRegion[['continent','country','region_1','region_2','points']]\n",
    "dfRegion.columns = ['continent','country','region_1','region_2','count']\n",
    "\n",
    "#I can't find a best way to show the data with region1 or 2 as null\n",
    "dfRegion.fillna('None', inplace=True)\n",
    "\n",
    "fig = px.treemap(dfRegion, path=[\"continent\", 'country', 'region_1', 'region_2'],branchvalues=\"total\", values='count', title='Wine distribution across countries')\n",
    "fig.show()\n",
    "\n",
    "#create a sunburst chart\n",
    "fig = px.sunburst(dfRegion, path=[\"continent\", 'country', 'region_1', 'region_2'], values='count', title='Wine distribution across countries')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above chart is an alternative way to see the distribution of the wines across the continents. It is more interactive and it is possible to see the exact number of wines produced in each continent, country and region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Points distribution\n",
    "Another interesting aspect of the dataset is the distribution of the points. The points are given by the tasters and they are on a scale from 80 to 100 and WineEnthusiast has another way to group the wine by 5 categories: \n",
    "- __80–82: ACCEPTABLE__ Can be employed\n",
    "- __83–86: GOOD__ Suitable for everyday consumption; often good value\n",
    "- __87–89: VERY GOOD__ Often good value; well recommended\n",
    "- __90–93: EXCELLENT__ Highly recommended\n",
    "- __94–97: SUPERB__ A great achievement\n",
    "- __98–100: CLASSIC__ The pinnacle of quality\n",
    "\n",
    "In the following section a new column called __pointsDescription__ is created that contains the description of the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Create new column with points description\n",
    "def pointsDispacher(points):\n",
    "    if points < 83:\n",
    "        val = 'Acceptable'\n",
    "    elif points < 87:\n",
    "        val = 'Good'\n",
    "    elif points < 90:\n",
    "        val = 'Very good'\n",
    "    elif points <93:\n",
    "        val = 'Excellent'\n",
    "    elif points <97:\n",
    "        val = 'Superb'\n",
    "    else:\n",
    "        val = 'Classic'\n",
    "\n",
    "    return val\n",
    "\n",
    "#Create new column with points description\n",
    "df['pointsDescription'] = df['points'].map(pointsDispacher)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Histogram of points description\n",
    "pointDistribution = px.histogram(df, x='points', color='pointsDescription', title='Points distribution', height=500,\n",
    " category_orders=dict(pointsDescription=['Classic', 'Superb', 'Excellent', 'Very good', 'Good','Acceptable']), \n",
    "                  labels={\n",
    "                     \"pointsDescription\": \"Point Description\"\n",
    "                 },\n",
    "                 color_discrete_map = {'Classic':'#903f5c','Superb':'#006179','Excellent':'#008377','Very good':'#09a259', 'Good':'#90b827', 'Acceptable':'#ffbf00'}\n",
    "\n",
    ")\n",
    "#Update axis\n",
    "pointDistribution.update_xaxes(title='Point',tickmode='linear')\n",
    "pointDistribution.update_yaxes(title='Count')\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "pointDistribution.update_layout(\n",
    "    title={\n",
    "        'x':0.5,\n",
    "        'xanchor': 'center',\n",
    "        'yanchor': 'top'},\n",
    "        font=dict(\n",
    "        size=18),\n",
    "        height=700,\n",
    "        width=2000)\n",
    "\"\"\"\n",
    "\n",
    "pointDistribution.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this graph it is possible to see that the majority of the wines are in the __Good__ category, followed by the __very good__ category (the middles scores are the most common).\n",
    "\n",
    "It is curious to see that there are more wines with 90 points than with 89 points. That is probably because the tasters are more likely to give a wine 90 points than 89 points to have the wine labeled as __Excellent__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vintage distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section it is possible to see the distribution of the vintage of the wines. The vintage is the year in which the grapes were harvested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "dfVintageWithoutNaN = df.copy()\n",
    "\n",
    "#Remove 'NV' string = NotVintage, when multiple kind of wine of different years are blended \n",
    "dfVintageWithoutNaN = dfVintageWithoutNaN[dfVintageWithoutNaN['vintage'] != 'NV']\n",
    "\n",
    "dfVintageWithoutNaN['vintage'] = pd.DatetimeIndex(dfVintageWithoutNaN['vintage']).year\n",
    "\n",
    "#Removing impossible data\n",
    "dfVintageWithoutNaN = dfVintageWithoutNaN[dfVintageWithoutNaN['vintage'] < datetime.datetime.now().year] #year in the future\n",
    "\n",
    "#Removing wine with year as a title (for doing that I assume that an old wine cost at least 100)\n",
    "dfVintageWithoutNaN = dfVintageWithoutNaN.drop(dfVintageWithoutNaN[(dfVintageWithoutNaN['vintage'] < 1980) & (dfVintageWithoutNaN['price'] < 100) |(dfVintageWithoutNaN['price'].isna())].index)\n",
    "\n",
    "#Histogram of vintage distribution\n",
    "vintageDistribution = px.histogram(dfVintageWithoutNaN, x=\"vintage\", title='Vintage review distribution')\n",
    "\n",
    "#Update axis\n",
    "vintageDistribution.update_xaxes(title='Year',dtick=1)\n",
    "vintageDistribution.update_yaxes(title='Count')\n",
    "\n",
    "vintageDistribution.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It must be remembered that the dataset contains wines reviewed beetwen 2017 and 2020. It is normal to see that the majority of the wines are from the past years. But, there are also some very old wines in the dataset. The oldest wine is from 1931 and surprisely it does not have a very high score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "dfVintageWithoutNaN.loc[dfVintageWithoutNaN['vintage'] == 1931]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine variety\n",
    "\n",
    "In this section it is possible to see the distribution of the variety of the wines. The variety is the type of grapes used to make the wine (ie Pinot Noir). In the dataset there are many different varieties of wines but I decided to show only the top 10 varieties. It is possible to change this settings by changing the __wineCountToShow__ variable.\n",
    "\n",
    "Firstly, I created different versions of the dataset that thy will be used to create the graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Wine to be shown\n",
    "wineCountToShow = 10\n",
    "\n",
    "# Top {wineCountToShow} wine variety with the highest count\n",
    "dfMostWineVariety = df.groupby(['variety']).size().to_frame().sort_values([0], ascending = False).head(wineCountToShow).reset_index()\n",
    "dfMostWineVariety.columns.values[1] = 'count'\n",
    "\n",
    "# Other wine variety\n",
    "dfOtherWineVariety = df.groupby(['variety']).size().to_frame().sort_values([0], ascending = False).tail(len(df.groupby(['variety']).size()) - wineCountToShow).reset_index()\n",
    "dfOtherWineVariety.columns.values[1] = 'count'\n",
    "\n",
    "\n",
    "#Create order of bars\n",
    "order = dfMostWineVariety['variety'].tolist()\n",
    "order.reverse()\n",
    "order = ['Other'] + order\n",
    "\n",
    "# Top {wineCountToShow} wine variety with the highest count and the price\n",
    "dfFiltered = df.copy()\n",
    "dfFiltered = dfFiltered.loc[df['variety'].isin(dfMostWineVariety['variety'])]\n",
    "dfFilteredPoints = dfFiltered.groupby(['variety']).agg({'points': ['mean']}).reset_index()\n",
    "\n",
    "# Other wine variety\n",
    "dfFilteredOtherWine = df.loc[df['variety'].isin(dfOtherWineVariety['variety'])]\n",
    "dfFilteredOtherWinePoints = dfFilteredOtherWine.groupby(['variety']).agg({'points': ['mean']}).reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now is finally the time to create the graphs. The left graph is a bar chart that shows the distribution of the wines, the center graph is another bar chart that shows the average points of the wines and the right graph is a box plot that shows the distribution of the prices of the wines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "groupbypoints = df.groupby(['pointsDescription','points']).size().to_frame().reset_index()\n",
    "groupbypoints.columns.values[2] = 'count'\n",
    "topReviewedWines = make_subplots(rows=1, cols=3,subplot_titles=('Reviews count',\"Variety average points\",\"Price distribution\"), shared_yaxes=True,horizontal_spacing = 0.025\n",
    ")\n",
    "\n",
    "\n",
    "#Variety average points \n",
    "trace1 = go.Bar(y=dfFilteredPoints['variety'], x=dfFilteredPoints['points']['mean'],orientation='h',marker_color='rgba(101, 109, 255, 1)')\n",
    "trace2 = go.Bar(x=[dfFilteredOtherWinePoints['points']['mean'].mean()], y=['Other'], name='Other', orientation='h',marker_color='rgba(55, 83, 109, 0.6)')\n",
    "\n",
    "#Wine reviews based on variety\n",
    "trace3 = go.Bar(y=dfMostWineVariety['variety'], x=dfMostWineVariety['count'], name='Top variety', orientation='h',marker_color='rgba(101, 109, 255, 1)')\n",
    "\n",
    "trace4 = go.Bar(x=[dfOtherWineVariety['count'].sum()], y=['Other'], name='Other', orientation='h',marker_color='rgba(55, 83, 109, 0.6)')\n",
    "\n",
    "#Price distribution\n",
    "trace5 = go.Box(x=dfFiltered['price'], y=dfFiltered['variety'], orientation='h',marker_color='rgba(101, 109, 255, 1)')\n",
    "\n",
    "trace6 = go.Box(x=dfFilteredOtherWine['price'], name='Other', orientation='h',marker_color='rgba(55, 83, 109, 0.6)')\n",
    "\n",
    "#Add traces\n",
    "topReviewedWines.add_trace(trace1, row=1, col=2)\n",
    "topReviewedWines.add_trace(trace2, row=1, col=2)\n",
    "topReviewedWines.add_trace(trace3, row=1, col=1)\n",
    "topReviewedWines.add_trace(trace4, row=1, col=1)\n",
    "topReviewedWines.add_trace(trace5, row=1, col=3)\n",
    "topReviewedWines.add_trace(trace6, row=1, col=3)\n",
    "\n",
    "#General layout\n",
    "topReviewedWines.update_yaxes(categoryorder='array',categoryarray=order)\n",
    "topReviewedWines.update_layout(showlegend=False)\n",
    "topReviewedWines.update_layout(title=f'[top {wineCountToShow}] Reviewed wines')\n",
    "\n",
    "#update title yaxis\n",
    "topReviewedWines.update_yaxes(title_text='Wine variety', row=1, col=1)\n",
    "\n",
    "#left graph layout\n",
    "topReviewedWines.update_xaxes(title_text=\"Count\", col=1)\n",
    "topReviewedWines.update_xaxes(dtick=5000, col=1)\n",
    "\n",
    "#center graph layout\n",
    "topReviewedWines.update_xaxes(title_text=\"Point\",  col=2)\n",
    "topReviewedWines.update_xaxes(range=[80, 100], col=2)\n",
    "topReviewedWines.update_xaxes(dtick=2, col=2)\n",
    "#right graph layout\n",
    "topReviewedWines.update_xaxes(title_text=\"Price USD\", col=3)\n",
    "topReviewedWines.update_xaxes(type=\"log\", range=[0,4],  col=3)\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "topReviewedWines.update_layout(\n",
    "        font=dict(\n",
    "        size=25),\n",
    "        height=1000,\n",
    "        width=3000)\n",
    "topReviewedWines.update_layout(title_font_size=1)\n",
    "topReviewedWines.update_annotations(font_size=50)\n",
    "\"\"\"\n",
    "\n",
    "#Finally show the graph\n",
    "topReviewedWines.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is interesting to see that the other varieties have a lot more reviews than the top 10 varieties, this means that the dataframe is well balanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine - Price connection\n",
    "\n",
    "There are two principal graph in this section, the first one show a box plot rappresenting the distribution of the prices by points and the second one show a percentage  histogram of the prices grouped by a personal price description:\n",
    "- __x-10   usd: Low__ \n",
    "- __11–40  usd: Medium__ \n",
    "- __41–100 usd: Expensive__\n",
    "- __100–x  usd: Luxury__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Offsetting the price\n",
    "lowOffset = 10\n",
    "mediumOffset = 40\n",
    "expensiveOffset = 100\n",
    "\n",
    "#Function to create a new column with the price range\n",
    "def priceDispacher(price):\n",
    "    if price <= lowOffset:\n",
    "        val = 'Low'\n",
    "    elif price <= mediumOffset:\n",
    "        val = 'Medium'\n",
    "    elif price <= expensiveOffset:\n",
    "        val = 'Expensive'\n",
    "    else:\n",
    "        val = 'Luxury'\n",
    "    return val\n",
    "\n",
    "#Apply priceDispacher function to price column\n",
    "df['priceDescription'] = df['price'].map(priceDispacher)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "boxPricePoint = go.Figure()\n",
    "boxPricePoint.add_trace(go.Box(x=df['points'], y=df['price'], orientation='v',marker_color='rgba(101, 109, 255, 1)', boxmean=True))\n",
    "\n",
    "boxPricePoint.update_layout(xaxis_range=[79.5, 100.5], title='Price vs Points')\n",
    "\n",
    "boxPricePoint.update_xaxes(title='Point', dtick=1)\n",
    "boxPricePoint.update_yaxes(title='Price USD',type=\"log\")\n",
    "boxPricePoint.update_yaxes()\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "boxPricePoint.update_layout(\n",
    "        font=dict(\n",
    "        size=25),\n",
    "        height=800,\n",
    "        width=3000)\n",
    "\"\"\"\n",
    "boxPricePoint.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the box plot it is possible to see that the wines with the highest points are the most expensive as could be expected, so there is a strong connection between the price and the points. This is also confirmed by the following histogram that shows that the wines with the highest points are the most expensive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "averagepricePoint = px.histogram(df,x='points', color='priceDescription', barmode='stack', barnorm='percent',\n",
    " category_orders=dict(priceDescription=['Low', 'Medium', 'Expensive', 'Luxury']), title='Price distribution by points', labels={\n",
    "                     \"priceDescription\": \"Price Description\"\n",
    "                 }, color_discrete_sequence=px.colors.sequential.Teal\n",
    "                 )\n",
    "\n",
    "averagepricePoint.update_xaxes(title='Point', dtick=1)\n",
    "averagepricePoint.update_yaxes(title='Count %')\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "averagepricePoint.update_layout(\n",
    "        font=dict(\n",
    "        size=25),\n",
    "        height=800,\n",
    "        width=3000)\n",
    "\"\"\"\n",
    "\n",
    "averagepricePoint.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is curious to see that there are some wines with a very high price and a very low points and in the other side there are some wines with a very low price and a very high points. This means that the price is not the only factor that influence the points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "I tried to create a graph object with the past two graph connected by the x-axis but it is currently not possible to do that with plotly.\n",
    "Further information: https://community.plotly.com/t/how-to-set-barmode-for-individual-subplots/47931"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reviewer distribution\n",
    "\n",
    "Now it is time to see the distribution of the reviewers. I am interested in seeing how many reviewers there are and how many reviews each of them has done. I also want to see if there are some reviewers that are more reliable than others and if there are some reviewers that are more likely to review wines from a specific continent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from itertools import product\n",
    "\n",
    "tasterDistribution = make_subplots(rows=1, cols=3,subplot_titles=('Count',\"Points distribution\",\"Continent distribution\"), shared_yaxes=True,horizontal_spacing = 0.01)\n",
    "\n",
    "#Taster review count\n",
    "trace1 = go.Histogram(y=df['taster_name'], name='Taster review count', marker_color='rgba(101, 109, 255, 1)')\n",
    "\n",
    "#Point awarded\n",
    "trace2 = go.Box(x=df['points'], y=df['taster_name'], name='Point awarded', orientation='h',marker_color='rgba(101, 109, 255, 1)' )\n",
    "\n",
    "#Continent preference by taster\n",
    "#groupby continent and taster and average\n",
    "dfContinentTaster = df.groupby(['continent','taster_name']).size().reset_index(name='reviewPerContinent')\n",
    "totReviewPerTaster = df.groupby(['taster_name'])['continent'].count().reset_index(name='totalReview')\n",
    "\n",
    "##Merge the two dataframe into one##\n",
    "#create a list of all the possible combination of taster and continent\n",
    "combs = pd.DataFrame(list(product(df['continent'].unique(), df['taster_name'].unique())), \n",
    "                     columns=['continent', 'taster_name'])\n",
    "\n",
    "#merge dfContinentTaster and combs for all the possible combination (goal: fill the missing value with 0)\n",
    "dfContinentTaster = dfContinentTaster.merge(combs, how = 'right').fillna(0)\n",
    "\n",
    "#finally merge with the total review per taster\n",
    "dfContinentTaster = dfContinentTaster.merge(totReviewPerTaster, on='taster_name')\n",
    "\n",
    "trace3 = go.Heatmap( x=dfContinentTaster['continent'], y=dfContinentTaster['taster_name'],z=(dfContinentTaster['reviewPerContinent']/dfContinentTaster['totalReview'])*100, name='Continent preference by taster', colorscale='Blues', colorbar=dict(title='Count %')) \n",
    "\n",
    "#create order by review count\n",
    "order = df['taster_name'].value_counts().index \n",
    "\n",
    "#update layout\n",
    "tasterDistribution.update_yaxes(categoryorder='array',categoryarray=order)\n",
    "tasterDistribution.update_layout(showlegend=False, title='Taster review')\n",
    "\n",
    "#layout for the first graph\n",
    "tasterDistribution.update_xaxes(title='Count', row=1, col=1)\n",
    "tasterDistribution.update_yaxes(title='Taster name', row=1, col=1)\n",
    "\n",
    "#layout for the secondo graph\n",
    "tasterDistribution.update_xaxes(title='Point awarded', dtick=2, range=[79.5, 100.5],row=1, col=2)\n",
    "\n",
    "#layout for the third graph\n",
    "tasterDistribution.update_xaxes(title='Continent', row=1, col=3)\n",
    "\n",
    "#set background color\n",
    "\n",
    "#add traces to the graph\n",
    "tasterDistribution.add_trace(trace1, row=1, col=1)\n",
    "tasterDistribution.add_trace(trace2, row=1, col=2)\n",
    "tasterDistribution.add_trace(trace3, row=1, col=3)\n",
    "\n",
    "#update layout for export\n",
    "\"\"\"\n",
    "tasterDistribution.update_layout(\n",
    "        font=dict(\n",
    "        size=25),\n",
    "        height=1000,\n",
    "        width=3000)\n",
    "tasterDistribution.update_layout(title_font_size=1)\n",
    "tasterDistribution.update_annotations(font_size=50)\n",
    "\"\"\"\n",
    "\n",
    "tasterDistribution.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are different considerations to make:\n",
    "- There are in total 19 reviewers and some of them have done a huge amount of reviews, as example the reviewer __Roger Voss__ has more than 17k reviews, that are more than 15 reviews per day for 3 years.\n",
    "- The graph in the center shows the distribution of the point awarded by the reviewers. It is possible to see that the reviewers are very consistent in the points they give to the wines.\n",
    "- The graph on the right shows the preference of the reviewers for a specific continent. It is possible to see that the reviewers are more likely to review wines from their continent (example: Roger Voss and Kerin O'Keefe live in Europe and Virginie Boone and Matt Kettmann live in North America)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most used words in wine description for points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I decided to represent the most used words in the description of the wines for each point. I used the __description__ column to extract the words after a cleaning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Most used words in wine description for each point\n",
    "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
    "\n",
    "import string\n",
    "import re\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import nltk.corpus\n",
    "#nltk.download('stopwords')\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "#Function to clean the description\n",
    "def cleanDescription(description):\n",
    "    #remove punctuation\n",
    "    description = description.translate(str.maketrans('', '', string.punctuation))\n",
    "    #remove number\n",
    "    description = re.sub(r'\\d+', '', description)\n",
    "    #remove space\n",
    "    description = description.strip()\n",
    "    #remove stopword\n",
    "    description = [word for word in description.split() if word not in stopwords.words('english')]\n",
    "    #remove short word\n",
    "    description = [word for word in description if len(word) > 2]\n",
    "    #remove word with number\n",
    "    description = [word for word in description if not any(c.isdigit() for c in word)]\n",
    "    #remove word with special character\n",
    "    description = [word for word in description if not any(c in string.punctuation for c in word)]\n",
    "    #remove string The (trivial word)\n",
    "    description = [word for word in description if not word == 'The']\n",
    "    #remove string Wine (trivial word)\n",
    "    description = [word for word in description if not word == 'Wine']\n",
    "    description = [word for word in description if not word == 'wine']\n",
    "    #remove string This (trivial word)\n",
    "    description = [word for word in description if not word == 'This']\n",
    "    #remove word with underscore\n",
    "    description = [word for word in description if not any(c == '_' for c in word)]\n",
    "    #remove word with dash\n",
    "    description = [word for word in description if not any(c == '-' for c in word)]\n",
    "    #remove word with slash\n",
    "    description = [word for word in description if not any(c == '/' for c in word)]\n",
    "    #remove word with backslash\n",
    "    description = [word for word in description if not any(c == '\\\\' for c in word)]\n",
    "    #remove word with dot\n",
    "    description = [word for word in description if not any(c == '.' for c in word)]\n",
    "    #remove word with comma\n",
    "    description = [word for word in description if not any(c == ',' for c in word)]\n",
    "    #remove word with colon\n",
    "    description = [word for word in description if not any(c == ':' for c in word)]\n",
    "    #remove word with semicolon\n",
    "    description = [word for word in description if not any(c == ';' for c in word)]\n",
    "    #remove word with exclamation mark\n",
    "    description = [word for word in description if not any(c == '!' for c in word)]\n",
    "    #remove word with question mark\n",
    "    description = [word for word in description if not any(c == '?' for c in word)]\n",
    "    \n",
    "    return description\n",
    "\n",
    "#Function to create the wordcloud\n",
    "def createWordCloud(description, title):\n",
    "    #create wordcloud\n",
    "    wordcloud = WordCloud(width = 500, height = 500,\n",
    "                min_font_size = 10, \n",
    "                background_color ='white').generate(description) \n",
    "    # plot the WordCloud image                        \n",
    "    #plt.figure(figsize = (25,25), facecolor = None) \n",
    "    plt.imshow(wordcloud) \n",
    "    plt.axis(\"off\") \n",
    "    plt.tight_layout(pad = 0) \n",
    "    #plt.title(title, fontsize=50)\n",
    "    plt.show()\n",
    "\n",
    "#Function to create the wordcloud for each point\n",
    "def createWordCloudForPoint(df, pointsDescription):\n",
    "    #filter by point\n",
    "    dfPoint = df.loc[df['pointsDescription'] == pointsDescription]\n",
    "    #clean description\n",
    "    dfPoint['description'] = dfPoint['description'].apply(lambda x: cleanDescription(x))\n",
    "    #join all the description\n",
    "    description = ''.join(' '.join(l) for l in dfPoint['description'].values)\n",
    "\n",
    "########################\n",
    "#Remove the comment below to save in a dataframe the most used word for each point\n",
    "#find most 10 used word in the description save it in a dataframe\n",
    "#dfMostUsedWord = pd.DataFrame(description.split(), columns=['word']).word.value_counts().reset_index().rename(columns={'index':'word', 'word':'count'}).head(10)\n",
    "#print(point)\n",
    "#display(dfMostUsedWord)\n",
    "########################\n",
    "    #create wordcloud\n",
    "    createWordCloud(description, 'Most used words of \\'' + str(pointsDescription) + '\\' category')\n",
    "\n",
    "#remove warning\n",
    "pd.set_option('mode.chained_assignment', None)\n",
    "#Create wordcloud for each point\n",
    "for point in set(df['pointsDescription']):\n",
    "    print(point)\n",
    "    createWordCloudForPoint(df, point)\n",
    "\n",
    "#reset warning\n",
    "pd.reset_option('mode.chained_assignment')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "81210f027eba10c5114049bce6bccbd99970a43b87478dd01b0835817ecff633"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}