hw1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HSE 2021: Mathematical Methods for Data Analysis\n",
    "\n",
    "## Homework 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention!\n",
    "* For tasks where <ins>text answer</ins> is required **Russian language** is **allowed**.\n",
    "* If a task asks you to describe something (make coclusions) then **text answer** is **mandatory** and **is** part of the task\n",
    "* **Do not** upload the dataset (titanic.csv) to the grading system (we already have it)\n",
    "* We **only** accept **ipynb** notebooks. If you use Google Colab then you'll have to download the notebook before passing the homework\n",
    "* **Do not** use python loops instead of NumPy vector operations over NumPy vectors - it significantly decreases performance (see why https://blog.paperspace.com/numpy-optimization-vectorization-and-broadcasting/), will be punished with -0.25 for **every** task. \n",
    "Loops are only allowed in part 1 (Tasks 1 - 4). \n",
    "* Some tasks contain tests. They only test you solution on a simple example, thus, passing the test does **not** guarantee you the full grade for the task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Python (2 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 1** (0.5 points)<br/> \n",
    "Enter numbers `a`,  `d` и `N`. Find sum of the first `N` members of arithmetic progression with the first member `a` and difference `d`, without using formula for the sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_sum(a,d,N):\n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 2\n",
    "d = 3\n",
    "N = 15\n",
    "assert find_sum(a,d,N) == 345, 'Wrong answer'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 2** (0.5 points) <br/> \n",
    "Enter an integer number `N`. Check if it is a prime number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_prime(N):\n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp = [3, 8, 13, 111, 113, 237, 307, 479, 481]\n",
    "answer = [True, False, True, False, True, False, True, True, False]\n",
    "\n",
    "for N, ans in zip(inp, answer):\n",
    "    assert check_prime(N) == ans, f'Wrong answer for input {N}, {ans} is expected'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 3** (0.5 points) <br/> \n",
    "Find the first `N` prime numbers (you may use the function from the previous task)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_primes(N):\n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert find_primes(5) == [2, 3, 5, 7, 11], 'Wrong answer'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 4** (0.5 points) <br/> \n",
    "There are numbers: `a`, `b`, `c`. Without using functions `min`, `max` or any other functions, find the maximum number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_max(a, b, c):\n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import randint\n",
    "for i in range(10):    \n",
    "    a = randint(-100, 100)\n",
    "    b = randint(-100, 100)\n",
    "    c = randint(-100, 100)\n",
    "    answer = np.max([a, b, c])  # You cannot use `np.max` in the solution ;)\n",
    "    print(f'Checking on {a, b, c}')\n",
    "    assert find_max(a, b, c) == answer, f'Wrong maximum, {answer} is expected'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Numpy (4 points)\n",
    "### **!Do not use loops in the tasks below, see disclamer at the top of the notebook**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 1** (0.5 points) <br/>\n",
    "Create a random array (`np.random.rand()`) with length of 20 and with sum of its elements equals to 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_array = ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(my_array) == 20\n",
    "assert np.allclose(np.sum(my_array), 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 2** (0.5 points) <br/>\n",
    "Create two random arrays $a$ and $b$ with the same length. \n",
    "\n",
    "Calculate the following distances between the arrays **without using special functions. You may only use basic numpy operations (first level functions like `np.dot` are ok, higher level functions like `np.linalg.*` are prohibited).**:\n",
    "\n",
    "* Euclidean Distance\n",
    "$$ d(a, b) = \\sqrt{\\sum_i (a_i - b_i)^2} $$\n",
    "* Manhattan Distance\n",
    "$$ d(a, b) = \\sum_i |a_i - b_i| $$\n",
    "* Cosine Distance\n",
    "$$ d(a, b) = 1 - \\frac{a^\\top b}{||a||_2\\cdot||b||_2}, \\,\\, \\text{where}\\,\\, ||a||_2 = \\sqrt{\\sum_i a_i^2} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cal_euclidean(a, b):    \n",
    "    ## Your code here\n",
    "    \n",
    "\n",
    "def cal_manhattan(a, b):    \n",
    "    ## Your code here\n",
    "\n",
    "    \n",
    "def cal_cosine(a, b):    \n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.ones(10)*0.1\n",
    "b = np.ones(10)\n",
    "\n",
    "assert np.allclose(cal_euclidean(a, b), 2.846049)\n",
    "assert np.allclose(cal_manhattan(a, b), 9)\n",
    "assert np.allclose(cal_cosine(a, b), 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 3** (0.5 points) <br/>\n",
    "Create a function `custom_scaler`, which takes as input an array with integer values and transforms an array, so that:\n",
    "* Maximum element(s) value is 2 (it **doesn't** mean `a[max_index] = 2`)\n",
    "* Minimum element(s) value is -2 (it **doesn't** mean `a[min_index] = -2`)\n",
    "* Other values are in the interval [-2, 2] (using **arithmetic** transformations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def custom_scaler(my_array):\n",
    "    ## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_array = np.random.randint(-100, 100, size=4)\n",
    "result = scaler(test_array)\n",
    "print(test_array, '->>', result)\n",
    "\n",
    "assert np.allclose(np.max(result), 2), f'Maximum value shoul be 2, got {np.max(result)} instead'\n",
    "assert np.allclose(np.min(result), -2), f'Minimum value should be -2, got {np.min(result)} instead'\n",
    "assert np.sum(result < -2) + np.sum(result > 2) == 0, 'All elements are supposed to be between -2 and 2'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 4** (0.5 points) <br/>\n",
    "Create an array with shape of $5 \\times 6$ with integers from [0,50]. Select a column which contains the maximum element of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_array = ## Your code here\n",
    "selected_column = ## Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Array:\\n {my_array}')\n",
    "print(f'Shape: {my_array.shape}')\n",
    "print(f'Selected column {selected_column}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 5** (0.5 points) <br/>\n",
    "\n",
    "Replace all missing values in the following array with mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.random.rand(10)\n",
    "idx = np.random.randint(0, 10, 4)\n",
    "arr[idx] = np.nan\n",
    "\n",
    "print(f'Given array: {np.array2string(arr, precision=2)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your code here\n",
    "\n",
    "print(f'Result: {np.array2string(arr, precision=2)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 6** (0.5 points) <br/>\n",
    "Create a function which takes an image ```X``` (3D array of the shape (n, m, 3)) as an input and returns the mean for each of 3 channels (a vector of shape 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_channel(X):\n",
    "    # Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 28\n",
    "m = 16\n",
    "X =  np.random.randint(-10, 10, size=(n,m,3))\n",
    "mean_channel(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 7** (1 points) <br/>\n",
    "Create a function which takes a matrix ```X``` as an input and returns all its unique rows as a matrix, i.e. in a 3D matrix you need to return all unique horizontal layers.\n",
    "\n",
    "Sample input:\n",
    "\n",
    "     ([[[4, 4],\n",
    "        [5, 5]],\n",
    "\n",
    "       [[4, 5],\n",
    "        [5, 5]],\n",
    "\n",
    "       [[4, 4],\n",
    "        [5, 4]],\n",
    "\n",
    "       [[4, 5],\n",
    "        [5, 5]],\n",
    "\n",
    "       [[5, 4],\n",
    "        [4, 4]],\n",
    "\n",
    "       [[5, 5],\n",
    "        [4, 5]]])\n",
    "        \n",
    "Sample output:\n",
    "\n",
    "     ([[[4, 4],\n",
    "        [5, 5]],\n",
    "\n",
    "       [[4, 5],\n",
    "        [5, 5]],\n",
    "\n",
    "       [[4, 4],\n",
    "        [5, 4]],\n",
    "\n",
    "       [[5, 4],\n",
    "        [4, 4]],\n",
    "\n",
    "       [[5, 5],\n",
    "        [4, 5]]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_unique_rows(X):\n",
    "    # Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X =  np.random.randint(4, 6, size=(n, 2, 2))\n",
    "print(X)\n",
    "get_unique_rows(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Pandas & Visualization (4 points)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are going to work with *Titanic* dataset which contains information about passengers of Titanic:\n",
    "- **Survived** - 1 - survived, 0 - died (0); **Target variable**\n",
    "- **pclass** - passengers's class;\n",
    "- **sex** - passengers's sex\n",
    "- **Age** - passengers's age in years\n",
    "- **sibsp**    - is the passenger someones siblings   \n",
    "- **parch**    - is the passenger someones child or parent\n",
    "- **ticket** - ticket number    \n",
    "- **fare** - ticket price    \n",
    "- **cabin** - cabin number\n",
    "- **embarked** - port of Embarkation; C = Cherbourg, Q = Queenstown, S = Southampton\n",
    "\n",
    "**Note** for all visualizations use matplotlib or seaborn but NOT plotly! Plotly's graphics sometimes vanish after saving. In this case the task wont't be graded.\n",
    "\n",
    "**Note** support all your answers with necessary code, computations, vizualization, and explanation. Answers without code and explanation won't be graded.\n",
    "\n",
    "**Task 0** (0 points) \\\n",
    "Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset =  ## Your code here\n",
    "dataset.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 1** (1 points) <br/>\n",
    "Answer the following questions:\n",
    "    \n",
    "    * Are there any missing values? In which columns?\n",
    "    * How many classes (pclass) were there on Tinanic?\n",
    "    * What is the percentage of survived passengers? Are the classes balanced?\n",
    "    * What is the average ticket fare?\n",
    "    * What was the most popular port of embarkation?\n",
    "Please, write not only the answers but the code, proving it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 2** (1 points) <br/>\n",
    "Find all the titles of the passengers (example, *Capt., Mr.,Mme.*), which are written in the column `Name`, and answer the following questions:\n",
    "\n",
    "    * How many are there unique titles?\n",
    "    * How many are there passengers with every title?\n",
    "    * What is the most popular man's title? And woman's title?\n",
    "    \n",
    "**Hint** You may select the title from the name as a word which contains a dot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 3** (0.5 points) <br/>\n",
    "Visualize age distribution (you may use a histogram, for example). \n",
    "\n",
    "    * What is the minimal and maximum ages of the passengers? Visualize it on the plot\n",
    "    * What is the mean age? Visualize it on the plot\n",
    "    * Make conclusions about what you see on the plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 4** (0.5 points) <br/>\n",
    "Visualize age distribution for survived and not survived passengers separately and calculate the mean age for each class. Are they different? Make conclusions about what you see on the plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 5** (0.5 points) <br/>\n",
    "Is there correlation between the port of embarkation and the ticket price? Calculate mean price for each port and visualize price distribution for each port. Make conclusions about what you see on the plot\n",
    "\n",
    "Hint: you could make one or several plot types i.e.  box, violine, pair, histogram (see additional notebooks for Seminar 1 \"Visualization with Seaborn\"). The main point here is to **choose** plots wisely and **make meaningful conclusions**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Task 6** (0.5 points) <br/>\n",
    "The same question as in task 5 about correlation between *pclass* and *ticket priсe*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## You code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}