-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy path01_tensors
1 lines (1 loc) · 48.5 KB
/
01_tensors
1
{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"EN_01_tensors","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"REmctrquQjG0","colab_type":"text"},"source":["# Deep Learning: PyTorch Basics\n","\n","\n","\n","*Tutorial by* **[Alessandro Torcinovich](https://aretor.github.io)** @ Ca' Foscari University"]},{"cell_type":"markdown","metadata":{"id":"leIuKzCoRZ4R","colab_type":"text"},"source":["---\n","## 1. `tensor`\n","In this tutorial we'll see how to manipulate the `tensor`, the base data structure of this library\n","\n","A `tensor` represents the mathematical concept of a tensor:\n","\n","- 1D tensor ➡️ vector\n","\n"," $\\begin{pmatrix}\n"," 1 & 5 & 7 & 2\n"," \\end{pmatrix}$\n","- 2D tensor ➡️ matrix\n","\n"," $\\begin{pmatrix}\n"," 1 & 5 & 7 & 2 \\\\\n"," 4 & 3 & 2 & 0 \\\\\n"," 3 & 6 & 1 & 8\n"," \\end{pmatrix}$\n","\n","- 3D tensor ...\n","- 4D tensor ...\n","- ..."]},{"cell_type":"code","metadata":{"id":"yxanJ1-jQnV3","colab_type":"code","colab":{}},"source":["import torch"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bUSTHBUVYEMg","colab_type":"text"},"source":["Let's build our first `tensor`"]},{"cell_type":"code","metadata":{"id":"D-7_ulnhYKqH","colab_type":"code","outputId":"86df1f08-9241-4454-a30b-86115414bb21","executionInfo":{"status":"ok","timestamp":1590661436754,"user_tz":-120,"elapsed":963,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["t = torch.tensor([[0,1,2],\n"," [3,4,5],\n"," [6,7,8]])\n","t"],"execution_count":3,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[0, 1, 2],\n"," [3, 4, 5],\n"," [6, 7, 8]])"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"id":"PnIb9CMLYwbd","colab_type":"text"},"source":["A `tensor` is characterized by three important metadata:\n","- `shape` ➡️ the tensor dimensions\n","- `dtype` ➡️ the type of data contained in the (`float`, `int`, etc.)\n","- `device` ➡️ the device where the `tensor` has been allocated (`\"cpu\"` = RAM/CPU, `\"cuda\"` = VRAM/GPU)"]},{"cell_type":"code","metadata":{"id":"dfHmBLmzf-LC","colab_type":"code","outputId":"f9ae78c4-4458-4c9a-a58e-22b4233e8697","executionInfo":{"status":"ok","timestamp":1590661517453,"user_tz":-120,"elapsed":874,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["print(f't shape: {t.shape}')\n","print(f't dtype: {t.dtype}')\n","print(f't device: {t.device}')"],"execution_count":4,"outputs":[{"output_type":"stream","text":["t shape: torch.Size([3, 3])\n","t dtype: torch.int64\n","t device: cpu\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"C2r2XT0thwWQ","colab_type":"text"},"source":["Let's build a `tensor` with `shape` (3, 2, 2)"]},{"cell_type":"code","metadata":{"id":"En1wPwbKhteX","colab_type":"code","outputId":"f199ddf0-bb73-4fc4-9383-d5c56e1ed8df","executionInfo":{"status":"ok","timestamp":1590661586639,"user_tz":-120,"elapsed":860,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":182}},"source":["t = torch.tensor([[[1,2], [3,4]], [[5,6], [7,8]], [[9,10], [11,12]]])\n","print(t)\n","print(t.shape)"],"execution_count":5,"outputs":[{"output_type":"stream","text":["tensor([[[ 1, 2],\n"," [ 3, 4]],\n","\n"," [[ 5, 6],\n"," [ 7, 8]],\n","\n"," [[ 9, 10],\n"," [11, 12]]])\n","torch.Size([3, 2, 2])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"uiz8pOFPz-6K","colab_type":"text"},"source":["A `tensor` with `shape` `(n,)` behaves like one with `shape` `(1, n)`, in general, but it's different from one with `shape` `(n, 1)`. In math they correspond, respectively, to the vectors $u \\in \\mathbb{R}^n$, $v \\in \\mathbb{R}^{1 \\times n}$ and $w \\in \\mathbb{R}^{n \\times 1}$"]},{"cell_type":"code","metadata":{"id":"aGG0Z8Jdz9Fe","colab_type":"code","outputId":"b88707e8-f36c-46a1-ae1f-1396148b329b","executionInfo":{"status":"ok","timestamp":1590661694206,"user_tz":-120,"elapsed":931,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":109}},"source":["u = torch.rand(3)\n","v = torch.rand(1, 3)\n","w = torch.rand(3, 1)\n","print(f'u: {u}, of shape {u.shape}')\n","print(f'v: {v}, of shape {v.shape}')\n","print(f'w: {w}, of shape {w.shape}')"],"execution_count":6,"outputs":[{"output_type":"stream","text":["u: tensor([0.5828, 0.3503, 0.3972]), of shape torch.Size([3])\n","v: tensor([[0.6677, 0.0228, 0.1275]]), of shape torch.Size([1, 3])\n","w: tensor([[0.2907],\n"," [0.6823],\n"," [0.3515]]), of shape torch.Size([3, 1])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"8_MRdQwKyAaI","colab_type":"text"},"source":["---\n","## 2. Creating a `tensor`\n","In the previous example we have used a new function, `torch.rand` that has generated some values without having explicitly defined them.\n","\n","Writing down, by hand, all values of a `tensor` can take a lot of time or even be impossible to do. For this reason, there exist some rather common commodity functions to create `tensor`s\n"]},{"cell_type":"markdown","metadata":{"id":"riJgB55sDrhQ","colab_type":"text"},"source":["- `torch.zeros(shape)`: `tensor` with zeros only"]},{"cell_type":"code","metadata":{"id":"0i_hEsJgDrEY","colab_type":"code","outputId":"a30bc259-db7a-4b2b-db30-d7a88af8edac","executionInfo":{"status":"ok","timestamp":1576271885216,"user_tz":-60,"elapsed":892,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":215}},"source":["z = torch.zeros(3, 3, 3)\n","z"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[[0., 0., 0.],\n"," [0., 0., 0.],\n"," [0., 0., 0.]],\n","\n"," [[0., 0., 0.],\n"," [0., 0., 0.],\n"," [0., 0., 0.]],\n","\n"," [[0., 0., 0.],\n"," [0., 0., 0.],\n"," [0., 0., 0.]]])"]},"metadata":{"tags":[]},"execution_count":26}]},{"cell_type":"markdown","metadata":{"id":"lsuUpV0eDsBf","colab_type":"text"},"source":["- `torch.ones(shape)`: `tensor` with ones only"]},{"cell_type":"code","metadata":{"id":"boumXmk7EXXs","colab_type":"code","outputId":"27947256-b143-4ff4-86a5-ae303bceb100","executionInfo":{"status":"ok","timestamp":1576271898062,"user_tz":-60,"elapsed":1043,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["o = torch.ones(5)\n","o"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([1., 1., 1., 1., 1.])"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"markdown","metadata":{"id":"okoc0HgIElkh","colab_type":"text"},"source":["- `torch.empty(shape)`: `tensor` with uninitialized values"]},{"cell_type":"code","metadata":{"id":"S744mG8GEp9o","colab_type":"code","outputId":"b5306a3e-1bd2-4c01-f8eb-f408ae278ce9","executionInfo":{"status":"ok","timestamp":1576271902392,"user_tz":-60,"elapsed":1113,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["e = torch.empty(3, 4)\n","e"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[1.0096e-35, 0.0000e+00, 4.4842e-44, 0.0000e+00],\n"," [ nan, 0.0000e+00, 1.7202e-04, 6.7510e-07],\n"," [1.7567e-04, 2.7178e+23, 2.1066e-07, 8.1346e+20]])"]},"metadata":{"tags":[]},"execution_count":28}]},{"cell_type":"markdown","metadata":{"id":"RLJ__e9CEqVr","colab_type":"text"},"source":["- `torch.rand(shape)`: `tensor` with random values between 0 (included) and 1 (excluded)"]},{"cell_type":"code","metadata":{"id":"I4rQxijtEqta","colab_type":"code","outputId":"1afbca98-163b-41d7-a469-3c96d4f11e8a","executionInfo":{"status":"ok","timestamp":1576271912217,"user_tz":-60,"elapsed":1331,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":53}},"source":["rnd = torch.rand(2, 3)\n","rnd"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[0.6984, 0.6383, 0.9288],\n"," [0.5574, 0.1240, 0.8312]])"]},"metadata":{"tags":[]},"execution_count":29}]},{"cell_type":"markdown","metadata":{"id":"6UY9waq-Eq8-","colab_type":"text"},"source":["- `torch.arange(low, high)`: `tensor` with an ordered sequenze of integer between `low` (included) and `high` (excluded)"]},{"cell_type":"code","metadata":{"id":"BQPBGTQmEraQ","colab_type":"code","outputId":"f4161aad-3b65-4f40-af4d-b74d14a3b91b","executionInfo":{"status":"ok","timestamp":1590661954621,"user_tz":-120,"elapsed":833,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":54}},"source":["rng = torch.arange(9) # dtype = torch.int\n","print(rng)\n","frng = torch.arange(1., 10.) # dtype = torch.float\n","print(frng)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["tensor([0, 1, 2, 3, 4, 5, 6, 7, 8])\n","tensor([1., 2., 3., 4., 5., 6., 7., 8., 9.])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"oO2yVX9NEsFo","colab_type":"text"},"source":["---\n","## 3. Operations with `tensor`\n","The `shape` of a `tensor` can be modified with the `view(shape)` method"]},{"cell_type":"code","metadata":{"id":"KABGevjwEoTH","colab_type":"code","outputId":"8059f806-ca45-486f-dc26-fd6c46ff5104","executionInfo":{"status":"ok","timestamp":1576272009938,"user_tz":-60,"elapsed":812,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":89}},"source":["rng = rng.view(3, 3)\n","print(rng.shape)\n","print(rng)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([3, 3])\n","tensor([[0, 1, 2],\n"," [3, 4, 5],\n"," [6, 7, 8]])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"67HBq5fKIk7a","colab_type":"text"},"source":["Let's try with some simple operations with `tensor`s\n","- Sum:\n","$\n","\\begin{pmatrix}\n"," 3 & 2 & 4 \\\\ \n"," 5 & 1 & 10 \\\\\n"," 7 & 2 & 9\n","\\end{pmatrix} +\n","\\begin{pmatrix}\n"," 1 & 4 & 7 \\\\\n"," 12 & 5 & 0 \\\\\n"," 3 & 2 & 1 \\\\\n","\\end{pmatrix} =\n","\\begin{pmatrix}\n"," 4 & 6 & 11 \\\\\n"," 17 & 6 & 10 \\\\\n"," 10 & 4 & 10 \\\\\n","\\end{pmatrix}\n","$\n"]},{"cell_type":"code","metadata":{"id":"T53aHBIjXwaR","colab_type":"code","outputId":"b9ec6685-7fc6-4121-ef6e-e2a7d62f578d","executionInfo":{"status":"ok","timestamp":1576273087782,"user_tz":-60,"elapsed":1643,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["x = torch.tensor([[3., 2., 4.], [5., 1., 10.], [7., 2., 9.]])\n","y = torch.tensor([[1., 4., 7.], [12., 5., 0.], [3., 2., 1.]])\n","x + y"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[ 4., 6., 11.],\n"," [17., 6., 10.],\n"," [10., 4., 10.]])"]},"metadata":{"tags":[]},"execution_count":39}]},{"cell_type":"markdown","metadata":{"id":"bJCB-4I5I02j","colab_type":"text"},"source":["- Hadamard product: $\n","\\begin{pmatrix}\n"," 3 & 2 & 4 \\\\ \n"," 5 & 1 & 10 \\\\\n"," 7 & 2 & 9\n","\\end{pmatrix} \\odot\n","\\begin{pmatrix}\n"," 1 & 4 & 7 \\\\\n"," 12 & 5 & 0 \\\\\n"," 3 & 2 & 1 \\\\\n","\\end{pmatrix} =\n","\\begin{pmatrix}\n"," 3 & 8 & 28 \\\\\n"," 60 & 5 & 0 \\\\\n"," 21 & 4 & 9 \\\\\n","\\end{pmatrix}\n","$"]},{"cell_type":"code","metadata":{"id":"0X5_gR0RI1Fy","colab_type":"code","outputId":"667fa31d-dc62-422c-97c6-31bd8434fd54","executionInfo":{"status":"ok","timestamp":1576273091348,"user_tz":-60,"elapsed":1638,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["x * y"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[ 3., 8., 28.],\n"," [60., 5., 0.],\n"," [21., 4., 9.]])"]},"metadata":{"tags":[]},"execution_count":40}]},{"cell_type":"markdown","metadata":{"id":"BPIi3PTDI1fu","colab_type":"text"},"source":["- Matrix product: $\n","\\begin{pmatrix}\n"," 3 & 2 & 4 \\\\ \n"," 5 & 1 & 10 \\\\\n"," 7 & 2 & 9\n","\\end{pmatrix} \\times\n","\\begin{pmatrix}\n"," 1 & 4 & 7 \\\\\n"," 12 & 5 & 0 \\\\\n"," 3 & 2 & 1 \\\\\n","\\end{pmatrix} =\n","\\begin{pmatrix}\n"," 39 & 30 & 25 \\\\\n"," 47 & 45 & 45 \\\\\n"," 58 & 56 & 58 \\\\\n","\\end{pmatrix}\n","$"]},{"cell_type":"code","metadata":{"id":"mUCIMEudI123","colab_type":"code","outputId":"617aeeca-97b3-45e6-8402-32eb826eee98","executionInfo":{"status":"ok","timestamp":1576273094947,"user_tz":-60,"elapsed":1409,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["x @ y # o torch.matmul(x, y) o torch.mm(x, y)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[39., 30., 25.],\n"," [47., 45., 45.],\n"," [58., 56., 58.]])"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"markdown","metadata":{"id":"ssmI2VGPI2Zk","colab_type":"text"},"source":["- exp, sin, cos, log, etc."]},{"cell_type":"code","metadata":{"id":"dk6GD2_3I3F1","colab_type":"code","outputId":"2df7904e-891d-4cdc-fa87-6aae3139464b","executionInfo":{"status":"ok","timestamp":1576273097206,"user_tz":-60,"elapsed":1607,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":125}},"source":["print(torch.exp(x))\n","print(torch.sin(y))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[2.0086e+01, 7.3891e+00, 5.4598e+01],\n"," [1.4841e+02, 2.7183e+00, 2.2026e+04],\n"," [1.0966e+03, 7.3891e+00, 8.1031e+03]])\n","tensor([[ 0.8415, -0.7568, 0.6570],\n"," [-0.5366, -0.9589, 0.0000],\n"," [ 0.1411, 0.9093, 0.8415]])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"LcedbJWUREi2","colab_type":"text"},"source":["---\n","## 4. `tensor` Indexing\n","Sometimes it's necessary to access to a subset of the values in a `tensor`. This operation is called *indexing*. In the following, some examples (rows and columns start from 0):\n","\n","$ m = \n","\\begin{pmatrix}\n"," 0 & 1 & 2 \\\\\n"," 3 & 4 & 5 \\\\\n"," 6 & 7 & 8 \\\\\n"," 9 & 10 & 11 \\\\\n"," 12 & 13 & 14\n","\\end{pmatrix}$\n"]},{"cell_type":"code","metadata":{"id":"OERNFE5WMbzF","colab_type":"code","outputId":"adebc0bd-94d7-425c-e16f-6a92fcb4cb79","executionInfo":{"status":"ok","timestamp":1590662219638,"user_tz":-120,"elapsed":957,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":109}},"source":["m = torch.arange(15.).view(5, 3)\n","m"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[ 0., 1., 2.],\n"," [ 3., 4., 5.],\n"," [ 6., 7., 8.],\n"," [ 9., 10., 11.],\n"," [12., 13., 14.]])"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"btRMvSX2UNbH","colab_type":"text"},"source":["- $m_{12} = \\begin{pmatrix} 5 \\end{pmatrix}$"]},{"cell_type":"code","metadata":{"id":"xv2ORif_UQft","colab_type":"code","outputId":"48d77df2-b37e-446f-95c5-0b8c7ee9e1b1","executionInfo":{"status":"ok","timestamp":1576275229444,"user_tz":-60,"elapsed":990,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["m[1, 2]"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor(5.)"]},"metadata":{"tags":[]},"execution_count":44}]},{"cell_type":"markdown","metadata":{"id":"YkMgJ-3VURB8","colab_type":"text"},"source":["- $m_{1*} = \\begin{pmatrix} 3 & 4 & 5 \\end{pmatrix}$"]},{"cell_type":"code","metadata":{"id":"12FLgLoiURRZ","colab_type":"code","outputId":"5fa536fc-b2a8-446f-e3ab-23e91c2448fa","executionInfo":{"status":"ok","timestamp":1576275754036,"user_tz":-60,"elapsed":1024,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["m[1, :]"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([3., 4., 5.])"]},"metadata":{"tags":[]},"execution_count":45}]},{"cell_type":"markdown","metadata":{"id":"c6UMTs-sURfJ","colab_type":"text"},"source":["$m_{1:5, *} = \\begin{pmatrix} \n"," 3 & 4 & 5 \\\\\n"," 6 & 7 & 8 \\\\\n"," 9 & 10 & 11 \\\\\n"," 12 & 13 & 14\n"," \\end{pmatrix}$\n"," \n"," N.B. If the interval goes over the numer of rows/columns of the matrix, PyTorch returns nonetheless the max number of possible rows/columns. For example, if the row indices of $m$ go from $0$ to $1$, by using `1:5` I'm selecting the rows from $1$ to $4$ "]},{"cell_type":"code","metadata":{"id":"22j6ZdkUURuM","colab_type":"code","outputId":"06d9d959-1601-4454-8306-ceb60cea7e3e","executionInfo":{"status":"ok","timestamp":1576275848759,"user_tz":-60,"elapsed":1141,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":89}},"source":["m[1:5, :]"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[ 3., 4., 5.],\n"," [ 6., 7., 8.],\n"," [ 9., 10., 11.],\n"," [12., 13., 14.]])"]},"metadata":{"tags":[]},"execution_count":46}]},{"cell_type":"markdown","metadata":{"id":"ZaMIz9wMUR_O","colab_type":"text"},"source":["$m_{1:5, 2:} = \n","\\begin{pmatrix} \n"," 5 \\\\\n"," 8 \\\\ \n"," 11 \\\\\n"," 14\n","\\end{pmatrix}$"]},{"cell_type":"code","metadata":{"id":"HtyDRCuXUSVW","colab_type":"code","outputId":"74b75939-5127-4b96-89db-cf7d5d4ae958","executionInfo":{"status":"ok","timestamp":1590662419008,"user_tz":-120,"elapsed":883,"user":{"displayName":"Alessandro TORCINOVICH","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi96E7HK6ff0uWSudqbJvEMactRTPtsT5-4448y=s64","userId":"03260776184459049943"}},"colab":{"base_uri":"https://localhost:8080/","height":90}},"source":["m[1:5, 1:]"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[ 4., 5.],\n"," [ 7., 8.],\n"," [10., 11.],\n"," [13., 14.]])"]},"metadata":{"tags":[]},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"sjSt_ASLUSj4","colab_type":"text"},"source":["### Fancy Indexing\n","\n","$\\begin{pmatrix}\n"," m_{10} & m_{12} & m_{31}\n","\\end{pmatrix} =\n","\\begin{pmatrix}\n"," 3 & 5 & 10\n","\\end{pmatrix}$\n","\n","N.B. In this case the returned `tensor` will simply be with `shape` `(n,)`"]},{"cell_type":"code","metadata":{"id":"EQNgWEqaUSwd","colab_type":"code","outputId":"251d76da-e364-4e7d-e1c7-9a828c40aa78","executionInfo":{"status":"ok","timestamp":1576276470715,"user_tz":-60,"elapsed":1653,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["m[[1, 1, 3], [0, 2, 1]] "],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([ 3., 5., 10.])"]},"metadata":{"tags":[]},"execution_count":48}]},{"cell_type":"markdown","metadata":{"id":"_iM7bhzRUS_p","colab_type":"text"},"source":["With the boolean masks, I can select the elements corresponding to the value `True`\n","\n","$\\begin{pmatrix}\n"," m_{00} & m_{12} & m_{20} & m_{21} & m_{30} & m_{31} & m_{32}\n","\\end{pmatrix} = \n","\\begin{pmatrix}\n"," 0 & 5 & 6 & 7 & 9 & 10 & 11\n","\\end{pmatrix}$\n","\n","N.B. With PyTorch `<= 1.2.0` the boolean type does not exists and `torch.byte` is used instead\n","\n","N.B. In this case too, the returned `tensor` will always be with `shape` `(n,)`"]},{"cell_type":"code","metadata":{"id":"koFhOn4tUTM8","colab_type":"code","outputId":"d22b0468-377b-429a-dc49-8a9666b9679b","executionInfo":{"status":"ok","timestamp":1576277245354,"user_tz":-60,"elapsed":967,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["mask = torch.tensor([[True, False, False],\n"," [False, False, True],\n"," [True, True, False],\n"," [True, True, True],\n"," [False, False, False]])\n","m[mask]"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([ 0., 5., 6., 7., 9., 10., 11.])"]},"metadata":{"tags":[]},"execution_count":51}]},{"cell_type":"markdown","metadata":{"id":"t_tnC9YiRqF4","colab_type":"text"},"source":["In general, PyTorch follows the [NumPy indexing rules](https://docs.scipy.org/doc/numpy/user/basics.indexing.html)"]},{"cell_type":"markdown","metadata":{"id":"TOd0GvyEclaL","colab_type":"text"},"source":["---\n","# 5. `device`\n","Differently from NumPy `ndarray`s, in PyTorch, the `tensor` can also be allocated in the VRAM of the GPU to speed up the execution. To utilize this feature you must have an Nvidia GPU that is [CUDA enabled](https://developer.nvidia.com/cuda-gpus) and you must, moreover, install the specific version of the `libcuda` and `libcudnn` libraries required by the PyTorch version you are using.\n","\n","Let's see the different execution times of the CPU and the GPU"]},{"cell_type":"code","metadata":{"id":"fUekEJaRfXTV","colab_type":"code","outputId":"11d9584f-1b99-4a52-b8f8-fd286f93554c","executionInfo":{"status":"ok","timestamp":1576578828908,"user_tz":-60,"elapsed":733,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["import time\n","r1 = torch.rand(1000, 1000) # 100 * 100 = 10000 random numbers in [0, 1)\n","r2 = torch.rand(1000, 4000)\n","print(f'r1 {r1.device}')\n","print(f'r2 {r2.device}')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["r1 cpu\n","r2 cpu\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"HqD9OYXAcvka","colab_type":"code","outputId":"9ee21481-6baf-4907-89dc-78416daec68c","executionInfo":{"status":"ok","timestamp":1576578850729,"user_tz":-60,"elapsed":690,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["start = time.time()\n","r1 @ r2\n","print(f'r1 @ r2 total time (CPU): {time.time() - start} sec')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["r1 @ r2 total time (CPU): 0.12951326370239258 sec\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"VFnBcQR2fveH","colab_type":"text"},"source":["Now let's see the difference in the GPU. First of all, let's move the tensor in the correct `device`\n","\n","N.B. When perfoming the first computation in GPU, PyTorch has a computational overhead due to the initialization of CUDA runtime API"]},{"cell_type":"code","metadata":{"id":"Sfn79HNffusK","colab_type":"code","outputId":"faf5cce4-08bc-4b85-d709-b964c2a7c942","executionInfo":{"status":"ok","timestamp":1576578910141,"user_tz":-60,"elapsed":978,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["r1 = r1.to('cuda') # oppure cuda:n per indicare il device da usare (default 0)\n","r2 = r2.to('cuda')\n","print(f'r1 {r1.device}')\n","print(f'r2 {r2.device}')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["r1 cuda:0\n","r2 cuda:0\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"jbHhMJB0hSCG","colab_type":"code","outputId":"ccdc3e9e-1353-4dc8-d67c-510f09aa5d48","executionInfo":{"status":"ok","timestamp":1576578977190,"user_tz":-60,"elapsed":731,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["start = time.time()\n","r1 @ r2\n","print(f'r1 @ r2 total time (GPU): {time.time() - start} sec')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["r1 @ r2 total time (GPU): 0.005323648452758789 sec\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"TIukhcLhvnsF","colab_type":"text"},"source":["We can also allocate the `tensor` directly in the GPU, by specifying the `device` parameter during the initialization"]},{"cell_type":"code","metadata":{"id":"-xdiFg5kt3r3","colab_type":"code","outputId":"171974d9-5c35-4068-d036-514e21345b5d","executionInfo":{"status":"ok","timestamp":1576316142312,"user_tz":-60,"elapsed":994,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":35}},"source":["r3 = torch.rand(1000, 1000, device='cuda')\n","print(f'r3 {r3.device}')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["r3 cuda:0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kc9WGv_6w58H","colab_type":"text"},"source":["## 6. Gradient\n","PyTorch is a Deep Learning library, and it's therefore important that it fast compute the derivatives of the operations to perform the backpropagation, without manually write the derivatives.\n","To this end, PyTorch uses the differentiation engine `autograd`, that computes the derivatives using the [Automatic Differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)\n","\n","Let's see how to compute the derivative of $y = x_1^2$ in $x_1 = 3$, that is:\n","\n","$$\\left.\\frac{\\partial y}{\\partial x_1}\\right|_{x_1 = 3} = 2 \\cdot x_1|_{x_1 = 3} = 2 \\cdot 3 = 6$$"]},{"cell_type":"code","metadata":{"id":"ZJpRnbJ_wqHD","colab_type":"code","outputId":"e25e5ea5-9bd9-4613-dead-96f7cb2d357b","executionInfo":{"status":"ok","timestamp":1576579755766,"user_tz":-60,"elapsed":1184,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":86}},"source":["x1 = torch.tensor(3., requires_grad=True)\n","print(x1)\n","print(f'grad before backward: {x1.grad}')\n","y = x1 ** 2.\n","print(y)\n","y.backward()\n","print(f'grad after backward: {x1.grad}')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor(3., requires_grad=True)\n","grad before backward: None\n","tensor(9., grad_fn=<PowBackward0>)\n","grad after backward: 6.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CtVs4OGu9hhw","colab_type":"text"},"source":["From the previous example we can understand some things:\n","1. A `tensor` takes care to store both the outpu and its derivative\n","2. At the moment of its creation I state the parameter `requires_grad=True`\n","3. the computation of the derivatives is not automatic and must be started by the user by calling the `backward()` method\n","\n","To check if a `tensor` requires the gradient, I can check the `tensor.requires_grad` attribute.\n","To modify this parameter, I can call the `tensor.requires_grad_()` function (with the ending underscore) that set to `True` the `requires_grad` parameter (but doesn't do the inverse)\n","\n","N.B. Be careful to not confuse them!!"]},{"cell_type":"code","metadata":{"id":"fecJZkam3HsX","colab_type":"code","outputId":"64e9bf88-44fd-46c5-f3fd-78fb907420d7","executionInfo":{"status":"ok","timestamp":1576320593897,"user_tz":-60,"elapsed":967,"user":{"displayName":"aretor","photoUrl":"","userId":"05696733233927156664"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["x2 = torch.tensor(4.) # requires_grad è a False\n","print(f'1. x2 requires the gradient? {x2.requires_grad}')\n","x2.requires_grad_() # setta a True\n","print(f'2. x2 requires the gradient? {x2.requires_grad}')\n","x2.requires_grad_() # non cambia nulla\n","print(f'3. x2 requires the gradient? {x2.requires_grad}')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["1. x2 requires the gradient? False\n","2. x2 requires the gradient? True\n","3. x2 requires the gradient? True\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6E9VWCeuAiv2","colab_type":"text"},"source":["There are other ways to compute the gradient computation, for example by manually setting the `requires_grad`, or by using the context manager `torch.no_grad()`"]},{"cell_type":"code","metadata":{"id":"Hm1rM4bF_Ia9","colab_type":"code","outputId":"0a79d238-ae07-4e5c-9643-05bbd745111e","executionInfo":{"status":"error","timestamp":1576579963407,"user_tz":-60,"elapsed":953,"user":{"displayName":"Mikolaj Kopernik","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mCdyrsrJJna9WiIjRrncAz2FeFdudoHh9xwNtkJ=s64","userId":"13542018321342352491"}},"colab":{"base_uri":"https://localhost:8080/","height":380}},"source":["x3 = torch.tensor(5., requires_grad=True)\n","with torch.no_grad():\n"," print(f'x3 requires grad? {x3.requires_grad}')\n"," y = 4. * x3\n"," y.backward() # Errore!"],"execution_count":0,"outputs":[{"output_type":"stream","text":["x3 requires grad? True\n"],"name":"stdout"},{"output_type":"error","ename":"RuntimeError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)","\u001b[0;32m<ipython-input-13-e5a5eaf38016>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'x3 requires grad? {x3.requires_grad}'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m4.\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mx3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# Errore!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m 164\u001b[0m \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 165\u001b[0m \"\"\"\n\u001b[0;32m--> 166\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 168\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m 97\u001b[0m Variable._execution_engine.run_backward(\n\u001b[1;32m 98\u001b[0m \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m allow_unreachable=True) # allow_unreachable flag\n\u001b[0m\u001b[1;32m 100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mRuntimeError\u001b[0m: element 0 of tensors does not require grad and does not have a grad_fn"]}]},{"cell_type":"markdown","metadata":{"id":"v697Fvw-FWtU","colab_type":"text"},"source":["<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n","<!-- Created with Inkscape (http://www.inkscape.org/) -->\n","\n","<svg\n"," xmlns:dc=\"http://purl.org/dc/elements/1.1/\"\n"," xmlns:cc=\"http://creativecommons.org/ns#\"\n"," xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"\n"," xmlns:svg=\"http://www.w3.org/2000/svg\"\n"," xmlns=\"http://www.w3.org/2000/svg\"\n"," xmlns:sodipodi=\"http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd\"\n"," xmlns:inkscape=\"http://www.inkscape.org/namespaces/inkscape\"\n"," width=\"210mm\"\n"," height=\"297mm\"\n"," viewBox=\"0 0 210 297\"\n"," version=\"1.1\"\n"," id=\"svg8\"\n"," inkscape:version=\"0.92.3 (2405546, 2018-03-11)\"\n"," sodipodi:docname=\"tensor.svg\">\n"," <defs\n"," id=\"defs2\" />\n"," <sodipodi:namedview\n"," id=\"base\"\n"," pagecolor=\"#ffffff\"\n"," bordercolor=\"#666666\"\n"," borderopacity=\"1.0\"\n"," inkscape:pageopacity=\"0.0\"\n"," inkscape:pageshadow=\"2\"\n"," inkscape:zoom=\"1\"\n"," inkscape:cx=\"349.18916\"\n"," inkscape:cy=\"776.97161\"\n"," inkscape:document-units=\"mm\"\n"," inkscape:current-layer=\"layer1\"\n"," showgrid=\"true\"\n"," showguides=\"false\"\n"," inkscape:window-width=\"1853\"\n"," inkscape:window-height=\"1025\"\n"," inkscape:window-x=\"67\"\n"," inkscape:window-y=\"27\"\n"," inkscape:window-maximized=\"1\">\n"," <inkscape:grid\n"," type=\"xygrid\"\n"," id=\"grid815\" />\n"," </sodipodi:namedview>\n"," <metadata\n"," id=\"metadata5\">\n"," <rdf:RDF>\n"," <cc:Work\n"," rdf:about=\"\">\n"," <dc:format>image/svg+xml</dc:format>\n"," <dc:type\n"," rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\" />\n"," <dc:title></dc:title>\n"," </cc:Work>\n"," </rdf:RDF>\n"," </metadata>\n"," <g\n"," inkscape:label=\"Layer 1\"\n"," inkscape:groupmode=\"layer\"\n"," id=\"layer1\">\n"," <g\n"," id=\"g1035\">\n"," <g\n"," transform=\"translate(15.875,-15.875)\"\n"," id=\"g972\">\n"," <circle\n"," style=\"opacity:0.97000002;fill:#d2d2d2;fill-opacity:1;stroke:#808080;stroke-width:0.40000001;stroke-linecap:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1\"\n"," id=\"path817\"\n"," cx=\"66.145844\"\n"," cy=\"172.64586\"\n"," r=\"13.229166\" />\n"," <text\n"," xml:space=\"preserve\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," x=\"57.446629\"\n"," y=\"173.96255\"\n"," id=\"text821\"><tspan\n"," sodipodi:role=\"line\"\n"," id=\"tspan819\"\n"," x=\"57.446629\"\n"," y=\"173.96255\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:6.3499999px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start\">x<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan958\">1</tspan> = 3</tspan></text>\n"," </g>\n"," <g\n"," transform=\"translate(-15.875,-15.875)\"\n"," id=\"g978\">\n"," <circle\n"," r=\"13.229166\"\n"," cy=\"172.64586\"\n"," cx=\"171.97919\"\n"," id=\"circle829\"\n"," style=\"opacity:0.97000002;fill:#d2d2d2;fill-opacity:1;stroke:#808080;stroke-width:0.40000001;stroke-linecap:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1\" />\n"," <text\n"," id=\"text835\"\n"," y=\"173.96255\"\n"," x=\"163.25775\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," xml:space=\"preserve\"><tspan\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:6.3499999px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start\"\n"," y=\"173.96255\"\n"," x=\"163.25775\"\n"," id=\"tspan833\"\n"," sodipodi:role=\"line\">x<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan962\">2</tspan> = 2</tspan></text>\n"," </g>\n"," <g\n"," id=\"g986\">\n"," <ellipse\n"," style=\"opacity:0.97000002;fill:#d2d2d2;fill-opacity:1;stroke:#808080;stroke-width:0.48934704;stroke-linecap:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1\"\n"," id=\"circle837\"\n"," cx=\"119.06248\"\n"," cy=\"119.72916\"\n"," rx=\"19.86623\"\n"," ry=\"13.184492\" />\n"," <text\n"," xml:space=\"preserve\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," x=\"101.85271\"\n"," y=\"121.04585\"\n"," id=\"text843\"><tspan\n"," sodipodi:role=\"line\"\n"," id=\"tspan841\"\n"," x=\"101.85271\"\n"," y=\"121.04585\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:6.3499999px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start\">h<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan948\">1</tspan> = x<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan952\">1</tspan> + x<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan954\">2</tspan></tspan></text>\n"," </g>\n"," <g\n"," transform=\"translate(0,15.875)\"\n"," id=\"g993\">\n"," <ellipse\n"," ry=\"13.184492\"\n"," rx=\"19.86623\"\n"," cy=\"66.812508\"\n"," cx=\"119.06248\"\n"," id=\"ellipse966\"\n"," style=\"opacity:0.97000002;fill:#d2d2d2;fill-opacity:1;stroke:#808080;stroke-width:0.48934704;stroke-linecap:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1\" />\n"," <text\n"," id=\"text851\"\n"," y=\"68.129204\"\n"," x=\"106.95779\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," xml:space=\"preserve\"><tspan\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:6.3499999px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start\"\n"," y=\"68.129204\"\n"," x=\"106.95779\"\n"," id=\"tspan849\"\n"," sodipodi:role=\"line\">h<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan942\">2</tspan> = 5h<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan944\">1</tspan></tspan></text>\n"," </g>\n"," <g\n"," id=\"g1045\">\n"," <ellipse\n"," ry=\"13.184492\"\n"," rx=\"19.86623\"\n"," cy=\"45.645836\"\n"," cx=\"119.06248\"\n"," id=\"ellipse964\"\n"," style=\"opacity:0.97000002;fill:#d2d2d2;fill-opacity:1;stroke:#808080;stroke-width:0.48934704;stroke-linecap:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1\" />\n"," <text\n"," xml:space=\"preserve\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," x=\"105.34373\"\n"," y=\"46.962532\"\n"," id=\"text859\"><tspan\n"," sodipodi:role=\"line\"\n"," id=\"tspan857\"\n"," x=\"105.34373\"\n"," y=\"46.962532\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:6.3499999px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start\">y =ln(h<tspan\n"," style=\"font-size:64.99999762%;baseline-shift:sub\"\n"," id=\"tspan938\">2</tspan>)</tspan></text>\n"," </g>\n"," <path\n"," sodipodi:nodetypes=\"cc\"\n"," inkscape:connector-curvature=\"0\"\n"," id=\"path884\"\n"," d=\"M 89.958333,146.1875 103.1875,127.66667\"\n"," style=\"fill:none;fill-rule:evenodd;stroke:#808080;stroke-width:0.565;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1\" />\n"," <path\n"," sodipodi:nodetypes=\"cc\"\n"," inkscape:connector-curvature=\"0\"\n"," id=\"path886\"\n"," d=\"m 134.9375,127.66667 13.22917,18.52083\"\n"," style=\"fill:none;fill-rule:evenodd;stroke:#808080;stroke-width:0.565;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1\" />\n"," <path\n"," sodipodi:nodetypes=\"cc\"\n"," inkscape:connector-curvature=\"0\"\n"," id=\"path888\"\n"," d=\"m 119.0625,106.5 0,-10.583333\"\n"," style=\"fill:none;fill-rule:evenodd;stroke:#808080;stroke-width:0.66500002;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1\" />\n"," <path\n"," sodipodi:nodetypes=\"cc\"\n"," inkscape:connector-curvature=\"0\"\n"," id=\"path890\"\n"," d=\"m 119.0625,69.458335 0,-10.583335\"\n"," style=\"fill:none;fill-rule:evenodd;stroke:#808080;stroke-width:0.565;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1\" />\n"," </g>\n"," <text\n"," xml:space=\"preserve\"\n"," style=\"font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," x=\"52.916664\"\n"," y=\"140.89584\"\n"," id=\"text894\"><tspan\n"," sodipodi:role=\"line\"\n"," id=\"tspan892\"\n"," x=\"52.916664\"\n"," y=\"150.25961\"\n"," style=\"stroke-width:0.26458332px\" /></text>\n"," <text\n"," id=\"text900\"\n"," y=\"136.92085\"\n"," x=\"77.721703\"\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:9.90000248px;line-height:125%;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;baseline-shift:baseline;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"\n"," xml:space=\"preserve\"><tspan\n"," style=\"font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:9.90000248px;font-family:'TeX Gyre Schola Math';-inkscape-font-specification:'TeX Gyre Schola Math, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start\"\n"," y=\"0\"\n"," x=\"0\"\n"," id=\"tspan898\"\n"," sodipodi:role=\"line\"><tspan\n"," id=\"tspan896\"\n"," style=\"font-size:64.99999762%;baseline-shift:sub\" /></tspan></text>\n"," </g>\n","</svg>\n"]},{"cell_type":"markdown","metadata":{"id":"QeU39sGWE3XV","colab_type":"text"},"source":["In general, managing the gradient of `tensor` can be complicated. There exist many debugging functions that make the gradient checking simple!"]}]}