AI_CV_Dic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cc2cf4e-f94a-403b-bb35-b8a976515cf1",
   "metadata": {},
   "source": [
    "<a id='0.0'></a>\n",
    "* [1. Many ways of performance test of model](#1)\n",
    "    * [1.1 Classification](#1.1)\\\n",
    "            * [1.1.1 Receiver Operating Characterestic (ROC) Curve (Supervised, Labelled)](#1.1.1)\\\n",
    "            * [1.1.2 Confusion Matrix (Supervised, Labelled)](#1.1.2)\\\n",
    "            * [1.1.3 Precision-Recall Curve (Supervised, Labelled)](#1.1.3)\\\n",
    "            * [1.1.4 F1 Score (Supervised, Labelled)](#1.1.4)\\\n",
    "            * [1.1.5 Silhouette Score (Supervised, Labelled)](#1.1.5)\\\n",
    "            * [1.1.6 Cross-Entropy Loss (Supervised, Labelled)](#1.1.6)\n",
    "    * [1.2 Regression](#1.2)\\\n",
    "            * [1.2.1 R-Square](#1.2.1)\\\n",
    "            * [1.2.2 Root Mean Square Error (RMSE)](#1.2.2)\n",
    "* [2. Mahcine Learning Methods](#2)\n",
    "    * [2.1 Classification](#2.1)\\\n",
    "            * [2.1.1 K-Neighbours](#2.1.1)\\\n",
    "            * [2.1.2 Logistic Regression](#2.1.2)\\\n",
    "            * [2.1.3 Supportive Vector Machine (SVM)](#2.1.3)\\\n",
    "            * [2.1.4 Random Forest](#2.1.4)\n",
    "    * [2.2 Regression](#2.2)\\\n",
    "            * [2.2.1 Multiple Linear Regression, Ridge, Lasso Regression and Polynomial Regression](#2.2.1)\\\n",
    "            * [2.2.2 Ridge and Lasso Regularisation](#2.2.2)\\\n",
    "            * [2.2.3 LAD (Least Absolute Deviations) Regression](#2.2.3)\\\n",
    "            * [2.2.4 Bayesian Ridge and Lasso Regression](#2.2.4)\\\n",
    "            * [2.2.5 Neural Networks Regression](#2.2.5)\\\n",
    "            * [2.2.6 Supportive Vector Regression](#2.2.6)\n",
    "* [3. Neural Networks Methods](#3)\n",
    "    * [3.1 Autoencoder and Variate Autoencoder (VAE)](#3.1)\n",
    "    * [3.2 U-Net](#3.2)\n",
    "    * [3.3 Generative Adversarial Networks (GAN) and Deep Convolutional GAN](#3.3)\n",
    "    * [3.4 Self-Normalising Neural Networks (SELU](#3.4)\n",
    "    * [3.5 (Self) Compatitivly Generative Network (CGN)-Ensemble model of GAN](#3.5)\n",
    "    * [3.6 Recurrent Neural Networks (RNN)](#3.6)\n",
    "* [4. Computer Vision Methods](#4)\n",
    "    * [4.1 Object Detection](#4.1)\n",
    "    * [4.2 Object Segmentation](#4.2)\n",
    "    * [4.3 Style Transfer](#4.3)\n",
    "    * [4.4 Super-Resolution with GAN, Autoencoder](#4.4)\n",
    "    * [4.5 3D Reconstruction Methods](#4.5)\\\n",
    "            * [4.5.1 Stereopsis](#4.5.1)\\\n",
    "            * [4.5.2 Multiview Stereo](#4.5.1)\\\n",
    "            * [4.5.3 Structure from Motion](#4.5.1)\n",
    "* [5. Tensorflow 2](#5)\n",
    "    * [5.1 Layers (Dense, Flatten, Convolution, Dropout](#5.1)\\\n",
    "            * [5.1.1 Dense Layer](#5.1.1)\\\n",
    "            * [5.1.2 Flatten Layer](#5.1.2)\\\n",
    "            * [5.1.3 Convolution Layer](#5.1.3)\\\n",
    "            * [5.1.4 Dropout Layer](#5.1.4)\\\n",
    "            * [5.1.5 Recurrent Layer](#5.1.5)\\\n",
    "            * [5.1.6 Embedding Layer](#5.1.6)\n",
    "    * [5.2 Activations(ReLU, Sigmoid, LeakyReLU, Hyperbolic Tangent](#5.2)\n",
    "    * [5.3 Batch and Batch Normalisation](#5.3)\n",
    "    * [5.4 Functional API in Tensorflow 2](#5.4)\n",
    "    * [5.5 Gradient Tape in Tensorflow 2](#5.5)\n",
    "* [7. Reinforcement Learning](#7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e6ccbb-7878-4e6d-a2e9-b2fde0fad4cb",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "# 1. Many ways of performance test of model\n",
    "<a id='1.1'></a>\n",
    "## 1.1 Classification\n",
    "<a id='1.1.1'></a>\n",
    "### 1.1.1 Receiver Operating Characteristic (ROC) Curve (Supervised, labelled)\n",
    "\n",
    "ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.\n",
    "\n",
    "true positive (TP): \n",
    "A test result that correctly indicates the presence of a condition or characteristic\\\n",
    "true negative (TN): \n",
    "A test result that correctly indicates the absence of a condition or characteristic\\\n",
    "false positive (FP): \n",
    "A test result which wrongly indicates that a particular condition or attribute is present\\\n",
    "false negative (FN): \n",
    "A test result which wrongly indicates that a particular condition or attribute is absent\\\n",
    "${\\displaystyle \\mathrm {TPR} ={\\frac {\\mathrm {TP} }{\\mathrm {P} }}={\\frac {\\mathrm {TP} }{\\mathrm {TP} +\\mathrm {FN} }}=1-\\mathrm {FNR} }$\\\n",
    "${\\displaystyle \\mathrm {FPR} ={\\frac {\\mathrm {FP} }{\\mathrm {N} }}={\\frac {\\mathrm {FP} }{\\mathrm {FP} +\\mathrm {TN} }}=1-\\mathrm {TNR} }$\n",
    "<a id='1.1.2'></a>\n",
    "### 1.1.2 Confusion Matrix (Supervised, labelled)\n",
    "\n",
    "Confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix). It provides a summary of correct and incorrect predictions made by the algorithm. The matrix has four components: True Positives, False Positives, True Negatives, and False Negatives.\n",
    "\n",
    "<img src=\"img/ConfusionMatrix.webp\" alt=\"drawing\" width=\"400\"/>\n",
    "<a id='1.1.3'></a>\n",
    "### 1.1.3 Precision-Recall Curve (Supervised, labelled)\n",
    "\n",
    "Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances, while recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Both precision and recall are therefore based on relevance. The Precision-Recall curve is similar to the ROC curve but instead of the False Positive Rate, it plots the Precision (True Positives / (True Positives + False Positives)) against the Recall (True Positives / (True Positives + False Negatives)).\n",
    "<a id='1.1.4'></a>\n",
    "### 1.1.4 F1 Score (Supervised, labelled)\n",
    "\n",
    "the F-score or F-measure is a measure of a test's accuracy. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the number of all positive results, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified as positive. The F1 score is the harmonic mean of the precision and recall. The more generic ${\\displaystyle F_{\\beta }}$ score applies additional weights, valuing one of precision or recall more than the other.\n",
    "\n",
    "${\\displaystyle F_{1}={\\frac {2}{\\mathrm {recall} ^{-1}+\\mathrm {precision} ^{-1}}}=2{\\frac {\\mathrm {precision} \\cdot \\mathrm {recall} }{\\mathrm {precision} +\\mathrm {recall} }}={\\frac {2\\mathrm {tp} }{2\\mathrm {tp} +\\mathrm {fp} +\\mathrm {fn} }}}$\\\n",
    "$F_\\beta = \\frac {(1 + \\beta^2) \\cdot \\mathrm{true\\ positive} }{(1 + \\beta^2) \\cdot \\mathrm{true\\ positive} + \\beta^2 \\cdot \\mathrm{false\\ negative} + \\mathrm{false\\ positive}}\\,$\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab5813e-9ad1-4459-944e-0e7589db6781",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix, roc_auc_score, roc_curve, precision_recall_curve, f1_score\n",
    "np.random.seed(0) # Generate some fake data\n",
    "y_true = np.random.randint(2, size=100)\n",
    "y_probs = np.random.rand(100)\n",
    "cm = confusion_matrix(y_true, (y_probs > 0.5)) # Calculate the confusion matrix\n",
    "roc_auc = roc_auc_score(y_true, y_probs) # Calculate the ROC AUC score\n",
    "fpr, tpr, thresholds = roc_curve(y_true, y_probs) # Plot the ROC curve\n",
    "precision, recall, thresholds = precision_recall_curve(y_true, y_probs) # Calculate the Precision-Recall curve\n",
    "tick_marks = np.arange(2)\n",
    "# Calculate the F1 score\n",
    "f1 = f1_score(y_true, (y_probs > 0.5))\n",
    "print(\"F1 score:\", f1)\n",
    "fig, ax = plt.subplots(2,2, figsize=(10,10))\n",
    "a1=ax[0,0].imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)\n",
    "ax[0,0].set_title(\"Confusion Matrix\")\n",
    "fig.colorbar(a1)\n",
    "ax[0,0].set_xticks(tick_marks, [0, 1])\n",
    "ax[0,0].set_yticks(tick_marks, [0, 1])\n",
    "ax[0,0].set_xlabel(\"Predicted\")\n",
    "ax[0,0].set_ylabel(\"True\")\n",
    "ax[1,0].plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "ax[1,0].plot([0, 1], [0, 1], 'k--')\n",
    "ax[1,0].set_xlim([0.0, 1.0])\n",
    "ax[1,0].set_ylim([0.0, 1.05])\n",
    "ax[1,0].set_xlabel('False Positive Rate')\n",
    "ax[1,0].set_ylabel('True Positive Rate')\n",
    "ax[1,0].set_title('Receiver Operating Characteristic (ROC)')\n",
    "ax[1,0].legend(loc=\"lower right\")\n",
    "ax[1,1].plot(recall, precision, label='Precision-Recall curve') # Plot the Precision-Recall curve\n",
    "ax[1,1].set_xlabel('Recall')\n",
    "ax[1,1].set_ylabel('Precision')\n",
    "ax[1,1].set_title('Precision-Recall curve and ROC Curve')\n",
    "ax[1,1].legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef11051b-2bc1-4706-9b72-1b992e572e72",
   "metadata": {},
   "source": [
    "<a id='1.1.5'></a>\n",
    "### 1.1.5 Silhouette Score (Unsupervised, Unlabelled)\n",
    "\n",
    "The silhouette score is a measure of how well each sample in a clustering algorithm has been classified. It provides an estimate of the similarity between an observation and the other points within its own cluster, compared to the similarity between that observation and points in other clusters. The score ranges from -1 to 1, with a high score indicating that the sample is well-matched to its own cluster and poorly-matched to other clusters. A score of 0 indicates that the sample is on or close to the decision boundary between two clusters.\n",
    "\n",
    "The silhouette score can be a useful tool for determining the optimal number of clusters in a clustering algorithm. Generally, a higher silhouette score indicates a better clustering solution, although the optimal number of clusters may also depend on the specific requirements of the problem at hand. The silhouette score can also be visualized using a silhouette plot, which shows the silhouette score for each sample on the y-axis and the silhouette coefficient values on the x-axis.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "836f0f21-c9ce-4075-8a26-a568d1b32353",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.metrics import silhouette_samples, silhouette_score\n",
    "import os\n",
    "import cv2\n",
    "import imageio\n",
    "from IPython.display import Image as IpyImage\n",
    "X, y = make_blobs(n_samples=500, n_features=2, centers=4, random_state=1) # Generate sample data\n",
    "range_n_clusters = [2, 3, 4, 5, 6, 7, 8] # Number of clusters\n",
    "for n_clusters in range_n_clusters:\n",
    "    km = KMeans(n_clusters=n_clusters, random_state=1) # Initialize KMeans algorithm\n",
    "    cluster_labels = km.fit_predict(X)\n",
    "    silhouette_avg = silhouette_score(X, cluster_labels) # Compute the silhouette score\n",
    "    print(\"For n_clusters =\", n_clusters, \"The average silhouette_score is :\", silhouette_avg)\n",
    "    sample_silhouette_values = silhouette_samples(X, cluster_labels) # Compute the silhouette scores for each sample\n",
    "    y_lower = 10\n",
    "    for i in range(n_clusters): # Aggregate the silhouette scores for samples belonging to\n",
    "        ith_cluster_silhouette_values = \\\n",
    "            sample_silhouette_values[cluster_labels == i] # cluster i, and sort them\n",
    "        ith_cluster_silhouette_values.sort()\n",
    "        size_cluster_i = ith_cluster_silhouette_values.shape[0]\n",
    "        y_upper = y_lower + size_cluster_i\n",
    "        color = plt.cm.Spectral(float(i) / n_clusters)\n",
    "        plt.fill_betweenx(np.arange(y_lower, y_upper),\n",
    "                          0, ith_cluster_silhouette_values,\n",
    "                          facecolor=color, edgecolor=color, alpha=0.7)\n",
    "        plt.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i)) # Label the silhouette plots with their cluster numbers at the middle\n",
    "        y_lower = y_upper + 10 # Compute the new y_lower for next plot # 10 for the 0 samples\n",
    "    plt.title(f\"The silhouette plot for the various clusters.\")\n",
    "    plt.xlabel(\"The silhouette coefficient values\")\n",
    "    plt.ylabel(\"Cluster label\")\n",
    "    plt.axvline(x=silhouette_avg, color=\"red\", linestyle=\"--\") # The vertical line for average silhouette score of all the values\n",
    "    plt.yticks([])  # Clear the yaxis labels / ticks\n",
    "    plt.xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])\n",
    "    plt.savefig(f'graphs/silhou/silhouette0{i+1}.png')\n",
    "    plt.close()\n",
    "imgs = os.listdir('graphs/silhou/')\n",
    "imgs.sort()\n",
    "imgs=imgs[2:]\n",
    "imgs = [cv2.imread('graphs/silhou/' + i) for i in imgs]\n",
    "imgs = [cv2.cvtColor(i, cv2.COLOR_BGR2RGB) for i in imgs]\n",
    "imageio.mimsave('graphs/sil.gif', imgs, fps=2)\n",
    "path=\"graphs/sil.gif\"\n",
    "with open(path,'rb') as f:\n",
    "    display(IpyImage(data=f.read(), format='png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "964b5bdb-1ff8-45af-9f27-653ef89cb326",
   "metadata": {},
   "source": [
    "<a id='1.1.6'></a>\n",
    "### 1.1.6 Cross-Entropy Loss\n",
    "Cross-entropy loss is a commonly used loss function in machine learning for classification tasks. It measures the difference between the predicted probability distribution and the actual probability distribution of a classification problem.\n",
    "\n",
    "In other words, it calculates the distance between the predicted output and the actual output by taking into account the probability of the correct label.\n",
    "\n",
    "Cross-entropy loss is often used in neural network models, where the predicted output is generated by passing the input through the network's layers and applying a softmax activation function to obtain a probability distribution.\n",
    "\n",
    "The formula for cross-entropy loss can be expressed as:\n",
    "\n",
    "$L(\\theta)=-log(Softmax(z_i))=-z_i+log\\sum_j exp(z_j)$\\\n",
    "\n",
    "where y is the actual probability distribution, ŷ is the predicted probability distribution, and i iterates over the number of classes.\n",
    "\n",
    "The goal of training a model is to minimize the cross-entropy loss, which is achieved by adjusting the model's parameters through a process called backpropagation.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3933a3a-5f03-46b8-af05-9842e032fd75",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8c08b222-620d-45e8-9603-0da9961f9fe2",
   "metadata": {},
   "source": [
    "<a id='1.2'></a>\n",
    "## 1.2 Regression\n",
    "<a id='1.2.1'></a>\n",
    "### 1.2.1 R-square\n",
    "\n",
    "R-squared is a measure of the proportion of variance in the dependent variable (also known as the response or output variable) that can be explained by the independent variables (also known as predictors or inputs) in the regression model. R2 values range from 0 to 1, with a higher value indicating a better fit of the model to the data. R2 can be interpreted as the percentage of the total variance in the response variable that is explained by the model.\n",
    "\n",
    "${\\displaystyle {\\bar {y}}={\\frac {1}{n}}\\sum _{i=1}^{n}y_{i}}$\\\n",
    "${\\displaystyle SS_{\\text{res}}=\\sum _{i}(y_{i}-f_{i})^{2}=\\sum _{i}e_{i}^{2}\\,}$\\\n",
    "${\\displaystyle SS_{\\text{tot}}=\\sum _{i}(y_{i}-{\\bar {y}})^{2}}$\\\n",
    "${\\displaystyle R^{2}=1-{{\\sum _{i}e_{i}^{2}} \\over \\sum _{i}(y_{i}-{\\bar {y}})^{2}}}$\n",
    "<a id='1.2.2'></a>\n",
    "### 1.2.2 RMSE\n",
    "\n",
    "Root mean squared error (RMSE) is a measure of the difference between the actual and predicted values in a regression model. It is calculated by taking the square root of the mean of the squared differences between the actual and predicted values. The RMSE provides an estimate of the average magnitude of the errors in the predictions, with a lower RMSE indicating a better fit of the model to the data.\n",
    "\n",
    "${\\displaystyle \\operatorname {RMSD} ={\\sqrt {\\frac {\\sum _{n=1}^{N}({\\hat {y}}_{n}-y_{n})^{2}}{N}}}}$\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8b650ee-abd7-49fe-b0e4-e0eb71e2eb22",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import AdaBoostRegressor\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "rng = np.random.RandomState(2023)\n",
    "X = np.linspace(0, 10, 200)[:, np.newaxis]\n",
    "y = np.sin(X).ravel() + np.sin(2 * X).ravel() + rng.normal(0, 0.2, X.shape[0])\n",
    "reg1 = DecisionTreeRegressor(max_depth=4)\n",
    "reg2 = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=500, random_state=rng)\n",
    "reg1.fit(X, y)\n",
    "reg2.fit(X, y)\n",
    "y1 = reg1.predict(X)\n",
    "y2 = reg2.predict(X)\n",
    "colors = sns.color_palette(\"colorblind\")\n",
    "plt.figure()\n",
    "plt.scatter(X, y, color=colors[0], label=\"training samples\")\n",
    "plt.plot(X, y1, color=colors[1], label=\"n_estimators=1\", linewidth=2)\n",
    "plt.plot(X, y2, color=colors[2], label=\"n_estimators=500\", linewidth=2)\n",
    "plt.xlabel(\"data\")\n",
    "plt.ylabel(\"target\")\n",
    "plt.title(\"Boosted Decision Tree Regression\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "r2 = r2_score(y, y1) # Compute R-squared\n",
    "r22 = r2_score(y, y2)\n",
    "mse = mean_squared_error(y, y1) # Compute RMSE\n",
    "mse2 = mean_squared_error(y, y2)\n",
    "rmse = np.sqrt(mse)\n",
    "rmse2 = np.sqrt(mse2)\n",
    "print(\"R-squared of 1 estimator: \", r2, 'R-squared of 500 estimator', r22)\n",
    "print(\"RMSE of 1 estimator: \", rmse, 'RMSE of 500 estimator', rmse2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1340ba07-b51c-48d2-b674-140eb71fc727",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "# 2. Machine Learning methods\n",
    "<a id='2.1'></a>\n",
    "## 2.1 Classification\n",
    "<a id='2.1.1'></a>\n",
    "### 2.1.1 K-Neighbours\n",
    "K-nearest neighbors (KNN) is a simple machine learning algorithm that categorizes a new data point based on its similarity to the k nearest data points in a training dataset. The idea behind KNN is that data points that are close together in the feature space belong to the same class. To make a prediction for a new data point, the algorithm finds the k training points that are closest to the new point and then assigns the most common class among these nearest neighbors as the class for the new data point. The value of k is a hyperparameter that is chosen before the model is trained, and it determines the number of nearest neighbors that are used to make a prediction.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14ddf4ae-e315-4b1a-a24b-ad0893fd62f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from matplotlib.colors import ListedColormap\n",
    "data_dic = datasets.load_iris()\n",
    "features = data_dic['data']\n",
    "targets = data_dic['target']\n",
    "c1 = features[targets==0]\n",
    "c2 = features[targets==1]\n",
    "c3 = features[targets==2]\n",
    "ind1, ind2 = 0,1\n",
    "plt.scatter(c1[:,ind1],c1[:,ind2], color='red', marker='s', alpha=0.5, label=\"Setosa\")\n",
    "plt.scatter(c2[:,ind1],c2[:,ind2], color='blue', marker='x', alpha=0.5, label=\"Versicolour\")\n",
    "plt.scatter(c3[:,ind1],c3[:,ind2], color='green', marker='o', alpha=0.5, label=\"Versicolour\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"sepal length [cm]\")\n",
    "plt.ylabel(\"sepal width [cm]\");\n",
    "def subSample(nData):\n",
    "    X = np.empty((3*nData,2))\n",
    "    X[:nData] = c1[:nData,(ind1, ind2)]\n",
    "    X[nData:2*nData] = c2[:nData,(ind1, ind2)]\n",
    "    X[2*nData:] = c3[:nData,(ind1, ind2)]\n",
    "    Y = np.empty(3*nData)\n",
    "    Y[:nData] = np.zeros(nData)\n",
    "    Y[nData:2*nData] = np.ones(nData)\n",
    "    Y[2*nData:] = 2*np.ones(nData)\n",
    "    return X,Y\n",
    "X, Y = subSample(50)\n",
    "scaler = StandardScaler()\n",
    "X = scaler.fit_transform(X)\n",
    "resolution=0.01\n",
    "def regions3(X, y, classifier, colors = ['red','blue','green']):\n",
    "    markers = ('s', 'x', 'o', '^', 'v') # setup marker generator and color map\n",
    "    cmap = ListedColormap(colors[:len(np.unique(y))])\n",
    "    x1_min, x1_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1 # plot the decision surface\n",
    "    x2_min, x2_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1\n",
    "    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),\n",
    "                            np.arange(x2_min, x2_max, resolution))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    rescaledX = scaler.inverse_transform(np.array([xx1.ravel(), xx2.ravel()]).T) # inverse scaling of xx1 and xx2\n",
    "    xx1_rescaled = rescaledX[:,0].reshape(xx1.shape)\n",
    "    xx2_rescaled = rescaledX[:,1].reshape(xx2.shape)\n",
    "    plt.contourf(xx1_rescaled, xx2_rescaled, Z, alpha=0.1, cmap=cmap)\n",
    "    plt.xlim(xx1_rescaled.min(), xx1_rescaled.max())\n",
    "    plt.ylim(xx2_rescaled.min(), xx2_rescaled.max())\n",
    "    c2 = scaler.inverse_transform(X[y==2])\n",
    "    c1 = scaler.inverse_transform(X[y==1])\n",
    "    c0 = scaler.inverse_transform(X[y==0])\n",
    "    xb,yb=c0[:,0],c0[:,1]\n",
    "    plt.scatter(xb,yb,color=colors[0],alpha=0.4)\n",
    "    xb,yb=c1[:,0],c1[:,1]\n",
    "    plt.scatter(xb,yb,color=colors[1],alpha=0.4)\n",
    "    xb,yb=c2[:,0],c2[:,1]\n",
    "    plt.scatter(xb,yb,color=colors[2],alpha=0.4)\n",
    "    plt.xlabel('sepal length [cm]')\n",
    "    plt.ylabel('sepal width [cm]');\n",
    "for k in [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,30]:\n",
    "    KNclass = KNeighborsClassifier(n_neighbors=k)\n",
    "    KNclass.fit(X, Y)\n",
    "    regions3(X, Y, KNclass)\n",
    "    plt.title(\"n={}\".format(k))\n",
    "    if k <= 9:\n",
    "        plt.savefig(f'graphs/K/k0{k}.png')\n",
    "    else :\n",
    "        plt.savefig(f'graphs/K/k{k}.png')\n",
    "    plt.close()\n",
    "imgs = os.listdir('graphs/K/')\n",
    "imgs.sort()\n",
    "imgs=imgs[1:]\n",
    "imgs = [cv2.imread('graphs/K/' + i) for i in imgs]\n",
    "imgs = [cv2.cvtColor(i, cv2.COLOR_BGR2RGB) for i in imgs]\n",
    "imageio.mimsave('graphs/k.gif', imgs, fps=1.5)\n",
    "path=\"graphs/k.gif\"\n",
    "with open(path,'rb') as f:\n",
    "    display(IpyImage(data=f.read(), format='png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7934a148-1cd2-403c-9c43-8e315db0424d",
   "metadata": {},
   "source": [
    "<a id='2.1.2'></a>\n",
    "### 2.1.2 Logistic Regression\n",
    "\n",
    "Logistic Regression is a statistical method used for binary classification problems, where the target variable can only take two values (e.g. yes/no, 0/1, etc.). It is a type of Generalized Linear Model (GLM) and is used to model the relationship between a set of predictor variables (also known as independent variables, inputs, or features) and a binary response variable.\n",
    "\n",
    "In Logistic Regression, the predicted probabilities of the target variable are modeled using a logistic function, which maps the output of a linear combination of the predictor variables to a value between 0 and 1. The predicted probabilities can then be transformed into binary predictions using a threshold value (e.g. 0.5). The coefficients of the predictor variables in the linear combination are estimated using maximum likelihood estimation.\n",
    "\n",
    "Logistic Regression is a simple and widely used method for binary classification problems and is particularly useful when the relationship between the predictor variables and the target variable is approximately linear. However, it can be limited in its ability to model complex non-linear relationships and may not perform well when the target variable is highly imbalanced.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "097c8e83-e1a4-47ab-bb97-9c0c155f217e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "data = load_breast_cancer() # Load the breast cancer dataset\n",
    "X, y = data.data, data.target\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0) # Split the data into training and test sets\n",
    "logistic_regression = LogisticRegression(solver='sag', max_iter=100000) # Train the logistic regression model\n",
    "logistic_regression.fit(X_train, y_train)\n",
    "y_pred = logistic_regression.predict(X_test) # Make predictions on the test set\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print('Accuracy:', accuracy)\n",
    "plt.scatter(np.arange(len(y_test)), y_test, c='blue', marker='o', label='True Value') # Plot the results\n",
    "plt.scatter(np.arange(len(y_pred)), y_pred, c='red', marker='x', label='Prediction')\n",
    "plt.legend()\n",
    "plt.xlabel('Sample Index')\n",
    "plt.ylabel('Target Value')\n",
    "plt.title('Logistic Regression Results')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "402e6089-c0ee-4a7d-a7a6-4c1c1b475792",
   "metadata": {},
   "source": [
    "<a id='2.1.3'></a>\n",
    "### 2.1.3 Supportive Vector Machine (SVM)\n",
    "\n",
    "Support Vector Machine (SVM) is a type of supervised learning algorithm used for classification and regression problems. SVM tries to find a hyperplane that separates the data into classes in a way that maximizes the margin between the classes. The margin is defined as the distance between the closest data points from each class and the hyperplane. These closest data points are called the support vectors, and they play a key role in determining the location of the hyperplane.\n",
    "\n",
    "In binary classification problems, SVM finds a hyperplane that separates the positive and negative classes. In multi-class classification problems, SVM creates multiple binary classifiers, one for each class pair, and chooses the best classifier based on the maximum margin.\n",
    "\n",
    "SVM can handle non-linear classification problems by transforming the data into a higher dimensional space and finding a hyperplane in that space. This is achieved through a technique called kernel trick, where a kernel function is used to project the data into a higher dimensional space without actually computing the projections.\n",
    "\n",
    "SVM has several advantages over other machine learning algorithms, such as good performance on high dimensional data, ability to handle non-linear classification problems, and robustness against overfitting. However, SVM can be computationally expensive for large datasets and may not perform well when the number of features is much larger than the number of samples.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3881e4ba-04b1-407f-bf78-798c3ec35c75",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.inspection import DecisionBoundaryDisplay\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, :2] # Take the first two features\n",
    "y = iris.target\n",
    "C = 2.0  # SVM regularization parameter\n",
    "models = (\n",
    "    svm.SVC(kernel=\"linear\", C=C),\n",
    "    svm.LinearSVC(C=C, max_iter=100000),\n",
    "    svm.SVC(kernel=\"rbf\", gamma=0.7, C=C),\n",
    "    svm.SVC(kernel=\"poly\", degree=4, gamma=\"auto\", C=C),)\n",
    "models = (SVM.fit(X, y) for SVM in models)\n",
    "titles = ( # title for the plots\n",
    "    \"SVC with linear kernel\",\n",
    "    \"LinearSVC (linear kernel)\",\n",
    "    \"SVC with RBF kernel\",\n",
    "    \"SVC with polynomial (degree 4) kernel\",)\n",
    "fig, sub = plt.subplots(2, 2,figsize=(10,10)) # Set-up 2x2 grid for plotting.\n",
    "plt.subplots_adjust(wspace=0.4, hspace=0.4)\n",
    "X0, X1 = X[:, 0], X[:, 1]\n",
    "for SVM, title, ax in zip(models, titles, sub.flatten()):\n",
    "    disp = DecisionBoundaryDisplay.from_estimator(SVM, X, response_method=\"predict\",\n",
    "        cmap=plt.cm.twilight,\n",
    "        alpha=0.8, ax=ax,\n",
    "        xlabel=iris.feature_names[0],\n",
    "        ylabel=iris.feature_names[1],)\n",
    "    ax.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors=\"k\")\n",
    "    ax.set_xticks(())\n",
    "    ax.set_yticks(())\n",
    "    ax.set_title(title)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32b29a45-fbe6-4696-a47e-4045648a4537",
   "metadata": {},
   "source": [
    "<a id='2.1.4'></a>\n",
    "### 2.1.4 Random Forest\n",
    "ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the mean or average prediction of the individual trees is returned.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7380281-6ef4-4448-9cce-ced54e5ea7a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=0, random_state=42) # Generate a toy dataset\n",
    "train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.2, random_state=42) # Split the dataset into training and testing sets\n",
    "rf = RandomForestClassifier(n_estimators=100, random_state=42) # Train a Random Forest classifier\n",
    "rf.fit(train_X, train_y)\n",
    "probs = rf.predict_proba(test_X) # Make predictions on the testing set\n",
    "preds = probs[:,1]\n",
    "fpr, tpr, _ = roc_curve(test_y, preds) # Compute the ROC curve and AUC\n",
    "roc_auc = auc(fpr, tpr)\n",
    "fig, ax = plt.subplots() # Plot the ROC curve and AUC\n",
    "ax.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "ax.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "ax.set_xlim([0.0, 1.0])\n",
    "ax.set_ylim([0.0, 1.05])\n",
    "ax.set_xlabel('False Positive Rate')\n",
    "ax.set_ylabel('True Positive Rate')\n",
    "ax.set_title('Receiver operating characteristic')\n",
    "ax.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72810a63-b703-46ff-9cd0-375a86cebb7b",
   "metadata": {},
   "source": [
    "<a id='2.2'></a>\n",
    "## 2.2 Regression\n",
    "<a id='2.2.1'></a>\n",
    "### 2.2.1 Multiple Linear Regression, Ridge, Lasso Regression and Polynomial Regression\n",
    "Multiple Linear Regression: Multiple linear regression is a statistical method used to model the relationship between multiple independent variables (also known as predictors, features, or inputs) and a dependent variable (also known as the response, target, or output). The goal of multiple linear regression is to find the linear combination of the independent variables that best predicts the dependent variable. The relationship between the variables is modeled as a linear equation, where the coefficients represent the strengths of the relationships between each independent variable and the dependent variable.\n",
    "\n",
    "Polynomial Regression: Polynomial regression is a type of regression analysis that models the relationship between the independent variable and the dependent variable as an nth degree polynomial. The polynomial equation can capture non-linear relationships between the variables, and is especially useful when the relationship is not well-approximated by a linear model. The polynomial coefficients are estimated using regression techniques, and the polynomial equation is used to make predictions for new values of the independent variable.\n",
    "\n",
    "Ridge Regression: Ridge regression is a regularized version of linear regression that adds a penalty term to the cost function used to estimate the coefficients of the model. The penalty term is the sum of the squares of the coefficients, multiplied by a regularization parameter, alpha. The regularization parameter controls the amount of shrinkage applied to the coefficients, and helps to prevent overfitting in the model. Overfitting occurs when the model is too complex and fits the training data too closely, leading to poor performance on new, unseen data. Ridge regression reduces the magnitude of the coefficients and improves the generalization ability of the model.\n",
    "\n",
    "Lasso Regression: Lasso regression is another regularized version of linear regression that adds a penalty term to the cost function used to estimate the coefficients of the model. The penalty term is the sum of the absolute values of the coefficients, multiplied by a regularization parameter, alpha. The regularization parameter controls the amount of shrinkage applied to the coefficients, and helps to prevent overfitting in the model. Unlike ridge regression, lasso regression has the added advantage of performing feature selection by setting some coefficients to zero, effectively removing those features from the model. This can improve the interpretability of the model and reduce the risk of overfitting.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d0e13c0-69db-49bd-a0b9-633e4da318d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.model_selection import train_test_split\n",
    "data_url = \"http://lib.stat.cmu.edu/datasets/boston\" # Load the Boston housing dataset\n",
    "raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
    "X = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
    "y = raw_df.values[1::2, 2]\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=2023) # Split the data into training and testing sets\n",
    "regressor = LinearRegression() # Multiple Linear Regression\n",
    "regressor.fit(X_train, y_train)\n",
    "y_pred_mlr = regressor.predict(X_test)\n",
    "poly_reg = PolynomialFeatures(degree=3) # Polynomial Regression\n",
    "X_poly = poly_reg.fit_transform(X)\n",
    "regressor_poly = LinearRegression()\n",
    "regressor_poly.fit(X_poly, y)\n",
    "y_pred_poly = regressor_poly.predict(poly_reg.transform(X_test))\n",
    "ridge_reg = Ridge(alpha=0.8) # Ridge Regression\n",
    "ridge_reg.fit(X_train, y_train)\n",
    "y_pred_ridge = ridge_reg.predict(X_test)\n",
    "lasso_reg = Lasso(alpha=0.1)  # Lasso Regression\n",
    "lasso_reg.fit(X_train, y_train)\n",
    "y_pred_lasso = lasso_reg.predict(X_test)\n",
    "print(\"Multiple Linear Regression R2 Score:\", regressor.score(X_test, y_test)) # Evaluate the models\n",
    "print(\"Polynomial Regression R2 Score:\", regressor_poly.score(poly_reg.transform(X_test), y_test))\n",
    "print(\"Ridge Regression R2 Score:\", ridge_reg.score(X_test, y_test))\n",
    "print(\"Lasso Regression R2 Score:\", lasso_reg.score(X_test, y_test))\n",
    "plt.scatter(y_test, y_pred_mlr, color='blue', label='Multiple Linear Regression') # Plot the results\n",
    "plt.scatter(y_test, y_pred_poly, color='red', label='Polynomial Regression')\n",
    "plt.scatter(y_test, y_pred_ridge, color='green', label='Ridge Regression')\n",
    "plt.scatter(y_test, y_pred_lasso, color='yellow', label='Lasso Regression')\n",
    "plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'k--', lw=2)\n",
    "plt.xlabel('True Values')\n",
    "plt.ylabel('Predictions')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36304d50-e33d-4484-ab16-1554643d700c",
   "metadata": {},
   "source": [
    "<a id='2.2.2'></a>\n",
    "### 2.2.2 Ridge and Lasso Regularization\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c29b2720-32e2-4100-a62e-3ebed27c8934",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy import sparse\n",
    "from scipy import ndimage\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.linear_model import Ridge\n",
    "import matplotlib.pyplot as plt\n",
    "def _weights(x, dx=1, orig=0):\n",
    "    x = np.ravel(x)\n",
    "    floor_x = np.floor((x - orig) / dx).astype(np.int64)\n",
    "    alpha = (x - orig - floor_x * dx) / dx\n",
    "    return np.hstack((floor_x, floor_x + 1)), np.hstack((1 - alpha, alpha))\n",
    "def _generate_center_coordinates(l_x):\n",
    "    X, Y = np.mgrid[:l_x, :l_x].astype(np.float64)\n",
    "    center = l_x / 2.0\n",
    "    X += 0.5 - center\n",
    "    Y += 0.5 - center\n",
    "    return X, Y\n",
    "def build_projection_operator(l_x, n_dir):\n",
    "    X, Y = _generate_center_coordinates(l_x)\n",
    "    angles = np.linspace(0, np.pi, n_dir, endpoint=False)\n",
    "    data_inds, weights, camera_inds = [], [], []\n",
    "    data_unravel_indices = np.arange(l_x**2)\n",
    "    data_unravel_indices = np.hstack((data_unravel_indices, data_unravel_indices))\n",
    "    for i, angle in enumerate(angles):\n",
    "        Xrot = np.cos(angle) * X - np.sin(angle) * Y\n",
    "        inds, w = _weights(Xrot, dx=1, orig=X.min())\n",
    "        mask = np.logical_and(inds >= 0, inds < l_x)\n",
    "        weights += list(w[mask])\n",
    "        camera_inds += list(inds[mask] + i * l_x)\n",
    "        data_inds += list(data_unravel_indices[mask])\n",
    "    proj_operator = sparse.coo_matrix((weights, (camera_inds, data_inds)))\n",
    "    return proj_operator\n",
    "def generate_synthetic_data():\n",
    "    rs = np.random.RandomState(2023)\n",
    "    n_pts = 36\n",
    "    x, y = np.ogrid[0:l, 0:l]\n",
    "    mask_outer = (x - l / 2.0) ** 2 + (y - l / 2.0) ** 2 < (l / 2.0) ** 2\n",
    "    mask = np.zeros((l, l))\n",
    "    points = l * rs.rand(2, n_pts)\n",
    "    mask[(points[0]).astype(int), (points[1]).astype(int)] = 1\n",
    "    mask = ndimage.gaussian_filter(mask, sigma=l / n_pts)\n",
    "    res = np.logical_and(mask > mask.mean(), mask_outer)\n",
    "    return np.logical_xor(res, ndimage.binary_erosion(res))\n",
    "l = 200 # Generate synthetic images, and projections\n",
    "proj_operator = build_projection_operator(l, l // 7)\n",
    "data = generate_synthetic_data()\n",
    "proj = proj_operator @ data.ravel()[:, np.newaxis]\n",
    "proj += 0.15 * np.random.randn(*proj.shape)\n",
    "rgr_ridge = Ridge(alpha=0.2) # Reconstruction with L2 (Ridge) penalization\n",
    "rgr_ridge.fit(proj_operator, proj.ravel())\n",
    "rec_l2 = rgr_ridge.coef_.reshape(l, l)\n",
    "rgr_lasso = Lasso(alpha=0.001) # Reconstruction with L1 (Lasso) penalization\n",
    "rgr_lasso.fit(proj_operator, proj.ravel()) # the best value of alpha was determined using cross validation\n",
    "rec_l1 = rgr_lasso.coef_.reshape(l, l) # with LassoCV\n",
    "plt.figure(figsize=(8, 3.3))\n",
    "plt.subplot(131)\n",
    "plt.imshow(data, cmap=plt.cm.gray, interpolation=\"nearest\")\n",
    "plt.axis(\"off\")\n",
    "plt.title(\"original image\")\n",
    "plt.subplot(132)\n",
    "plt.imshow(rec_l2, cmap=plt.cm.gray, interpolation=\"nearest\")\n",
    "plt.title(\"L2 penalization\")\n",
    "plt.axis(\"off\")\n",
    "plt.subplot(133)\n",
    "plt.imshow(rec_l1, cmap=plt.cm.gray, interpolation=\"nearest\")\n",
    "plt.title(\"L1 penalization\")\n",
    "plt.axis(\"off\")\n",
    "plt.subplots_adjust(hspace=0.01, wspace=0.01, top=1, bottom=0, left=0, right=1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c812af6c-91df-423d-997d-dd36d3f83641",
   "metadata": {},
   "source": [
    "<a id='2.2.3'></a>\n",
    "### 2.2.3 LAD (Least Absolute Deviations) Regression\n",
    "LAD Regression (Least Absolute Deviations Regression) is a type of regression analysis that minimizes the sum of the absolute differences between the observed values and the predicted values of the dependent variable. Unlike least squares regression, which minimizes the sum of the squares of the differences between the observed and predicted values, LAD regression minimizes the sum of the absolute differences. This makes LAD regression more robust to outliers, as it is not affected by extreme values in the same way that least squares regression is.\n",
    "\n",
    "LAD regression is also known as the median regression or robust regression, as it is less sensitive to outliers than other regression methods. The goal of LAD regression is to find the coefficients of the independent variables that minimize the absolute differences between the observed and predicted values of the dependent variable. The coefficients are estimated using optimization techniques, and the model is used to make predictions for new values of the independent variables.\n",
    "\n",
    "LAD regression is a useful method for modeling non-linear relationships between the independent and dependent variables, and is especially useful in situations where the data contains outliers or other extreme values. LAD regression is also computationally efficient, as it can be solved using linear programming methods.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57fe9b00-db05-4690-ab35-5dfb26a8aa4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import statsmodels.api as sm\n",
    "import matplotlib.pyplot as plt\n",
    "x = np.random.rand(100, 1) # Load the data\n",
    "y = 2 + 3 * x + np.random.rand(100, 1)\n",
    "x = sm.add_constant(x) # Add a constant to the independent variable for the intercept term\n",
    "model = sm.RLM(y, x, M=sm.robust.norms.HuberT()) # Fit the LAD regression model\n",
    "results = model.fit()\n",
    "y_pred = results.predict(x) # Make predictions using the LAD regression model\n",
    "plt.scatter(y, y_pred) # Plot the observed values against the predicted values\n",
    "plt.xlabel('Observed Values')\n",
    "plt.ylabel('Predicted Values')\n",
    "plt.title('LAD Regression')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb9bed2-6b32-4527-8527-2ee2818ced8d",
   "metadata": {},
   "source": [
    "<a id='2.2.4'></a>\n",
    "### 2.2.4 Bayesian Ridge and Lasso Regression\n",
    "\n",
    "Bayesian Ridge Regression is a Bayesian version of Ridge Regression, which is a linear regression method that adds a L2 regularization term to the objective function. In Bayesian Ridge Regression, the regularization term is modeled as a Gaussian prior over the coefficients, with a mean of zero and a precision (inverse of the variance) parameter. The precision parameter is estimated along with the coefficients, which allows the model to account for uncertainty in the coefficients and also reduces the risk of overfitting.\n",
    "\n",
    "Bayesian Lasso Regression is a Bayesian version of Lasso Regression, which is a linear regression method that adds a L1 regularization term to the objective function. In Bayesian Lasso Regression, the regularization term is modeled as a Laplace prior over the coefficients, with a mean of zero and a scale parameter. The scale parameter is estimated along with the coefficients, which allows the model to account for uncertainty in the coefficients and also reduces the risk of overfitting.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a12ffad6-81d9-41c5-b1ba-e95ab5941e35",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import BayesianRidge\n",
    "import matplotlib.pyplot as plt\n",
    "def func(x):\n",
    "    return np.sin(2 * np.pi * x)\n",
    "size = 25\n",
    "rng = np.random.RandomState(2023)\n",
    "x_train = rng.uniform(0.0, 1.0, size)\n",
    "y_train = func(x_train) + rng.normal(scale=0.1, size=size)\n",
    "x_test = np.linspace(0.0, 1.0, 100)\n",
    "n_order = 3\n",
    "X_train = np.vander(x_train, n_order + 1, increasing=True)\n",
    "X_test = np.vander(x_test, n_order + 1, increasing=True)\n",
    "reg = BayesianRidge(tol=1e-6, fit_intercept=False, compute_score=True)\n",
    "fig, axes = plt.subplots(1, 2, figsize=(8, 4))\n",
    "for i, ax in enumerate(axes): # Bayesian ridge regression with different initial value pairs\n",
    "    if i == 0:\n",
    "        init = [1 / np.var(y_train), 1.0]  # Default values\n",
    "    elif i == 1:\n",
    "        init = [1.0, 1e-3]\n",
    "        reg.set_params(alpha_init=init[0], lambda_init=init[1])\n",
    "    reg.fit(X_train, y_train)\n",
    "    ymean, ystd = reg.predict(X_test, return_std=True)\n",
    "    ax.plot(x_test, func(x_test), color=\"blue\", label=\"sin($2\\\\pi x$)\")\n",
    "    ax.scatter(x_train, y_train, s=50, alpha=0.5, label=\"observation\")\n",
    "    ax.plot(x_test, ymean, color=\"red\", label=\"predict mean\")\n",
    "    ax.fill_between(x_test, ymean - ystd, ymean + ystd, color=\"pink\", alpha=0.5, label=\"predict std\")\n",
    "    ax.set_ylim(-1.3, 1.3)\n",
    "    ax.legend()\n",
    "    title = \"$\\\\alpha$_init$={:.2f},\\\\ \\\\lambda$_init$={}$\".format(init[0], init[1])\n",
    "    if i == 0:\n",
    "        title += \" (Default)\"\n",
    "    ax.set_title(title, fontsize=12)\n",
    "    text = \"$\\\\alpha={:.1f}$\\n$\\\\lambda={:.3f}$\\n$L={:.1f}$\".format(reg.alpha_, reg.lambda_, reg.scores_[-1])\n",
    "    ax.text(0.05, -1.0, text, fontsize=12)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b23794-3408-46b2-9473-4585b209b6a6",
   "metadata": {},
   "source": [
    "<a id='2.2.5'></a>\n",
    "### 2.2.5 Neural Network Regression\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1118353-f03f-43f7-9f36-dc1ebfe61581",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "class MyLayer(tf.keras.layers.Layer):\n",
    "    weights=[]\n",
    "    biases=[]\n",
    "    def __init__(self, units=32, activation=None):\n",
    "        super(MyLayer, self).__init__()\n",
    "        self.units = units\n",
    "        self.activation = tf.keras.activations.get(activation)\n",
    "        self.weights=[]\n",
    "        self.biases=[]\n",
    "        weights=[]\n",
    "        biases=[]\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(shape=(input_shape[-1], self.units), initializer='random_normal', trainable=True)\n",
    "        self.b = self.add_weight(shape=(self.units,), initializer='random_normal',trainable=True)\n",
    "    def call(self, inputs):\n",
    "        self.weights.append(self.w)\n",
    "        self.biases.append(self.b)\n",
    "        self.avw=sum(self.weights)/len(self.weights)\n",
    "        self.avb=sum(self.biases)/len(self.biases)\n",
    "        x = tf.matmul(inputs, self.avw) + self.avb\n",
    "        if self.activation is not None:\n",
    "            x = self.activation(x)\n",
    "        return x\n",
    "dataset_path = keras.utils.get_file(\"auto-mpg.data\", \"http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data\")\n",
    "column_names = ['MPG','Cylinders','Displacement','Horsepower','Weight', 'Acceleration', 'Model Year', 'Origin']\n",
    "raw_dataset = pd.read_csv(dataset_path, names=column_names, na_values = \"?\", comment='\\t', sep=\" \", skipinitialspace=True)\n",
    "dataset = raw_dataset.copy()\n",
    "dataset.tail()\n",
    "dataset.isna().sum()\n",
    "dataset = dataset.dropna()\n",
    "origin = dataset.pop('Origin')\n",
    "dataset['USA'] = (origin == 1)*1.0\n",
    "dataset['Europe'] = (origin == 2)*1.0\n",
    "dataset['Japan'] = (origin == 3)*1.0\n",
    "dataset.tail()\n",
    "train_dataset = dataset.sample(frac=0.8,random_state=0)\n",
    "test_dataset = dataset.drop(train_dataset.index)\n",
    "train_stats = train_dataset.describe()\n",
    "train_stats.pop(\"MPG\")\n",
    "train_stats = train_stats.transpose()\n",
    "train_labels = train_dataset.pop('MPG')\n",
    "test_labels = test_dataset.pop('MPG')\n",
    "def norm(x):\n",
    "    return (x - train_stats['mean']) / train_stats['std']\n",
    "normed_train_data = norm(train_dataset)\n",
    "normed_test_data = norm(test_dataset)\n",
    "def build_model():\n",
    "    model = keras.Sequential([Input(shape=(len(train_dataset.keys()),)),\n",
    "    MyLayer(64, activation='relu'),\n",
    "    MyLayer(64, activation='relu'),\n",
    "    layers.Dense(1)])\n",
    "    optimizer = tf.keras.optimizers.Adam(0.001)\n",
    "    model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse'])\n",
    "    return model\n",
    "def build_model2():\n",
    "    model = keras.Sequential([Input(shape=(len(train_dataset.keys()),)),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(1)])\n",
    "    optimizer = tf.keras.optimizers.Adam(0.001)\n",
    "    model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse'])\n",
    "    return model\n",
    "class PrintDot(keras.callbacks.Callback):\n",
    "    def on_epoch_end(self, epoch, logs):\n",
    "        if epoch % 100 == 0: print('')\n",
    "        print('.', end='')\n",
    "def plot_history(history, history2):\n",
    "    hist = pd.DataFrame(history.history)\n",
    "    hist['epoch'] = history.epoch\n",
    "    hist2 = pd.DataFrame(history2.history)\n",
    "    hist2['epoch'] = history2.epoch\n",
    "    plt.figure(figsize=(8,12))\n",
    "    plt.subplot(2,1,1)\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Mean Abs Error [MPG]')\n",
    "    plt.plot(hist['epoch'], hist['val_mae'], label = 'Val Error')\n",
    "    plt.plot(hist2['epoch'], hist2['mae'], label='Val Error2')\n",
    "    plt.ylim([0,5])\n",
    "    plt.legend()\n",
    "    plt.subplot(2,1,2)\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Mean Square Error [$MPG^2$]')\n",
    "    plt.plot(hist['epoch'], hist['val_mse'], label = 'Val Error')\n",
    "    plt.plot(hist2['epoch'], hist2['mse'], label='Val Error2')\n",
    "    plt.ylim([0,20])\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "EPOCHS = 2000\n",
    "model = build_model()\n",
    "model2 = build_model2()\n",
    "early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=100)\n",
    "history = model.fit(normed_train_data, train_labels, epochs=EPOCHS, validation_split = 0.2, verbose=0, callbacks=[early_stop, PrintDot()])\n",
    "history2 = model2.fit(normed_train_data, train_labels, epochs=EPOCHS, validation_split = 0.2, verbose=0, callbacks=[early_stop, PrintDot()])\n",
    "plot_history(history, history2)\n",
    "loss, mae, mse = model.evaluate(normed_test_data, test_labels, verbose=2)\n",
    "loss2, mae2, mse2 = model2.evaluate(normed_test_data, test_labels, verbose=2)\n",
    "print(\"Gaussian weighted and bias NN error: {:5.2f} MPG\".format(mae))\n",
    "print(\"default weighted and bias NN error: {:5.2f} MPG\".format(mae2))\n",
    "test_predictions = model.predict(normed_test_data).flatten()\n",
    "test_predictions2 = model2.predict(normed_test_data).flatten()\n",
    "plt.scatter(test_labels, test_predictions)\n",
    "plt.scatter(test_labels, test_predictions2)\n",
    "plt.xlabel('True Values [MPG]')\n",
    "plt.ylabel('Predictions [MPG]')\n",
    "plt.axis('equal')\n",
    "plt.axis('square')\n",
    "plt.xlim([0,plt.xlim()[1]])\n",
    "plt.ylim([0,plt.ylim()[1]])\n",
    "_ = plt.plot([-100, 100], [-100, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "420bf942-5b86-42c4-8725-80b93a8e4ad9",
   "metadata": {},
   "source": [
    "<a id='2.2.6'></a>\n",
    "### 2.2.6 Supportive Vector Regression\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51b3122d-cc99-461b-8a88-d8ddeb1e64ab",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "15609fb9-2763-452f-8dcd-551ef689fb3c",
   "metadata": {},
   "source": [
    "<a id='3'></a>\n",
    "# 3. Neural Networks methods\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "81695ffb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 10])\n",
      "Checking accuracy on training data\n",
      "Got 56989/60000 with accuracy 94.98\n",
      "Checking accuracy on test data\n",
      "Got 9462/10000 with accuracy 94.62\n"
     ]
    }
   ],
   "source": [
    "# Basic Neural Networks sample\n",
    "# imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.transforms as transforms\n",
    "# Create Fully connected network\n",
    "class NN(nn.Module):\n",
    "    def __init__(self, input_size, num_classes):\n",
    "        super(NN, self).__init__()\n",
    "        self.fc1=nn.Linear(input_size, 50)\n",
    "        self.fc2=nn.Linear(50, num_classes)\n",
    "    def forward(self, x):\n",
    "        x=F.relu(self.fc1(x))\n",
    "        x=self.fc2(x)\n",
    "        return x\n",
    "model = NN(784, 10)\n",
    "x=torch.randn(64, 784)\n",
    "print(model(x).shape)\n",
    "# Set device\n",
    "device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "# Hyperparameters\n",
    "input_size=784\n",
    "num_classes=10\n",
    "learning_rate=0.001\n",
    "batch_size=64\n",
    "num_epochs=2\n",
    "# Load Data\n",
    "train_dataset=datasets.MNIST(root='dataset/', train=True, transform=transforms.ToTensor(), download=True)\n",
    "train_loader=DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)\n",
    "test_dataset=datasets.MNIST(root='dataset/', train=False, transform=transforms.ToTensor(), download=True)\n",
    "test_loader=DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True)\n",
    "# Initialize network\n",
    "model=NN(input_size=input_size, num_classes=num_classes).to(device)\n",
    "# Loss and Optimizer\n",
    "criterion=nn.CrossEntropyLoss()\n",
    "optimizer=optim.Adam(model.parameters(), lr=learning_rate)\n",
    "# Train networks\n",
    "for epoch in range(num_epochs):\n",
    "    for batch_idx, (data, targets) in enumerate(train_loader):\n",
    "        data=data.to(device=device)\n",
    "        targets=targets.to(device=device)\n",
    "        data=data.reshape(data.shape[0], -1) # Flatten the data\n",
    "        # Forward\n",
    "        scores=model(data)\n",
    "        loss=criterion(scores, targets)\n",
    "        # Backward\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "# Check accuracy on training & test to see how good our model\n",
    "def check_accuracy(loader, model):\n",
    "    if loader.dataset.train:\n",
    "        print(\"Checking accuracy on training data\")\n",
    "    else:\n",
    "        print(\"Checking accuracy on test data\")\n",
    "    num_correct=0\n",
    "    num_samples=0\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for x, y in loader:\n",
    "            x=x.to(device=device)\n",
    "            y=y.to(device=device)\n",
    "            x=x.reshape(x.shape[0], -1)\n",
    "            scores=model(x)\n",
    "            _, predictions=scores.max(1)\n",
    "            num_correct+=(predictions==y).sum()\n",
    "            num_samples+=predictions.size(0)\n",
    "    model.train()\n",
    "    print(f'Got {num_correct}/{num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}')\n",
    "    \n",
    "check_accuracy(train_loader, model)\n",
    "check_accuracy(test_loader, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757d08f9-9878-4776-81a2-d38ca73f83fe",
   "metadata": {},
   "source": [
    "<a id='3.1'></a>\n",
    "## 3.1 Autoencoder and Variate Autoencoder (VAE)\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "949b145a-3600-4d62-8b06-5bd5786a8acd",
   "metadata": {},
   "source": [
    "<a id='3.2'></a>\n",
    "## 3.2 U-Net\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942f2f79-1af1-4e6e-acff-8727f14088e5",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='3.3'></a>\n",
    "## 3.3 Generative Adversarial Network (GAN) and Deep Convolutional GAN\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf0f4b0-45f2-4944-bdee-2ca3111779fa",
   "metadata": {},
   "source": [
    "<a id='3.4'></a>\n",
    "## 3.4 Self-Normalizing Neural Networks (SELU)\n",
    "\n",
    "SELU is found to be a good activation function for GANs and we use that in the first two dense networks. \n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4590e7ab-23bc-4434-a3ec-21a08336d7c0",
   "metadata": {},
   "source": [
    "<a id='3.5'></a>\n",
    "## 3.5 (Self) Compatitivly Generative Network (CGN) - Ensemble model of GAN\n",
    "\n",
    "Two GAN models trained by discriminators with the same dataset. After few epochs later, two GANs generate the same prediction compatitively and final discriminator choose better one and worse one get feedback from final discriminator.\n",
    "\n",
    "develop to tournament model\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d96c74af-000e-4948-a9d4-3d170b483df8",
   "metadata": {},
   "source": [
    "<a id='3.6'></a>\n",
    "## 3.6 Recurrent Neural Networks (RNNs)\n",
    "\n",
    "![RNN](Img/RNN.png)\n",
    "\n",
    "Recurrent Neural Networks (RNNs) are designed to process sequential data, such as time series data, sentences, where the order of information matters. Examples of sequential data include stock market prices, text, or audio.  RNNs have a unique \"memory-like\" ability that allows them to process information while maintaining a sense of context from previous inputs in the sequence. Unlike traditional neural networks, which process inputs independently, RNNs have the unique feature of retaining a form of memory. They achieve this by taking the output of a processing step and feeding it back into the network as part of the input for the next step. This loop enables the network to carry forward information as it processes each piece of the sequence.\n",
    "\n",
    "### How RNNs Work:  At each step in the sequence:\n",
    "\n",
    "An RNN takes the current input data and combines it with information from its previous internal state (the \"memory\"). Calculations are performed using weights and biases (adjustable parameters). The RNN produces an output and updates its internal state for the next step.  Theoretically, this allows RNNs to make predictions based on the entire history of inputs it has seen so far.\n",
    "\n",
    "### The Challenges of Vanishing and Exploding Gradients:\n",
    "\n",
    "Vanishing Gradients: When the weights used to propagate information (Usually in long sequences) through the network are small, causing the gradient (which guides learning) (signals used to update the weights during training) to become so small that it effectively stops updating the weights, making learning stall.\n",
    "\n",
    "Exploding Gradients: Conversely, the exploding gradient problem happens when the weights are too large, causing the gradients to grow exponentially and making the training process unstable.\n",
    "\n",
    "Why LSTMs Are Important:  Due to these challenges, RNNs can be difficult to train over sequences with long distances between relevant information. This limitation led to the development of more sophisticated variants, such as Long Short-Term Memory (LSTM) networks. LSTMs include mechanisms called gates that regulate the flow of information, making it easier to learn and retain long-term dependencies without succumbing to the vanishing or exploding gradient problems. As a result, LSTMs and similar architectures have become more popular for tasks involving sequential data, offering a more robust solution for capturing temporal dependencies.\n",
    "\n",
    "\n",
    "순환 신경망(RNN)은 시계열 데이터, 문장 등 정보의 순서가 중요한 순차적 데이터를 처리하도록 설계되었습니다. 순차적 데이터의 예로는 주식 시세, 텍스트 또는 오디오 등이 있습니다.  RNN은 시퀀스 내 이전 입력의 맥락을 유지하면서 정보를 처리할 수 있는 고유한 '메모리 같은' 능력을 가지고 있습니다. 입력을 독립적으로 처리하는 기존 신경망과 달리 RNN은 일종의 기억을 유지하는 독특한 특징을 가지고 있습니다. 이는 처리 단계의 출력을 다음 단계의 입력의 일부로 네트워크에 다시 공급함으로써 이루어집니다. 이 루프를 통해 네트워크는 시퀀스의 각 부분을 처리할 때 정보를 전달할 수 있습니다.\n",
    "\n",
    "### RNN의 작동 방식:  시퀀스의 각 단계에서:\n",
    "\n",
    "RNN은 현재 입력 데이터를 가져와 이전 내부 상태(\"메모리\")의 정보와 결합합니다. 계산은 가중치와 바이어스(조정 가능한 매개변수)를 사용하여 수행됩니다. RNN은 출력을 생성하고 다음 단계를 위해 내부 상태를 업데이트합니다.  이론적으로 RNN은 지금까지 본 입력의 전체 기록을 바탕으로 예측을 할 수 있습니다.\n",
    "\n",
    "### 소실 및 폭발하는 그라데이션의 과제:\n",
    "\n",
    "소실 그라데이션: 네트워크를 통해 정보를 전파하는 데 사용되는 가중치(보통 긴 시퀀스)가 작아지면 (학습을 안내하는) 기울기(학습 중 가중치 업데이트에 사용되는 신호)가 너무 작아져 가중치 업데이트가 효과적으로 중단되어 학습이 멈추게 되는 경우입니다.\n",
    "\n",
    "폭발하는 그라데이션: 반대로 가중치가 너무 크면 기울기 폭발 문제가 발생하여 기울기가 기하급수적으로 커지고 훈련 과정이 불안정해집니다.\n",
    "\n",
    "LSTM이 중요한 이유  이러한 문제로 인해 RNN은 관련 정보 사이의 거리가 먼 시퀀스에 대해 훈련하기 어려울 수 있습니다. 이러한 한계로 인해 장단기 기억(LSTM) 네트워크와 같은 보다 정교한 변형이 개발되었습니다. LSTM에는 정보의 흐름을 조절하는 게이트라는 메커니즘이 포함되어 있어, 사라지거나 폭발하는 경사 문제에 굴복하지 않고 장기적인 의존성을 더 쉽게 학습하고 유지할 수 있습니다. 그 결과, 순차적 데이터와 관련된 작업에서 LSTM과 유사한 아키텍처가 더욱 인기를 얻고 있으며, 시간적 종속성을 캡처하는 데 더욱 강력한 솔루션을 제공합니다.\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4edac354-cec9-4bb8-bcdc-d8d7e41c4f24",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e129e2c3",
   "metadata": {},
   "source": [
    "<a id='3.7'></a>\n",
    "## 3.7 Long Short Term Memory (LSTM)\n",
    "\n",
    "![LSTM2](Img/LSTM2.png)\n",
    "\n",
    "Long Short-Term Memory (LSTM) networks were developed to address the limitations faced by Recurrent Neural Networks (RNNs), specifically the problem of gradient decay and explosion. Unlike RNNs, LSTMs follow a more complex structure that allows them to retain and manipulate information for longer periods of time.\n",
    "\n",
    "Unlike some neural network contexts that rely on traditional Rectified Linear Units (ReLUs), LSTMs use a unique combination of sigmoid and tangent activation functions. These functions are important for managing the flow of information, allowing the network to maintain long-term dependencies while mitigating the risk of information loss.\n",
    "\n",
    "The structure of an LSTM cell includes several key components\n",
    "\n",
    "### Input gates: \n",
    "Determine how much new input will be incorporated into the cell state. A sigmoid layer filters the input, and a tangent layer generates a vector of candidate values that can be added to the state.\n",
    "- It takes the first input data and calculates it using Short-Term Memory, Weight, and Bias.\n",
    "- Use the Sigmoid function to update the Long-Term Memory.\n",
    "\n",
    "### Forgetting gate: \n",
    "Controls how long values are retained in the cell state. The sigmoid function determines which parts of the cell state are no longer needed and removes them, allowing the model to forget irrelevant information.\n",
    "Calculate Potential using the existing Short-Term Memory and Long-Term Memory.\n",
    "- Use the Tanh function to calculate the Potential Long-Term Memory value.\n",
    "- Use the Sigmoid function to calculate the Potential Memory to Remember and add it to the existing Long-Term Memory.\n",
    "\n",
    "### Cell State: \n",
    "The core ability of LSTM to retain long-term or short-term information. It is updated by removing information deemed unnecessary by the forgetting gate and adding new information from the input gate. \n",
    "updated by the input gate.\n",
    "\n",
    "### Output gate: \n",
    "Determines what the next hiding state will be. The hidden state contains information about the previous input, which can be used for prediction or as input to the next LSTM cell. The process involves another sigmoid layer that determines which part of the cell state will lead to the output, and a tangent layer that adjusts the cell state to fit the output.\n",
    "- Calculate the memorability for Short-Term Memory the same as in the main block.\n",
    "- Output the new Short-Term Memory and Long-Term Memory.\n",
    "\n",
    "LSTMs process information through these gates and cell states, allowing for a dynamic updating process. Input gates manage the incorporation of new information into the cell state, forgetting gates remove irrelevant information, and output gates determine what information is passed forward. This complex process allows LSTMs to effectively handle short-term and long-term dependencies, making them highly effective for tasks that require understanding long sequences, such as language translation, text generation, and time series analysis.\n",
    "\n",
    "An LSTM is composed of three blocks (input, main, and output).\n",
    "Each block uses Sigmoid and Tanh functions to process and update information.\n",
    "LSTMs are more complex than traditional RNNs, but they excel at learning long sequence data.\n",
    "\n",
    "LSTM(Long Short-Term Memory) 네트워크는 RNN(Recurrent Neural Networks)이 직면한 한계점, 특히 그래디언트 소실 및 폭발 문제를 해결하기 위해 개발되었습니다. RNN과 달리, LSTM은 더 긴 시간 동안 정보를 유지하고 조작할 수 있는 더 복잡한 구조를 따릅니다.\n",
    "\n",
    "LSTM은 전통적인 ReLU(Rectified Linear Unit)에 의존하는 일부 신경망 컨텍스트와 다르게, 시그모이드 및 탄젠트 활성화 함수의 독특한 조합을 사용합니다. 이러한 함수들은 정보의 흐름을 관리하는 데 중요하며, 네트워크가 장기 의존성을 유지하면서 정보 손실의 위험을 완화할 수 있도록 합니다.\n",
    "\n",
    "LSTM 셀의 구조에는 여러 핵심 구성 요소가 포함됩니다:\n",
    "\n",
    "### 입력 게이트: \n",
    "새로운 입력이 셀 상태에 얼마나 통합될지 결정합니다. 시그모이드 계층이 입력을 필터링하고, 탄젠트 계층이 상태에 추가될 수 있는 후보 값 벡터를 생성합니다.\n",
    "- 첫 번째 입력 데이터를 받아 Short-Term Memory와 Weight, Bias를 사용하여 계산합니다.\n",
    "- Sigmoid 함수를 사용하여 Long-Term Memory를 업데이트합니다.\n",
    "\n",
    "### 망각 게이트: \n",
    "셀 상태에 값이 유지될 정도를 제어합니다. 시그모이드 함수를 통해 셀 상태의 어떤 부분이 더 이상 필요하지 않은지 결정하고 제거하여 모델이 관련 없는 정보를 잊게 합니다.\n",
    "기존 Short-Term Memory와 Long-Term Memory를 사용하여 Potential을 계산합니다.\n",
    "- Tanh 함수를 사용하여 Potential Long-Term Memory 값을 계산합니다.\n",
    "- Sigmoid 함수를 사용하여 Potential Memory to Remember를 계산하고 기존 Long-Term Memory에 추가합니다.\n",
    "\n",
    "### 셀 상태: \n",
    "LSTM이 장기 또는 단기 정보를 유지할 수 있는 핵심 능력입니다. 망각 게이트에 의해 필요 없다고 판단된 정보를 제거하고 입력 게이트에서 새로운 정보를 추가하여 \n",
    "업데이트됩니다.\n",
    "\n",
    "### 출력 게이트: \n",
    "다음 은닉 상태가 무엇이 될지 결정합니다. 은닉 상태는 이전 입력에 대한 정보를 포함하고 있으며, 예측에 사용되거나 다음 LSTM 셀의 입력으로 사용될 수 있습니다. 이 과정은 셀 상태의 어떤 부분이 출력으로 이어질지 결정하는 또 다른 시그모이드 계층과, 셀 상태를 출력에 적합하게 조정하는 탄젠트 계층을 포함합니다.\n",
    "- 메인 블록과 동일하게 Short-Term Memory에 대한 기억 가능성을 계산합니다.\n",
    "- 새로운 Short-Term Memory와 Long-Term Memory를 출력합니다.\n",
    "\n",
    "LSTM은 이러한 게이트들과 셀 상태를 통해 정보를 처리하며, 동적인 업데이트 과정을 허용합니다. 입력 게이트는 셀 상태에 새로운 정보를 통합하는 것을 관리하고, 망각 게이트는 관련 없는 정보를 제거하며, 출력 게이트는 앞으로 전달될 정보를 결정합니다. 이 복잡한 과정을 통해 LSTM은 단기 및 장기 의존성을 효과적으로 처리할 수 있으며, 언어 번역, 텍스트 생성, 시계열 분석과 같이 긴 시퀀스를 이해해야 하는 작업에 매우 효과적입니다.\n",
    "\n",
    "LSTM은 3가지 블록 (입력, 메인, 출력)으로 구성됩니다.\n",
    "각 블록은 Sigmoid와 Tanh 함수를 사용하여 정보 처리 및 업데이트를 수행합니다.\n",
    "LSTM은 기존 RNN보다 복잡한 구조이지만, 장기 시퀀스 데이터 학습에 탁월한 성능을 발휘합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6b4d53e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "\n",
      "Now let's compare the observed and predicted values for the first 4 days\n",
      "Company A: observed = 0, predicted = tensor(-0.0166)\n",
      "Company B: observed = 1, predicted = tensor(-0.0202)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\trainer\\connectors\\logger_connector\\logger_connector.py:75: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `lightning.pytorch` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default\n",
      "You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name         | Type | Params\n",
      "--------------------------------------\n",
      "  | other params | n/a  | 12    \n",
      "--------------------------------------\n",
      "12        Trainable params\n",
      "0         Non-trainable params\n",
      "12        Total params\n",
      "0.000     Total estimated model params size (MB)\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\trainer\\connectors\\data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\loops\\fit_loop.py:298: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1999: 100%|██████████| 2/2 [00:00<00:00, 128.58it/s, v_num=7]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=2000` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1999: 100%|██████████| 2/2 [00:00<00:00, 97.28it/s, v_num=7] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Now let's compare the observed and predicted values for the first 4 days\n",
      "Company A: observed = 0, predicted = tensor(0.0017)\n",
      "Company B: observed = 1, predicted = tensor(0.9164)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "Restoring states from the checkpoint path at c:\\Users\\user\\Desktop\\Code\\Dictionary\\lightning_logs\\version_7\\checkpoints\\epoch=1999-step=4000.ckpt\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\callbacks\\model_checkpoint.py:360: The dirpath has changed from 'c:\\\\Users\\\\user\\\\Desktop\\\\Code\\\\Dictionary\\\\lightning_logs\\\\version_7\\\\checkpoints' to 'c:\\\\Users\\\\user\\\\Desktop\\\\Code\\\\Dictionary\\\\lightning_logs\\\\version_8\\\\checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name         | Type | Params\n",
      "--------------------------------------\n",
      "  | other params | n/a  | 12    \n",
      "--------------------------------------\n",
      "12        Trainable params\n",
      "0         Non-trainable params\n",
      "12        Total params\n",
      "0.000     Total estimated model params size (MB)\n",
      "Restored all states from the checkpoint at c:\\Users\\user\\Desktop\\Code\\Dictionary\\lightning_logs\\version_7\\checkpoints\\epoch=1999-step=4000.ckpt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2999: 100%|██████████| 2/2 [00:00<00:00, 133.17it/s, v_num=8]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=3000` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2999: 100%|██████████| 2/2 [00:00<00:00, 99.91it/s, v_num=8] \n",
      "\n",
      "Now let's compare the observed and predicted values for the first 4 days\n",
      "Company A: observed = 0, predicted = tensor(0.0002)\n",
      "Company B: observed = 1, predicted = tensor(0.9556)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.optim import Adam\n",
    "import lightning as L\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "print(torch.cuda.is_available())\n",
    "\n",
    "class myLSTM(L.LightningModule):\n",
    "    def __init__(self):\n",
    "        super().__init__() # get the init from the parent class\n",
    "        # Use Gausian distribution to initialize the weights and biases\n",
    "        mean = torch.tensor(0.0)\n",
    "        std = torch.tensor(1.0)\n",
    "        \n",
    "        self.weight_i1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) # Initialize the weights in the first layer with gaussian distribution, requires_grad is True for optimization\n",
    "        self.weight_i2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) # Initialize the weights in the first layer second weight with gaussian distribution \n",
    "        self.bias_i1 = nn.Parameter(torch.tensor(0.), requires_grad=True) # Initialize the bias in the first layer with 0\n",
    "        \n",
    "        self.weight_p1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)\n",
    "        self.weight_p2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) \n",
    "        self.bias_p1 = nn.Parameter(torch.tensor(0.), requires_grad=True)\n",
    "        \n",
    "        self.weight_sp1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) \n",
    "        self.weight_sp2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) \n",
    "        self.bias_sp1 = nn.Parameter(torch.tensor(0.), requires_grad=True)\n",
    "        \n",
    "        self.weight_o1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) \n",
    "        self.weight_o2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True) \n",
    "        self.bias_o1 = nn.Parameter(torch.tensor(0.), requires_grad=True)\n",
    "\n",
    "    def lstm_unit(self, input_value, long_memory, short_memory):\n",
    "        long_remember_percent = torch.sigmoid((short_memory*self.weight_i1)+(input_value*self.weight_i2)+self.bias_i1) # Calculate the long term remember percentage as input\n",
    "        potential_remember_percent = torch.sigmoid((short_memory*self.weight_p1)+(input_value*self.weight_p2)+self.bias_p1) # Calculate the potential remember percentage as main\n",
    "        potential_memory = torch.tanh((short_memory*self.weight_sp1)+(input_value*self.weight_sp2)+self.bias_sp1) # Calculate the potential memory\n",
    "        updated_long_memory = ((long_memory*long_remember_percent)+(potential_memory*potential_remember_percent)) # Update the long memory\n",
    "        output_percent = torch.sigmoid((short_memory*self.weight_o1)+(input_value*self.weight_o2)+self.bias_o1) # Calculate the output percentage\n",
    "        updated_short_memory = torch.tanh(updated_long_memory) * output_percent # Update the short memory\n",
    "        return( [updated_long_memory, updated_short_memory] ) # Return the updated long and short memory\n",
    "    \n",
    "    def forward(self, input):\n",
    "        long_memory = 0 # Initialize the long memory\n",
    "        short_memory = 0 # Initialize the short memory\n",
    "        day1 = input[0] # Get the input for the first day\n",
    "        day2 = input[1] # Get the input for the second day\n",
    "        day3 = input[2] # Get the input for the third day\n",
    "        day4 = input[3] # Get the input for the fourth day\n",
    "        long_memory, short_memory = self.lstm_unit(day1, long_memory, short_memory) # Run the LSTM unit for the first day\n",
    "        long_memory, short_memory = self.lstm_unit(day2, long_memory, short_memory) # Run the LSTM unit for the second day\n",
    "        long_memory, short_memory = self.lstm_unit(day3, long_memory, short_memory) # Run the LSTM unit for the third day\n",
    "        long_memory, short_memory = self.lstm_unit(day4, long_memory, short_memory) # Run the LSTM unit for the fourth day\n",
    "        return short_memory # Return the short memory\n",
    "    \n",
    "    def configure_optimizers(self):\n",
    "        return Adam(self.parameters(), lr=0.001)\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        input_i, label_i = batch # Get the input and label for the batch\n",
    "        output_i = self.forward(input_i[0]) # Run the forward function\n",
    "        loss = (output_i - label_i)**2 # Calculate the loss with sum of square residuals\n",
    "        \n",
    "        self.log('train_loss', loss) # Log the loss which is part of the lightning module that inherited\n",
    "        \n",
    "        if (label_i == 0):\n",
    "            self.log('out_0', output_i) # Log the output when the label is 0\n",
    "        else:\n",
    "            self.log('out_1', output_i)\n",
    "        return loss\n",
    "    \n",
    "model = myLSTM() # Initialize the model\n",
    "print(\"\\nNow let's compare the observed and predicted values for the first 4 days\")\n",
    "print(\"Company A: observed = 0, predicted =\", model(torch.tensor([0., 0.5, 0.25, 1.])).detach())\n",
    "print(\"Company B: observed = 1, predicted =\", model(torch.tensor([1., 0.5, 0.25, 1.])).detach())\n",
    "\n",
    "inputs = torch.tensor([[0., 0.5, 0.25, 1.], [1., 0.5, 0.25, 1.]]) # Create the input tensor\n",
    "labels = torch.tensor([0., 1.]) # Create the label tensor\n",
    "dataset = TensorDataset(inputs, labels) # Create the dataset\n",
    "dataloader = DataLoader(dataset, batch_size=1, shuffle=True) # Create the dataloader\n",
    "\n",
    "trainer = L.Trainer(max_epochs=2000. log_every_n_steps=2) # Initialize the trainer\n",
    "trainer.fit(model, dataloader) # Train the model\n",
    "print(\"\\nNow let's compare the observed and predicted values for the first 4 days\")\n",
    "print(\"Company A: observed = 0, predicted =\", model(torch.tensor([0., 0.5, 0.25, 1.])).detach())\n",
    "print(\"Company B: observed = 1, predicted =\", model(torch.tensor([1., 0.5, 0.25, 1.])).detach())\n",
    "\n",
    "path_to_best_checkpoint = trainer.checkpoint_callback.best_model_path # Get the path to the best checkpoint\n",
    "trainer = L.Trainer(max_epochs=3000)\n",
    "trainer.fit(model, train_dataloaders=dataloader, ckpt_path=path_to_best_checkpoint) # Train again the model from the best checkpoint\n",
    "print(\"\\nNow let's compare the observed and predicted values for the first 4 days\")\n",
    "print(\"Company A: observed = 0, predicted =\", model(torch.tensor([0., 0.5, 0.25, 1.])).detach())\n",
    "print(\"Company B: observed = 1, predicted =\", model(torch.tensor([1., 0.5, 0.25, 1.])).detach())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ff7f8fb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name | Type | Params\n",
      "------------------------------\n",
      "0 | lstm | LSTM | 16    \n",
      "------------------------------\n",
      "16        Trainable params\n",
      "0         Non-trainable params\n",
      "16        Total params\n",
      "0.000     Total estimated model params size (MB)\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\trainer\\connectors\\data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\loops\\fit_loop.py:298: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1999: 100%|██████████| 2/2 [00:00<00:00, 250.02it/s, v_num=9]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=2000` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1999: 100%|██████████| 2/2 [00:00<00:00, 160.26it/s, v_num=9]\n",
      "\n",
      "Now let's compare the observed and predicted values for the first 4 days\n",
      "Company A: observed = 0, predicted = tensor([1.9175e-06])\n",
      "Company B: observed = 1, predicted = tensor([0.9953])\n"
     ]
    }
   ],
   "source": [
    "class LightningLSTM(L.LightningModule):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.lstm = nn.LSTM(input_size=1, hidden_size=1) # Initialize the LSTM layer hidden size will be number of outputs\n",
    "    def forward(self, input):\n",
    "        input_trans = input.view(len(input),1) # Transposing the input to the in single row. one row per datapoint\n",
    "        lstm_out, temp = self.lstm(input_trans) # Run the LSTM layer. lstm_out contain all outputs of short memory which comes from number of inputs\n",
    "        prediction = lstm_out[-1] # Get the last output of the LSTM layer\n",
    "        return prediction\n",
    "    def configure_optimizers(self):\n",
    "        return Adam(self.parameters(), lr=0.1)\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        input_i, label_i = batch # Get the input and label for the batch\n",
    "        output_i = self.forward(input_i[0]) # Run the forward function\n",
    "        loss = (output_i - label_i)**2 # Calculate the loss with sum of square residuals\n",
    "        \n",
    "        self.log('train_loss', loss) # Log the loss which is part of the lightning module that inherited\n",
    "        \n",
    "        if (label_i == 0):\n",
    "            self.log('out_0', output_i) # Log the output when the label is 0\n",
    "        else:\n",
    "            self.log('out_1', output_i)\n",
    "        return loss\n",
    "\n",
    "model = LightningLSTM() # Initialize the model\n",
    "\n",
    "inputs = torch.tensor([[0., 0.5, 0.25, 1.], [1., 0.5, 0.25, 1.]]) # Create the input tensor\n",
    "labels = torch.tensor([0., 1.]) # Create the label tensor\n",
    "dataset = TensorDataset(inputs, labels) # Create the dataset\n",
    "dataloader = DataLoader(dataset, batch_size=1, shuffle=True) # Create the dataloader\n",
    "\n",
    "trainer = L.Trainer(max_epochs=2000) # Initialize the trainer\n",
    "trainer.fit(model, dataloader) # Train the model\n",
    "print(\"\\nNow let's compare the observed and predicted values for the first 4 days\")\n",
    "print(\"Company A: observed = 0, predicted =\", model(torch.tensor([0., 0.5, 0.25, 1.])).detach())\n",
    "print(\"Company B: observed = 1, predicted =\", model(torch.tensor([1., 0.5, 0.25, 1.])).detach())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29034a25",
   "metadata": {},
   "source": [
    "## 3.8 Word Embedding\n",
    "\n",
    "![WE](Img/Word_Embedding.png)\n",
    "\n",
    "Word embedding techniques represent words in a dense vector space where semantically similar words are mapped to nearby points. Unlike traditional one-hot encoding methods, which represent each word as an isolated unit without capturing semantic relationships, word embeddings encode words into low-dimensional vectors that preserve semantic meaning and context.\n",
    "\n",
    "### How Word Embeddings Work:\n",
    "Word embeddings are learned from the text in an unsupervised manner, typically using neural network models like Word2Vec, GloVe, or FastText. These models are trained to predict a word from its context (surrounding words) or to predict the context given a word, thereby learning representations that capture the meanings, connotations, and relationships among words.\n",
    "\n",
    "Vector Space Representation: Each word is represented as a point in a high-dimensional space. Words with similar meanings are positioned closer together, enabling algorithms to understand word relationships based on vector distances.\n",
    "\n",
    "Dimensionality Reduction: Word embeddings reduce the dimensionality of the word representation, making it more efficient for machine learning models to process language data. Instead of working with sparse vectors of thousands of dimensions (as in one-hot encoding), embeddings are dense with a much lower dimensionality.\n",
    "\n",
    "Semantic and Syntactic Relationships: Beyond capturing semantic similarity, word embeddings can also capture complex relationships between words, such as syntactic roles, analogies, and more. For example, the vector operations can reveal relationships like \"king - man + woman = queen\".\n",
    "\n",
    "### Applications of Word Embeddings:\n",
    "Word embeddings are crucial for various natural language processing (NLP) tasks, including:\n",
    "\n",
    "Text Classification: They enhance the performance of models for sentiment analysis, spam detection, and more by providing a rich representation of text input.\n",
    "\n",
    "Machine Translation: Embeddings provide a foundation for models to understand and translate languages by mapping words across languages to similar points in the vector space.\n",
    "\n",
    "Information Retrieval: They improve search algorithms by enabling more nuanced and context-aware search results based on semantic similarity rather than keyword matching.\n",
    "\n",
    "Question Answering and Chatbots: Embeddings help these systems understand the nuances of human language, improving their ability to respond to queries accurately.\n",
    "\n",
    "In summary, word embeddings represent a significant advancement in the way computers understand human language. By mapping words to vectors of real numbers, embeddings capture the richness of language, including semantics, syntax, and even nuances of meaning, making them indispensable for modern NLP tasks.\n",
    "\n",
    "단어 임베딩 기술은 의미적으로 유사한 단어가 가까운 지점에 매핑되는 고밀도 벡터 공간에서 단어를 표현합니다. 의미 관계를 파악하지 않고 각 단어를 고립된 단위로 표현하는 기존의 원핫 인코딩 방식과 달리, 단어 임베딩은 의미적 의미와 문맥을 보존하는 저차원 벡터로 단어를 인코딩합니다.\n",
    "\n",
    "### 단어 임베딩의 작동 방식:\n",
    "단어 임베딩은 비지도 방식으로 텍스트에서 학습되며, 일반적으로 Word2Vec, GloVe 또는 FastText와 같은 신경망 모델을 사용합니다. 이러한 모델은 문맥(주변 단어)에서 단어를 예측하거나 주어진 단어의 문맥을 예측하도록 학습되어 단어 간의 의미, 의미 및 관계를 포착하는 표현을 학습합니다.\n",
    "\n",
    "벡터 공간 표현: 각 단어는 고차원 공간에서 점으로 표현됩니다. 비슷한 의미를 가진 단어는 서로 가깝게 배치되어 알고리즘이 벡터 거리를 기반으로 단어 관계를 이해할 수 있습니다.\n",
    "\n",
    "차원 감소: 단어 임베딩은 단어 표현의 차원을 줄여 기계 학습 모델이 언어 데이터를 더 효율적으로 처리할 수 있게 해줍니다. 임베딩은 원핫 인코딩에서처럼 수천 차원의 희박한 벡터로 작업하는 대신 훨씬 낮은 차원으로 밀도가 높습니다.\n",
    "\n",
    "의미론적 및 구문론적 관계: 단어 임베딩은 의미적 유사성을 포착하는 것 외에도 구문적 역할, 유추 등 단어 간의 복잡한 관계도 포착할 수 있습니다. 예를 들어, 벡터 연산을 통해 '왕 - 남자 + 여자 = 여왕'과 같은 관계를 파악할 수 있습니다.\n",
    "\n",
    "### 단어 임베딩의 활용:\n",
    "단어 임베딩은 다음과 같은 다양한 자연어 처리(NLP) 작업에 매우 중요합니다:\n",
    "\n",
    "텍스트 분류: 텍스트 입력에 대한 풍부한 표현을 제공함으로써 감정 분석, 스팸 탐지 등을 위한 모델의 성능을 향상시킵니다.\n",
    "\n",
    "기계 번역: 임베딩은 여러 언어의 단어를 벡터 공간의 유사한 지점에 매핑하여 모델이 언어를 이해하고 번역할 수 있는 기반을 제공합니다.\n",
    "\n",
    "정보 검색: 임베딩은 키워드 매칭이 아닌 의미적 유사성을 기반으로 보다 미묘하고 문맥을 인식하는 검색 결과를 제공함으로써 검색 알고리즘을 개선합니다.\n",
    "\n",
    "질문 답변 및 챗봇: 임베딩은 이러한 시스템이 인간 언어의 뉘앙스를 이해하여 쿼리에 정확하게 응답하는 능력을 향상시키는 데 도움이 됩니다.\n",
    "\n",
    "요약하자면, 단어 임베딩은 컴퓨터가 인간의 언어를 이해하는 방식에서 상당한 발전을 이룩한 것입니다. 단어를 실수 벡터에 매핑함으로써 임베딩은 의미론, 구문, 심지어 의미의 뉘앙스까지 포함한 언어의 풍부함을 포착하여 최신 NLP 작업에 없어서는 안 될 필수 요소로 자리 잡았습니다.\n",
    "\n",
    "Translated with DeepL.com (free version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1c227b7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before optimization, the parameters are: \n",
      "input1_w1 tensor(0.1770)\n",
      "input1_w2 tensor(0.4064)\n",
      "input2_w1 tensor(-0.0607)\n",
      "input2_w2 tensor(0.4648)\n",
      "input3_w1 tensor(-0.1868)\n",
      "input3_w2 tensor(0.4411)\n",
      "input4_w1 tensor(0.1979)\n",
      "input4_w2 tensor(0.2793)\n",
      "output1_w1 tensor(0.0856)\n",
      "output1_w2 tensor(0.4840)\n",
      "output2_w1 tensor(0.0861)\n",
      "output2_w2 tensor(0.4803)\n",
      "output3_w1 tensor(0.0061)\n",
      "output3_w2 tensor(0.0942)\n",
      "output4_w1 tensor(0.4842)\n",
      "output4_w2 tensor(0.2281)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGwCAYAAABSN5pGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeJElEQVR4nO3deVyU1eIG8GdmYBj2AZFVFFADLQWUJSoVE4Wyxatd0SiV3MrUlCz1VuLSLyjL3fRmaeWGLbZcU1pI3CJQkfQa4JJoFuDKagyznN8fXCdnAAUcGMDn+/nMJ+Z9zznvOSDNw/ue97wSIYQAEREREelJzd0BIiIiotaGAYmIiIjICAMSERERkREGJCIiIiIjDEhERERERhiQiIiIiIwwIBEREREZsTB3B9oqnU6HP//8E/b29pBIJObuDhERETWAEALl5eXw9PSEVFr/eSIGpCb6888/4e3tbe5uEBERURP8/vvv6NSpU737GZCayN7eHkDNN9jBwcHMvSEiIqKGKCsrg7e3t/5zvD4MSE10/bKag4MDAxIREVEbc6vpMZykTURERGSEAYmIiIjICAMSERERkRHOQSIiIqqDTqdDdXW1ubtBjWRpaQmZTHbb7TAgEbUxPj4+OHv2LBITEzF//nxzd4eoXaqursaZM2eg0+nM3RVqAqVSCXd399tap5ABiaiNCQ4Ohru7+03X7yCiphNCoLCwEDKZDN7e3jddTJBaFyEErl27hgsXLgAAPDw8mtwWAxJRG/PFF180vLAQAFd6J2oUjUaDa9euwdPTEzY2NubuDjWStbU1AODChQtwdXVt8uU2xmKiNsbHxwcSiQTz58+HVqvF3Llz4efnB4VCAWdnZ4T07YvFb70JnP0JyHoPyHwPOH8I0GkAjcrc3Sdq9bRaLQBALpebuSfUVNeDrVqtbnIbPINE1IatXr0aycnJkMlkuPvuu3HtWiWOHTsKu9J8vHTN6K8ml7uAfgnA3cMBCyvzdJioDeFzNtsuU/zseAaJqA07efIkACA+Ph6/ZB/EyXcewuVZ1lg8sI7Cl04AXzwL7H6DZ5KIiG6BAYmoDXvkkUcgkUjw/vvvw8vDDQNfWIPX96rgbH2Tv54OLANOfQ9oNS3WTyKitoYBiagNi46ORnZ2Nv41dw6CXXU4cVmHNw9U4/71laioFvVX3L8MAG9fJroTREZGYsaMGebuRpvDgETUhh09ehQdO3bE/z3/BHY8ARyeZAsAKK4UyL90kwB0/iBQ+kcL9ZKITIVhp+UwIBG1YZ988gm8vb3ROSQafd+rQK81lQAAG0ugq/Mtfr0vn26BHhIRtU0MSERtWP/+/RETEwOdTuC/F3QQAnjQV4ZdcTZQKm5xF4f09pfiJ6KWM27cOOzZswfLly+HRCKBRCJBQUEB9uzZg7CwMFhZWcHDwwNz5syBRlP/HMNvvvkGjo6O2Lx5MwDg999/x8iRI6FUKuHs7IzHH38cBQUFBscdNmwY3n77bXh4eKBDhw54/vnnb+sW+raAt/kTtTE3/o8LAIYMGQJcPAGsDm14IxIJ4NbTtB0joma1fPlynDhxAvfccw8WLlwIoGbNpocffhjjxo3Dxx9/jLy8PEycOBEKhaLORxFt2bIFzz77LLZs2YJHHnkEarUa0dHRiIiIwL59+2BhYYHXX38dMTExOHr0qH4tqN27d8PDwwO7d+/GqVOnEBsbi6CgIEycOLElvwUtigGJqD1w9gU8+wB/ZjesfNdBgJVj8/aJiEzK0dERcrkcNjY2cHd3BwC88sor8Pb2xqpVqyCRSBAQEIA///wTs2fPxrx58wwek7J69Wq88sor+M9//oMBAwYAALZt2wadTof3339fv3bQhg0boFQqkZ6eXvMHGAAnJyesWrUKMpkMAQEBGDp0KNLS0hiQiKgNiJwDbBl563ISKTBgDmDBVYKJ2rrc3FxEREQYLIx4//33o6KiAufPn0fnzp0BAJ999hkuXLiAAwcOIDT077PNv/zyC06dOgV7e3uDdquqqnD69N/zFO+++26DR3Z4eHjg2LFjzTWsVoEBiag9kFkCfgOAIa8D371afzmJFHhsFeB+T83XRHRHCA4ORnZ2NtavX4+QkBB9oKqoqEDfvn3185Fu1LFjR/3XlpaWBvskEgl0uva9VAgDElF7YaEAQicCnsHAviXAbz/WPKwWqJmQ3T0a6P8S4NoDsLQ2b1+JqEnkcrn+WXEA0KNHD3z++ecQQuhDz4EDB2Bvb49OnTrpy3Xt2hXvvPMOIiMjIZPJsGrVKgBAnz59sG3bNri6usLBwaFlB9PK8U9IovbEUgF0vg8YvQV48QTwzHfA+O+BWaeAJ9YDHkEMR0RtmI+PDzIzM1FQUIBLly5hypQp+P333zFt2jTk5eXhq6++QmJiIhISEgzmHwHAXXfdhd27d+Pzzz/Xr6UUFxcHFxcXPP7449i3bx/OnDmD9PR0TJ8+HefPnzfDCFsPBiSi9kYqrTmbZOcKdA4HvMMAG+eaYCTlrzxRWzZr1izIZDL07NkTHTt2hFqtxs6dO5GVlYXAwEA8++yzGD9+PF59te5L7f7+/vjxxx+xdetWvPjii7CxscHevXvRuXNnDB8+HD169MD48eNRVVV1x59RkgghbvI8AqpPWVkZHB0dUVpaesf/IyIiak+qqqpw5swZ+Pr6QqFQmLs71AQ3+xk29PObf04SERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREb4qBEiIqJmUKXWQiqRwEImgUYroBMCCkvZrStSq8AzSERERCZUpdai9C811u37Df949wD6v7Ub/3j3ANbt+w2lf6lRpdbeupFWID09HRKJBCUlJQCADz/8EEql0qx9akkMSERERCZSrdHh44yzCHn9e7zz3Qkc/7MM56/+heN/luGd704g5PXv8XHGWVRrdCY9rkQiuelr/vz5Jj0eAGzfvh2DBw9Gx44d4eDggIiICHz77bcmP4658BIbERGRCVSptfg44yze2Jlbbxm1VuCNnbmQSICn7+1isktuhYWF+q+3bduGefPmIT8/X7/Nzs5O/7UQAlqtFhYWtxcB9u7di8GDB+ONN96AUqnEhg0b8OijjyIzMxPBwcG31XZrwDNIREREJqBSa7H427wGlX0rNQ8qE55Fcnd3178cHR0hkUj07/Py8mBvb49du3ahb9++sLKywv79+6FSqTB9+nS4urpCoVDggQcewMGDBxt8zGXLluHll19GaGgounfvjjfeeAPdu3fHf/7zH5ONy5wYkIiIiG5TlVqLj38+C7W2Yc9/V2sFNmYUtOh8pDlz5iA5ORm5ubno3bs3Xn75ZXz++ef46KOPkJ2djW7duiE6OhpXrlxpUvs6nQ7l5eVwdnY2cc/NgwGJiIjoNkklEqT+t6hRdXb9twhSiaSZelTbwoULMXjwYHTt2hVWVlZYs2YNFi9ejIceegg9e/bEunXrYG1tjQ8++KBJ7b/99tuoqKjAyJEjTdxz8zB7QFq9ejV8fHygUCgQHh6OrKysBtVLSUmBRCLBsGHDDLaPGzeu1uS0mJgYgzJXrlxBXFwcHBwcoFQqMX78eFRUVJhqSEREdIexkElQ+pe6UXXKqtSwkLVcQAoJCdF/ffr0aajVatx///36bZaWlggLC0Nubv1zqOqzZcsWLFiwAJ988glcXV1N0l9zM2tA2rZtGxISEpCYmIjs7GwEBgYiOjoaFy5cuGm9goICzJo1C/369atzf0xMDAoLC/WvrVu3GuyPi4vD8ePH8f3332PHjh3Yu3cvJk2aZLJxERHRnUWjFXC0tmxUHQeFJTQNvCRnCra2ts3SbkpKCiZMmIBPPvkEUVFRzXIMczBrQFqyZAkmTpyI+Ph49OzZE2vXroWNjQ3Wr19fbx2tVou4uDgsWLAAfn5+dZaxsrIymLDm5OSk35ebm4vU1FS8//77CA8PxwMPPICVK1ciJSUFf/75p8nHSERE7Z9OCMTc496oOg/d4w6daLmAdKOuXbtCLpfjwIED+m1qtRoHDx5Ez549G9zO1q1bER8fj61bt2Lo0KHN0VWzMVtAqq6uxuHDhw3SplQqRVRUFDIyMuqtt3DhQri6umL8+PH1lklPT4erqyv8/f3x3HPP4fLly/p9GRkZUCqVBqcao6KiIJVKkZmZWW+bKpUKZWVlBi8iIiIAUFjKMObeLrBs4CUzS5kET0f4mG1lbVtbWzz33HN46aWXkJqail9//RUTJ07EtWvXbvr5eqMtW7ZgzJgxeOeddxAeHo6ioiIUFRWhtLS0mXvfMswWkC5dugStVgs3NzeD7W5ubigqqnui2/79+/HBBx9g3bp19bYbExODjz/+GGlpaXjzzTexZ88ePPTQQ9Bqa+4UKCoqqnV91MLCAs7OzvUeFwCSkpLg6Oiof3l7ezd0qEREdAewspThpeiABpWdHRMAKwvzTgNOTk7GiBEj8PTTT6NPnz44deoUvv32W4OrLjfz3nvvQaPR4Pnnn4eHh4f+9cILLzRzz1tGm1kosry8HE8//TTWrVsHFxeXesuNGjVK/3WvXr3Qu3dvdO3aFenp6Rg0aFCTjz937lwkJCTo35eVlTEkERGRnsJShnH3+UAiqVnnqK5b/i1lErwcE4AxET6QN1NAGjduHMaNG6d/HxkZCVHHpTyFQoEVK1ZgxYoVdbZjXM+43fT0dFN1uVUyW0BycXGBTCZDcXGxwfbi4mK4u9e+jnv69GkUFBTg0Ucf1W/T6WoW2bKwsEB+fj66du1aq56fnx9cXFxw6tQpDBo0CO7u7rUmgWs0Gly5cqXO415nZWUFKyurRo2RiIjuLHILKZ6+twtGhnhjY0YBdv23CGVVajgoLPHQPe54OsIHVhbSZgtHZDpmC0hyuRx9+/ZFWlqa/lZ9nU6HtLQ0TJ06tVb5gIAAHDt2zGDbq6++ivLycixfvrzesznnz5/H5cuX4eHhAQCIiIhASUkJDh8+jL59+wIAfvzxR+h0OoSHh5twhEREdCdSWMqgsJRhQj8/TOrfFRYyCTRaAZ0QZptzRI1n1ktsCQkJGDt2LEJCQhAWFoZly5ahsrIS8fHxAIAxY8bAy8sLSUlJUCgUuOeeewzqX3+q8PXtFRUVWLBgAUaMGAF3d3ecPn0aL7/8sn51UADo0aMHYmJiMHHiRKxduxZqtRpTp07FqFGj4Onp2XKDJyKidu3GMCS3aLn1jsg0zBqQYmNjcfHiRcybNw9FRUUICgpCamqqfuL2uXPnIJU2/DSkTCbD0aNH8dFHH6GkpASenp4YMmQIFi1aZHB5bPPmzZg6dSoGDRoEqVSKESNG1HsNloiIiO48ElHXzC26pbKyMjg6OqK0tBQODg7m7g4REZlIVVUVzpw5A19fXygUCnN3h5rgZj/Dhn5+c5ZYC/Dx8YFEIsH8+fNr7bv+aBQfH58W7xcRERHVjQHJSHV1tVmPr9Vpodaq67wlk4iIiFpGuw1IV69eRWxsLGxsbNC5c2esWbMGkZGRkEgkiIyMBAD9w2zfeustDB8+HHZ2dvpnspWWluKFF15Aly5dIJfL0alTJyQkJODatWsGx4mJiYGrqyvkcjkcHBzQr18/7Nq1C0DNM+MkEgnOnj0LAFiwYIH+mHU5dvEYNhzfgPePvY8dv+2ASqvCX5q/muk7RERERPVpMwtFNtaECROwfft2AICNjQ1eeumlesu+9tprUCgU8PX1hVwuR3V1NSIjI5GTkwOFQoEePXrgxIkTWLp0KX755Rf88MMP+rqHDh2Ct7c3OnXqhJMnT2L//v147LHHcOjQIbi6uiI8PBxHjhxBdXU1vLy80KlTJ4Nja0XNCt8Xrl3AkzufNNj3+s+vY1i3YUgISYCl1BJSSbvNs0RE7Y+6CpBIAZkFoNUAQgdYck5TW9EuP3FPnz6tD0ezZs1CXl4eDh06BJVKVWd5Pz8/FBQU4NixY1izZg22bt2KnJwcyOVyHD16FL/88gt+/vlnADVrJv3444/6uqdOncLp06eRnZ2Nc+fOwd7eHhqNBp999hk8PDzw888/69dgmjBhAn7++Wd9W1WaKlzUXgQACEXtS2rXNNewJW8Lnkl9Bmqd2nTfICIiaj7qv4C/SoCMlcD7g4DlQTX/zVhZs13dPFcGIiMjMWPGjGZp+07ULs8gHT9+XP/1yJEjAdQsNNm7d29kZ2fXKj927Fj9s2dkMhmysrIA1MxHuuuuu2qV//nnnxEaGgoAmDJlCrKysnD58mX9yt4A8Oeff96yn+fKz+EXzS8AAMsOlvWWO3rpKJIzk/Fy2MuwtrC+ZbtERGQmGhVw8H0gbQGgNfrDtugosOdNYFAiEDYJsDDt0xm2b98OS8v6P0uocdplQGos4wfmXieXyxEcHFxr+40P8vvmm29gYWGBXr16QaFQ6C+nXX84bn3+0vyF9cfWw9Kl5h/zzQISAOz4bQdeCq3/MiEREZmZ+q+acPTdq/WX0ar/t18ChI4HLE33R6+zs7PJ2qJ2eontxhW3v/jiCwBAXl4ejh49Wmd540nT188OabVavPvuu/rLYunp6XjppZfw5JNP4sqVK/ryCxcuRE5ODlJSUuqcgG1jYwMAqKys/PuYkOC7s9/BprsNur/RHR2iO9x0TFXaKnx9+mtodJqbliMiIjPRVNWcOWqItPk1Z5tM6MZLbO+++y66d+8OhUIBNzc3PPHEEyY91p2gXQYkPz8/DB8+HACQlJSEHj16ICQkBHK5vEH1R48ejd69e0Or1SI0NBT33HMP/P39oVQq8cQTT6CkpMTgLFJiYiJ69eqFPn36wMKi9km5gIAAAMCKFSsQGhqK+Ph4XK26CrVOjeJPi3HyXydR8GbBLft1rvwcAxIRUWukrgKy3q99Wa0+WnXN2SZ1lcm7cujQIUyfPh0LFy5Efn4+UlNT0b9/f5Mfp71rlwEJAN5//33885//hLW1NcrLy5GcnIyePXsCAKytb35K08rKCnv27MH06dPh7e2NEydO4OrVqwgJCcH//d//wc3NTX+mqE+fPpDJZNBqtdi8eTNcXFxqtff666/j3nvvhVQqxaFDh3Ds2DHIpI1/YKGFhFdEiYhaJYkUyP26cXVyv66pZ2Lnzp2Dra0tHnnkEXTp0gXBwcGYPn26yY/T3rXbT9yKigp8/PHH+iXGT58+rb/VPygoCABuuhijUqnE8uXLsXz58jr3q9U1fyXs3r3bYKnygoKCWmV79uyJjIwMg20anQbOCmdgItBpYqdadeoS7BoMuaxhZ8GIiKgFySyAqtLG1akqralnYoMHD0aXLl3g5+eHmJgYxMTE4B//+Id+ugc1TLs9g/T555+jU6dOiI6ORkxMDAIDA1FVVQU3NzdMmzbN3N2DRqfBP7r9o8HlXaxd0K9TP66FRETUGmk1gMKxcXUUjjX1TMze3h7Z2dnYunUrPDw8MG/ePAQGBqKkpMTkx2rP2u2nba9evdC1a1f8/PPPSEtLg5OTE+Lj45GZmQlPT09zdw8KCwXG3j0WDvKGPeh2Yq+JnH9ERNRaCR3Q87HG1enxWE29ZmBhYYGoqCi89dZbOHr0KAoKCgzW8KNba7eX2AYNGoTMzExzd+OmbC1tsW7IOkz8biLKqsvqLTe251g8cdcTvLxGRNRaWSqA0Ak16xw1ZKK2zLKmfDOsrL1jxw789ttv6N+/P5ycnLBz507odDr4+/ub/FjtWbs9g9QWyGVydFV2xVfDvkJcjzjYW9rr90kgQYRHBP4d9W9M7zOd4YiIqLWzUNQsAtkQg+abfKHI65RKJbZv344HH3wQPXr0wNq1a7F161bcfffdzXK89koi+Nj4JikrK4OjoyNKS0sNJmk31V+avyCTyPBHxR9Q69RwsXaBraUtn8FGRNTCqqqqcObMGfj6+upv9GkwjQrIWlezzlFdZ5JkljXhKGxiswUkuvnPsKGf3+32Eltbc/0RIr6OvmbuCRERNZmFVc0K2cFP1axzlPt1zd1qCseaOUehE2rKMBy1egxIREREpmRpXfOKmArcN73mVn6tpmZCdjPMOaLmwYBERETUHG4MQxacR9rWcHILERERkREGJCIiIiIjDEhERERERhiQiIiIiIwwIBEREREZ4V1sREREzUClUUEqkUImlUGr00IndLDi+kdtBgMSERGRCVVpqlCtrUZKXgq+P/c9yqvLYS+3x+DOgzEqYBTkMjkUFlwPqbXjJTYiIiITuR6MBnwyACtzViLvSh7+qPgDeVfysDJnJQZ8MgApeSmo1labu6smlZ6eDolEgpKSklr7IiMjMWPGjBbv0+1iQCIiIjKBKk0VtuRuwTuH34FGp6mzjEanwTuH38HWvK2o0lS1cA9rq65uX0HNlBiQiIiITEClVWH5keUNKrsse1mznEUqLy9HXFwcbG1t4eHhgaVLlxqcwfHx8cGiRYswZswYODg4YNKkSQCA/fv3o1+/frC2toa3tzemT5+OyspKfbsbN25ESEgI7O3t4e7ujieffBIXLlwAABQUFGDgwIEAACcnJ0gkEowbN87kY2tpDEhERES3SaVRISUvpd4zR8Y0Og225W+DSqMyaT8SEhJw4MABfP311/j++++xb98+ZGdnG5R5++23ERgYiCNHjuC1117D6dOnERMTgxEjRuDo0aPYtm0b9u/fj6lTp+rrqNVqLFq0CL/88gu+/PJLFBQU6EOQt7c3Pv/8cwBAfn4+CgsLsXx5w4Jia8ZJ2kRERLdJKpHih3M/NKrO92e/x9i7x5qsD+Xl5fjoo4+wZcsWDBo0CACwYcMGeHp6GpR78MEH8eKLL+rfT5gwAXFxcfqzTN27d8eKFSswYMAArFmzBgqFAs8884y+vJ+fH1asWIHQ0FBUVFTAzs4Ozs7OAABXV1colUqTjcmceAaJiIjoNsmkMpRXlzeqTnl1OSykpjtP8dtvv0GtViMsLEy/zdHREf7+/gblQkJCDN7/8ssv+PDDD2FnZ6d/RUdHQ6fT4cyZMwCAw4cP49FHH0Xnzp1hb2+PAQMGAADOnTtnsv63NjyDREREdJu0Oi3s5faNqmMvt4dGp4FcJm+mXtXN1tbW4H1FRQUmT56M6dOn1yrbuXNnVFZWIjo6GtHR0di8eTM6duyIc+fOITo6ul1P8mZAIiIiuk06ocPgzoORdyWvwXUGdxkMIYTJ+uDn5wdLS0scPHgQnTt3BgCUlpbixIkT6N+/f731+vTpg19//RXdunWrc/+xY8dw+fJlJCcnw9vbGwBw6NAhgzJyeU3I02q1teqnp6c3ZThmZ/ZLbKtXr4aPjw8UCgXCw8ORlZXVoHopKSmQSCQYNmyYfptarcbs2bPRq1cv2NrawtPTE2PGjMGff/5pUNfHxwcSicTglZycbMphERHRHcTKwgqxAbENvmRmIbVArH+sSVfWtre3x9ixY/HSSy9h9+7dOH78OMaPHw+pVAqJRFJvvdmzZ+Onn37C1KlTkZOTg5MnT+Krr77ST9Lu3Lkz5HI5Vq5cid9++w1ff/01Fi1aZNBGly5dIJFIsGPHDly8eBEVFRX6fYMGDUJSUpLJxtlSzBqQtm3bhoSEBCQmJiI7OxuBgYGIjo7W3zpYn4KCAsyaNQv9+vUz2H7t2jVkZ2fjtddeQ3Z2NrZv3478/Hw89thjtdpYuHAhCgsL9a9p06aZdGxERHRnsZJZ4YXgFxpUdmafmc1yaW3JkiWIiIjAI488gqioKNx///3o0aMHFIr6V+7u3bs39uzZgxMnTqBfv34IDg7GvHnz9JO7O3bsiA8//BCffvopevbsieTkZLz99tsGbXh5eWHBggWYM2cO3NzcDO6AO336NIqLi00+1uYmEaY8v9dI4eHhCA0NxapVqwAAOp0O3t7emDZtGubMmVNnHa1Wi/79++OZZ57Bvn37UFJSgi+//LLeYxw8eBBhYWE4e/as/pSjj48PZsyYcVsre5aVlcHR0RGlpaVwcHBocjtERNS6VFVV4cyZM/D19b1psKhLtbYaW/O2Yln2sjpv+beQWmBGnxkYHTC6ReYeVVZWwsvLC++88w7Gjx/f7MdrLW72M2zo57fZziBVV1fj8OHDiIqK+rszUimioqKQkZFRb72FCxfC1dW1wT/o0tJSSCSSWrcdJicno0OHDggODsbixYuh0dx87QqVSoWysjKDFxER0Y3kMjli/WOxZ+QeTA+ejh7OPdDJrhN6OPfA9ODp2DNyD2L9Y5stHB05cgRbt27F6dOnkZ2djbi4OADA448/3izHa8/MNkn70qVL0Gq1cHNzM9ju5uaGvLy6J7nt378fH3zwAXJychp0jKqqKsyePRujR482SInTp09Hnz594OzsjJ9++glz585FYWEhlixZUm9bSUlJWLBgQYOOS0REdy6FhQIKCwXG9ByDsXePhYXUAhqdBkIIk845qs/bb7+N/Px8yOVy9O3bF/v27YOLi0uzH7e9aTN3sZWXl+Ppp5/GunXrGvSDVqvVGDlyJIQQWLNmjcG+hIQE/de9e/eGXC7H5MmTkZSUBCuruv/xzp0716BeWVmZfjY/ERGRsRvDUEvdyh8cHIzDhw+3yLHaO7MFJBcXF8hksloTt4qLi+Hu7l6r/OnTp1FQUIBHH31Uv02n0wEALCwskJ+fj65duwL4OxydPXsWP/744y3nCIWHh0Oj0aCgoKDWglrXWVlZ1RueiIiIqH0x2xyk66f+0tLS9Nt0Oh3S0tIQERFRq3xAQACOHTuGnJwc/euxxx7DwIEDkZOToz+bcz0cnTx5Ej/88AM6dOhwy77k5ORAKpXC1dXVdAMkIiKiNsusl9gSEhIwduxYhISEICwsDMuWLUNlZSXi4+MBAGPGjIGXlxeSkpKgUChwzz33GNS/PvH6+na1Wo0nnngC2dnZ2LFjB7RaLYqKigAAzs7OkMvlyMjIQGZmJgYOHAh7e3tkZGRg5syZeOqpp+Dk5NRygyciIqJWy6wBKTY2FhcvXsS8efNQVFSEoKAgpKam6idunzt3DlJpw09y/fHHH/j6668BAEFBQQb7du/ejcjISFhZWSElJQXz58+HSqWCr68vZs6caTC/iIiIiO5sZl0HqS3jOkhERO3T7ayDRK2DKdZBajN3sREREbUlOpWq5hEfFhaApuY2fylv9mkzGJCIiIhMSFdVBVFdjSubt6D8u++gKyuD1MEB9kOGwDnuSUjkckjb4Zmp9PR0DBw4EFevXq21OHNbZPaH1RIREbUXuupqXN2yFSfufwCXli+HKjcX6j/+gCo3F5eWL8eJ+x/A1S1boauubrY+FBUV4YUXXkC3bt2gUCjg5uaG+++/H2vWrMG1a9ea7bimEBkZeVuPATMlnkEiIiIyAV1VFa5u2YoLb71VfyG1uma/BHAaPdrkZ5J+++033H///VAqlXjjjTfQq1cvWFlZ4dixY3jvvffg5eVV5wPcqTaeQSIiIjIBoVLhwtKlDSp7YclSiGY4izRlyhRYWFjg0KFDGDlyJHr06AE/Pz88/vjj+Oabb/Doo4/imWeewSOPPGJQT61Ww9XVFR988AGAmjM506ZNw4wZM+Dk5AQ3NzesW7dOvxSPvb09unXrhl27dtXbl2vXruGhhx7C/fffj5KSEly+fBmjR4+Gl5cXbGxs0KtXL2zdulVffty4cdizZw+WL18OiUQCiUSCgoICaLVajB8/Hr6+vrC2toa/vz+WL19u8u+dMQYkIiKi26RTqXBlyxZArW5YBbW65lKbSmWyPly+fBnfffcdnn/+edja2tZZRiKRYMKECUhNTUVhYaF++44dO3Dt2jXExsbqt3300UdwcXFBVlYWpk2bhueeew7//Oc/cd999yE7OxtDhgzB008/Xedlu5KSEgwePBg6nQ7ff/89lEolqqqq0LdvX3zzzTf473//i0mTJuHpp59GVlYWAGD58uWIiIjAxIkTUVhYiMLCQnh7e0On06FTp0749NNP8euvv2LevHn417/+hU8++cRk37u6MCARERHdJolEgvLvvm9UnfLvvgMkEpP14dSpUxBC1HpklouLC+zs7GBnZ4fZs2fjvvvug7+/PzZu3Kgvs2HDBvzzn/+EnZ2dfltgYCBeffVVdO/eHXPnzoVCoYCLiwsmTpyI7t27Y968ebh8+TKOHj1qcLyioiIMGDAAHh4e+M9//gMbGxsAgJeXF2bNmoWgoCD4+flh2rRpiImJ0QcdR0dHyOVy2NjYwN3dHe7u7pDJZLC0tMSCBQsQEhICX19fxMXFIT4+ngGJiIio1bOwgK6srFFVtGVlkFg0/1TgrKws5OTk4O6774bqf2esJkyYgA0bNgCoeQbqrl278MwzzxjU6927t/5rmUyGDh06oFevXvpt1xd1vnDhgkG9wYMHo1u3bti2bRvk8r8f0qvVarFo0SL06tULzs7OsLOzw7fffotz587dcgyrV69G37590bFjR9jZ2eG9995rUL3bwYBERER0uzQaSBu5aLDMwQFCozFZF7p16waJRIL8/HyD7X5+fujWrRusra3128aMGYPffvsNGRkZ2LRpE3x9fdGvXz+DepaWlgbvJRKJwTbJ/85+XX9w/HVDhw7F3r178euvvxpsX7x4MZYvX47Zs2dj9+7dyMnJQXR0NKpvMRcrJSUFs2bNwvjx4/Hdd98hJycH8fHxt6x3u3gXGxER0W0SQsB+yBCocnMbXMd+yBDAhA+z6NChAwYPHoxVq1Zh2rRp9c5Dul522LBh2LBhAzIyMvTPQDWF5ORk2NnZYdCgQUhPT0fPnj0BAAcOHMDjjz+Op556CkBNsDpx4oR+P1DzIHutVmvQ3oEDB3DfffdhypQp+m2nT582WX/rwzNIREREt0lqZQXnJ0cDRmdd6mVpCacnR5t8Ze13330XGo0GISEh2LZtG3Jzc5Gfn49NmzYhLy8PMplMX3bChAn46KOPkJubi7Fjx5q0H2+//Tbi4uLw4IMPIi8vDwDQvXt3fP/99/jpp5+Qm5uLyZMno7i42KCej48PMjMzUVBQgEuXLkGn06F79+44dOgQvv32W5w4cQKvvfYaDh48aNL+1oUBiYiIyAQkVlZwnTmzQWVdExIguWF+jql07doVR44cQVRUFObOnYvAwECEhIRg5cqVmDVrFhYtWqQvGxUVBQ8PD0RHR8PT09PkfVm6dClGjhyJBx98ECdOnMCrr76KPn36IDo6GpGRkXB3d8ewYcMM6syaNQsymQw9e/ZEx44dce7cOUyePBnDhw9HbGwswsPDcfnyZYOzSc2FD6ttIj6sloiofbqdh9XqqqtxdfNmXFiytO5b/i0t4ZowE05xcZA2Q0BqjIqKCnh5eWHDhg0YPny4WftianxYLRERUSsilcvhNHo0lCNG4OqWrSj/7jtoy8og+9+z2JyeHF3zLDYzhiOdTodLly7hnXfegVKp5Mra9WBAIiIiMiGpQgEoFHCOHwfnZ+IhsbCouVtNCJPPOWqKc+fOwdfXF506dcKHH34IixZYaqAt4neFiIioGdwYhppjvlFT+fj4gLNrbo2TtImIiIiM8AwSERFRHdrbWZby8nL9IpK9evWClZUVzpw5g8uXL8Pe3r7WI0raMlP87BiQiIiIbnB9raDq6mqD1aeb29GjR2+5OrSnp2ez3JJ/o/Pnz6O8vBwqlQparRZyuRyOjo7w8PCotbp2a3X9Abq3018GJCIiohtYWFjAxsYGFy9ehKWlJaTSlpmNYmVlpQ9narUamv89hkShUOgf6wHU3MJ+nU6na3D/bgxfKpUKQgj9qtU6nU7fblFRkb4/18teuHABpaWl6NatG3Q3nJyRSmDQN3MTQuDatWu4cOEClEqlwcKYjcWAREREdAOJRAIPDw+cOXMGZ8+ebdFjXw87f/31F0pLSwEAXl5ekMlkKCoqwu+//w5bW1vIZDJUVFRAKpXCy8sLQgiUl5ejoqICarUaEokEVlZWcHR01K8DVFVVhUuXLgGoCT8WFha4dOkSKisrUV5eru+DWq2Gvb09ZDIZZDIZysrK9GdkVGoN1EIGnRCQSgBruQXkspqA1JqCklKphLu7+221wYBERERkRC6Xo3v37s3+QNT6rFq1CqtWrQIApKWlwcvLC4mJicjKyoJcLocQAr6+vpBIJPjqq6/w6quv4rPPPgMAdOnSBaWlpSgpKYGFhQXWr1+PsLAwZGVl4dlnnzVo87333sMXX3yBsLAwfPzxx3X2JW13Ot5Z/BYAwPWJRFg6eRjs93a2QcLg7vB3c4CVZdPP2JiKpaXlbZ05uo4BiYiIqA5SqbTRK2mbSkVFhf7slUwmg0KhwIULF3D27FnI5XJkZWUhMDAQWq0WBQUFWLJkCYQQeOGFF7Bs2TKUlpYiMDAQp0+fxmuvvYY9e/ZAp9PVavPq1as4e/YsfHx86hzr5ZIyrFq9BmfPnoWVV0/AwhUoN3yY7B/l5Xhy/RFsiA9FqI8zFK0gJJkCb/MnIiJqQwYOHIjAwEAANUHn8OHD+ru2nnzySQCAo6MjHn74YQDAoUOHmnScixcv4uHoITiR+19YOHeCy7A59ZbV6ASmbMpG67nIdvsYkIiIiNoQNze3Zj9Gfn4+7r33XmRlZULu6Q/3uDdhYed80zrlKg22H/kD1Rpds/evJTAgERERtSHGk6H79u2r37ZlyxYAQGlpKXbu3AkACAkJaVT7e/fuxX333YfffvsN3n0Gwn10EmQ2jg2q+8nB3xt1rNaMc5CIiIjasK5du+KZZ57BBx98gOXLl+Obb77BlStXcOXKFVhYWGDBggWNam/w4MGorq6GRCJB6cVCVG2Zq9/neP8o2HQNrbduUVkV5Bbt49wLAxIREVEb9+9//xsBAQFYv349Tp8+DSsrK0RFRWHevHno169fo9q6fueeEAJlv+cZ7NNdK71pXSsLKbQ6AZm07c9Gkoj2tpZ6CykrK4OjoyNKS0vh4OBg7u4QERGZ1F/VGrz1bT42HChocJ3YUG8kPtoTNvLWe/6loZ/f7eM8GBEREZmUtdwCE/v5NarO5P5+rTocNQYDEhEREdXJxc4KT/Tt1KCyD93jDm9nm2buUcthQCIiIqI6yS2kSBreCw/dc/PHdjwY4Irlo4JhKWs/saJ9nAcjIiKiZmEpk2LF6GBk/nYZ7+87gz0nL0IIQCIB7uvaARMe8MMD3V3aVTgCGJCIiIjoFixlUtzXzQWhPs6QSCS4Vq2BtbzmkSKWUimk7eCuNWMMSERERHRLUolE/zBauYXczL1pfmY/H7Z69Wr9Q/LCw8ORlZXVoHopKSmQSCQYNmyYwXYhBObNmwcPDw9YW1sjKioKJ0+eNChz5coVxMXFwcHBAUqlEuPHj0dFRYWphkRERERtnFkD0rZt25CQkIDExERkZ2cjMDAQ0dHRuHDhwk3rFRQUYNasWXUufvXWW29hxYoVWLt2LTIzM2Fra4vo6GhUVVXpy8TFxeH48eP4/vvvsWPHDuzduxeTJk0y+fiIiIiobTLrQpHh4eEIDQ3FqlWrAAA6nQ7e3t6YNm0a5syp+6nBWq0W/fv3xzPPPIN9+/ahpKQEX375JYCas0eenp548cUXMWvWLAA1z6Nxc3PDhx9+iFGjRiE3Nxc9e/bEwYMH9c+nSU1NxcMPP4zz58/D09OzzuOqVCqoVCr9+7KyMnh7e3OhSCIiojak1S8UWV1djcOHDyMqKurvzkiliIqKQkZGRr31Fi5cCFdXV4wfP77WvjNnzqCoqMigTUdHR4SHh+vbzMjIgFKpNHh4X1RUFKRSKTIzM+s9blJSEhwdHfUvb2/vRo2XiIiI2g6zBaRLly5Bq9XCzc3NYLubmxuKiorqrLN//3588MEHWLduXZ37r9e7WZtFRUVwdXU12G9hYQFnZ+d6jwsAc+fORWlpqf71++/t54nFREREZKjN3MVWXl6Op59+GuvWrYOLi0uLH9/KygpWVlYtflwiIiJqeWYLSC4uLpDJZCguLjbYXlxcDHf32it2nj59GgUFBXj00Uf123Q6HYCaM0D5+fn6esXFxfDw8DBoMygoCADg7u5eaxK4RqPBlStX6jwuERER3XnMdolNLpejb9++SEtL02/T6XRIS0tDRERErfIBAQE4duwYcnJy9K/HHnsMAwcORE5ODry9veHr6wt3d3eDNsvKypCZmalvMyIiAiUlJTh8+LC+zI8//gidTofw8PBmHDERERG1FWa9xJaQkICxY8ciJCQEYWFhWLZsGSorKxEfHw8AGDNmDLy8vJCUlASFQoF77rnHoL5SqQQAg+0zZszA66+/ju7du8PX1xevvfYaPD099esl9ejRAzExMZg4cSLWrl0LtVqNqVOnYtSoUfXewUZERER3FrMGpNjYWFy8eBHz5s1DUVERgoKCkJqaqp9kfe7cOUiljTvJ9fLLL6OyshKTJk1CSUkJHnjgAaSmpkKhUOjLbN68GVOnTsWgQYMglUoxYsQIrFixwqRjIyIiorbLrOsgtWUNXUeBiIiIWo9Wvw4SERERUWvFgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkxOwBafXq1fDx8YFCoUB4eDiysrLqLbt9+3aEhIRAqVTC1tYWQUFB2Lhxo0EZiURS52vx4sX6Mj4+PrX2JycnN9sYiYiIqG2xMOfBt23bhoSEBKxduxbh4eFYtmwZoqOjkZ+fD1dX11rlnZ2d8corryAgIAByuRw7duxAfHw8XF1dER0dDQAoLCw0qLNr1y6MHz8eI0aMMNi+cOFCTJw4Uf/e3t6+GUZIREREbZFECCHMdfDw8HCEhoZi1apVAACdTgdvb29MmzYNc+bMaVAbffr0wdChQ7Fo0aI69w8bNgzl5eVIS0vTb/Px8cGMGTMwY8aMJve9rKwMjo6OKC0thYODQ5PbISIiopbT0M/vRl9i27lzJyZMmICXX34ZeXl5BvuuXr2KBx98sEHtVFdX4/Dhw4iKivq7M1IpoqKikJGRccv6QgikpaUhPz8f/fv3r7NMcXExvvnmG4wfP77WvuTkZHTo0AHBwcFYvHgxNBrNTY+nUqlQVlZm8CIiIqL2qVEBacuWLXjsscdQVFSEjIwMBAcHY/Pmzfr91dXV2LNnT4PaunTpErRaLdzc3Ay2u7m5oaioqN56paWlsLOzg1wux9ChQ7Fy5UoMHjy4zrIfffQR7O3tMXz4cIPt06dPR0pKCnbv3o3JkyfjjTfewMsvv3zT/iYlJcHR0VH/8vb2btA4iYiIqO1p1BykxYsXY8mSJZg+fToA4JNPPsEzzzyDqqqqOs/SNAd7e3vk5OSgoqICaWlpSEhIgJ+fHyIjI2uVXb9+PeLi4qBQKAy2JyQk6L/u3bs35HI5Jk+ejKSkJFhZWdV53Llz5xrUKysrY0giIiJqpxoVkE6ePIlHH31U/37kyJHo2LEjHnvsMajVavzjH/9ocFsuLi6QyWQoLi422F5cXAx3d/d660mlUnTr1g0AEBQUhNzcXCQlJdUKSPv27UN+fj62bdt2y76Eh4dDo9GgoKAA/v7+dZaxsrKqNzwRERFR+9KoS2wODg61As3AgQOxY8cOvPTSS1i5cmWD25LL5ejbt6/B5GmdToe0tDREREQ0uB2dTgeVSlVr+wcffIC+ffsiMDDwlm3k5ORAKpXWeeccERER3XkadQYpLCwMu3btwr333muwfcCAAfjPf/6DRx55pFEHT0hIwNixYxESEoKwsDAsW7YMlZWViI+PBwCMGTMGXl5eSEpKAlAzDygkJARdu3aFSqXCzp07sXHjRqxZs8ag3bKyMnz66ad45513ah0zIyMDmZmZGDhwIOzt7ZGRkYGZM2fiqaeegpOTU6P6T0RERO1TowLSzJkz8dNPP9W5LzIyEv/5z3/w8ccfN7i92NhYXLx4EfPmzUNRURGCgoKQmpqqn7h97tw5SKV/n+SqrKzElClTcP78eVhbWyMgIACbNm1CbGysQbspKSkQQmD06NG1jmllZYWUlBTMnz8fKpUKvr6+mDlzpsH8IiIiIrqzNWkdpDFjxmDgwIHo378/unbt2hz9avW4DhIREVHb02zrIAE184eSkpLQvXt3eHt746mnnsL777+PkydPNrnDRERERK3Fba2k/ccff2Dv3r3Ys2cP9uzZgxMnTsDDwwPnz583ZR9bJZ5BIiIianua9QzSdU5OTujQoQOcnJygVCphYWGBjh073k6TRERERGbXpID0r3/9C/fddx86dOiAOXPmoKqqCnPmzEFRURGOHDli6j4SERERtagmXWKTSqXo2LEjZs6cieHDh+Ouu+5qjr61arzERkRE1PY09PO7Ubf5X3fkyBHs2bMH6enpeOeddyCXyzFgwABERkYiMjLyjgxMRERE1H7c1iTt63755RcsXboUmzdvhk6ng1arNUXfWjWeQSIiImp7mvUMkhACR44cQXp6OtLT07F//36UlZWhd+/eGDBgQJM7TURERNQaNCkgOTs7o6KiAoGBgRgwYAAmTpyIfv36QalUmrh7RERERC2vSQFp06ZN6NevHy8tERERUbvUpIA0dOhQU/eDiIiIqNW4rYUiiYiIiNojBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkxOwBafXq1fDx8YFCoUB4eDiysrLqLbt9+3aEhIRAqVTC1tYWQUFB2Lhxo0GZcePGQSKRGLxiYmIMyly5cgVxcXFwcHCAUqnE+PHjUVFR0SzjIyIiorbHrAFp27ZtSEhIQGJiIrKzsxEYGIjo6GhcuHChzvLOzs545ZVXkJGRgaNHjyI+Ph7x8fH49ttvDcrFxMSgsLBQ/9q6davB/ri4OBw/fhzff/89duzYgb1792LSpEnNNk4iIiJqWyRCCGGug4eHhyM0NBSrVq0CAOh0Onh7e2PatGmYM2dOg9ro06cPhg4dikWLFgGoOYNUUlKCL7/8ss7yubm56NmzJw4ePIiQkBAAQGpqKh5++GGcP38enp6eDTpuWVkZHB0dUVpaCgcHhwbVISIiIvNq6Oe32c4gVVdX4/Dhw4iKivq7M1IpoqKikJGRccv6QgikpaUhPz8f/fv3N9iXnp4OV1dX+Pv747nnnsPly5f1+zIyMqBUKvXhCACioqIglUqRmZlZ7/FUKhXKysoMXkRERNQ+WZjrwJcuXYJWq4Wbm5vBdjc3N+Tl5dVbr7S0FF5eXlCpVJDJZHj33XcxePBg/f6YmBgMHz4cvr6+OH36NP71r3/hoYceQkZGBmQyGYqKiuDq6mrQpoWFBZydnVFUVFTvcZOSkrBgwYImjpaIiIjaErMFpKayt7dHTk4OKioqkJaWhoSEBPj5+SEyMhIAMGrUKH3ZXr16oXfv3ujatSvS09MxaNCgJh937ty5SEhI0L8vKyuDt7d3k9sjIiKi1stsAcnFxQUymQzFxcUG24uLi+Hu7l5vPalUim7dugEAgoKCkJubi6SkJH1AMubn5wcXFxecOnUKgwYNgru7e61J4BqNBleuXLnpca2srGBlZdXA0REREVFbZrY5SHK5HH379kVaWpp+m06nQ1paGiIiIhrcjk6ng0qlqnf/+fPncfnyZXh4eAAAIiIiUFJSgsOHD+vL/Pjjj9DpdAgPD2/CSIiIiKi9MesltoSEBIwdOxYhISEICwvDsmXLUFlZifj4eADAmDFj4OXlhaSkJAA184BCQkLQtWtXqFQq7Ny5Exs3bsSaNWsAABUVFViwYAFGjBgBd3d3nD59Gi+//DK6deuG6OhoAECPHj0QExODiRMnYu3atVCr1Zg6dSpGjRrV4DvYiIiIqH0za0CKjY3FxYsXMW/ePBQVFSEoKAipqan6idvnzp2DVPr3Sa7KykpMmTIF58+fh7W1NQICArBp0ybExsYCAGQyGY4ePYqPPvoIJSUl8PT0xJAhQ7Bo0SKDy2ObN2/G1KlTMWjQIEilUowYMQIrVqxo2cETERFRq2XWdZDaMq6DRERE1Pa0+nWQiIiIiForBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREYYkIiIiIiMMCARERERGWFAIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBiQiIiIiIwxIREREREbMHpBWr14NHx8fKBQKhIeHIysrq96y27dvR0hICJRKJWxtbREUFISNGzfq96vVasyePRu9evWCra0tPD09MWbMGPz5558G7fj4+EAikRi8kpOTm22MRERE1LaYNSBt27YNCQkJSExMRHZ2NgIDAxEdHY0LFy7UWd7Z2RmvvPIKMjIycPToUcTHxyM+Ph7ffvstAODatWvIzs7Ga6+9huzsbGzfvh35+fl47LHHarW1cOFCFBYW6l/Tpk1r1rESERFR2yERQghzHTw8PByhoaFYtWoVAECn08Hb2xvTpk3DnDlzGtRGnz59MHToUCxatKjO/QcPHkRYWBjOnj2Lzp07A6g5gzRjxgzMmDGjwX1VqVRQqVT692VlZfD29kZpaSkcHBwa3A4RERGZT1lZGRwdHW/5+W22M0jV1dU4fPgwoqKi/u6MVIqoqChkZGTcsr4QAmlpacjPz0f//v3rLVdaWgqJRAKlUmmwPTk5GR06dEBwcDAWL14MjUZz0+MlJSXB0dFR//L29r5lH4mIiKhtsjDXgS9dugStVgs3NzeD7W5ubsjLy6u3XmlpKby8vKBSqSCTyfDuu+9i8ODBdZatqqrC7NmzMXr0aIOUOH36dPTp0wfOzs746aefMHfuXBQWFmLJkiX1Hnfu3LlISEjQv79+BomIiIjaH7MFpKayt7dHTk4OKioqkJaWhoSEBPj5+SEyMtKgnFqtxsiRIyGEwJo1awz23Rh0evfuDblcjsmTJyMpKQlWVlZ1HtfKyqrefURERNS+mC0gubi4QCaTobi42GB7cXEx3N3d660nlUrRrVs3AEBQUBByc3ORlJRkEJCuh6OzZ8/ixx9/vOUcofDwcGg0GhQUFMDf37/pgyIiIqJ2wWxzkORyOfr27Yu0tDT9Np1Oh7S0NERERDS4HZ1OZzB5+no4OnnyJH744Qd06NDhlm3k5ORAKpXC1dW1cYMgIiKidsmsl9gSEhIwduxYhISEICwsDMuWLUNlZSXi4+MBAGPGjIGXlxeSkpIA1EyUDgkJQdeuXaFSqbBz505s3LhRfwlNrVbjiSeeQHZ2Nnbs2AGtVouioiIANUsEyOVyZGRkIDMzEwMHDoS9vT0yMjIwc+ZMPPXUU3BycjLPN4KIiIhaFbMGpNjYWFy8eBHz5s1DUVERgoKCkJqaqp+4fe7cOUilf5/kqqysxJQpU3D+/HlYW1sjICAAmzZtQmxsLADgjz/+wNdffw2g5vLbjXbv3o3IyEhYWVkhJSUF8+fPh0qlgq+vL2bOnGkwL4mIiIjubGZdB6kta+g6CkRERNR6tPp1kIiIiIhaKwYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiNmD0irV6+Gj48PFAoFwsPDkZWVVW/Z7du3IyQkBEqlEra2tggKCsLGjRsNygghMG/ePHh4eMDa2hpRUVE4efKkQZkrV64gLi4ODg4OUCqVGD9+PCoqKpplfERERNT2mDUgbdu2DQkJCUhMTER2djYCAwMRHR2NCxcu1Fne2dkZr7zyCjIyMnD06FHEx8cjPj4e3377rb7MW2+9hRUrVmDt2rXIzMyEra0toqOjUVVVpS8TFxeH48eP4/vvv8eOHTuwd+9eTJo0qdnHS0RERG2DRAghzHXw8PBwhIaGYtWqVQAAnU4Hb29vTJs2DXPmzGlQG3369MHQoUOxaNEiCCHg6emJF198EbNmzQIAlJaWws3NDR9++CFGjRqF3Nxc9OzZEwcPHkRISAgAIDU1FQ8//DDOnz8PT0/POo+jUqmgUqn078vKyuDt7Y3S0lI4ODjczreBiIiIWkhZWRkcHR1v+flttjNI1dXVOHz4MKKiov7ujFSKqKgoZGRk3LK+EAJpaWnIz89H//79AQBnzpxBUVGRQZuOjo4IDw/Xt5mRkQGlUqkPRwAQFRUFqVSKzMzMeo+XlJQER0dH/cvb27vRYyYiIqK2wWwB6dKlS9BqtXBzczPY7ubmhqKionrrlZaWws7ODnK5HEOHDsXKlSsxePBgANDXu1mbRUVFcHV1NdhvYWEBZ2fnmx537ty5KC0t1b9+//33hg+WiIiI2hQLc3egsezt7ZGTk4OKigqkpaUhISEBfn5+iIyMbNbjWllZwcrKqlmPQURERK2D2QKSi4sLZDIZiouLDbYXFxfD3d293npSqRTdunUDAAQFBSE3NxdJSUmIjIzU1ysuLoaHh4dBm0FBQQAAd3f3WpPANRoNrly5ctPjEhER0Z3DbJfY5HI5+vbti7S0NP02nU6HtLQ0RERENLgdnU6nnzzt6+sLd3d3gzbLysqQmZmpbzMiIgIlJSU4fPiwvsyPP/4InU6H8PDw2x0WERERtQNmvcSWkJCAsWPHIiQkBGFhYVi2bBkqKysRHx8PABgzZgy8vLyQlJQEoGaidEhICLp27QqVSoWdO3di48aNWLNmDQBAIpFgxowZeP3119G9e3f4+vritddeg6enJ4YNGwYA6NGjB2JiYjBx4kSsXbsWarUaU6dOxahRo+q9g42IiIjuLGYNSLGxsbh48SLmzZuHoqIiBAUFITU1VT/J+ty5c5BK/z7JVVlZiSlTpuD8+fOwtrZGQEAANm3ahNjYWH2Zl19+GZWVlZg0aRJKSkrwwAMPIDU1FQqFQl9m8+bNmDp1KgYNGgSpVIoRI0ZgxYoVLTdwIiIiatXMug5SW9bQdRSIiIio9Wj16yARERERtVYMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIAQEFBASQSCSQSCdLT083dHbNiQCIiImolqqqqsHTpUtx3331QKpWwsrJC586dERUVhSVLlpi7e42Snp6uD1sFBQXm7k6jWZi7A0RERARcvnwZgwYNwi+//AIAsLGxwV133YXy8nLs2bMHaWlpSEhIMHMvW44QAhKJxGzH5xkkIiKiVmDq1Kn6cPTCCy/g8uXLOHbsGAoKCnDp0iVs2LABaWlp+rMyJ0+e1NdduXIlJBIJlEolqqqqMH/+fH25Xbt24a677oKtrS3i4uJQWVmJ119/HR07doSHhwcSExNv2q8VK1ZAIpFAJpNh48aNAIA5c+bg7rvvhlKphKWlJTw9PTF27FgUFhYCAObPn4+BAwfq2/D19YVEIsG4ceMAAEuXLkVQUBCcnZ1haWmJjh074h/DhiH/119RlZuHK5u34OrGTajIyIDQaqFTqUz5rW4YQU1SWloqAIjS0lJzd4WIiNq4q1evCplMJgCIwMBAodVq6yyn0+nEXXfdJQCIuXPn6rcPGDBAABCTJk0SQgiRmJgoAAgAws7OTvj7++vf9+jRQ1hbWws/Pz/9ttTUVCGEEGfOnNFv2717t/jggw+ERCIRMplMbNq0SX+8wMBA4ejoKO655x4REBAgJBKJACBCQ0OFEEKsW7dO9OjRQ99WUFCQCA8PFwsXLhRCCPH4448LW1tb0aNHD3HPPffox+6uUIgj3e8Sv/oH6F8n+g8QF9etE7rqapN8rxv6+c2A1EQMSEREZCqZmZn6MDF16lT99scff1y/HYDYsGGDWLJkiQAgvLy8hEajEcXFxUIqlQoAYv/+/UIIw4B0Pdjcf//9+m379+8XWq1WdOnSRQAQs2fPFkIYBqRnn31WSKVSIZPJxJYtWwz6e/ToUYMQt27dOn29U6dOCSGE2L17t37bmTNnDOofP35cVP8v8GirqsTnCQn6sh908jYISNdfv0+fbpKQ1NDPb15iIyIiakWk0r8/mv39/REYGGiwf9y4cbC2tsYff/yBb7/9Fl9++SV0Oh26deuG+++/v1Z7jz76KADAx8cHAODk5IT7778fUqkUXbp0AQAUFxfXqrd27VrodDosWbIEo0ePNtiXk5OD0NBQ2NnZQSKRYOLEifp9f/755y3HePbsWQwcOBAODg6wsLbGiBsmoF/UaOqsU/7td7i8fgN0VVW3bN8UGJCIiIjMzN/fHzKZDADw008/6be/+eabSElJMSjr5OSEUaNGAQA2bNiAzz//HAAwZsyYOtt2cHAAAFhYWBi8B6CfBC2EqFXPzs4OALBmzRpcunRJv33//v0YO3YssrOzoVAoEBoaih49euj3a7Xam471t99+w7Bhw3DgwAEAQK8OLgiwsvq7Pmr35bqrmzdBYtEy95cxIBEREZmZo6MjRo4cCQA4dOgQEhMTbxo0nnvuOQDA119/jd27d0MikeDpp582aZ9Wr14NT09P5OXl4eGHH0ZFRQUAIDMzUx+ojh07hqysrDrDmY2Njf7ryspK/ddHjhxBdXU1AOCbTz9FSseOmODcoUF90ly4iMobAmRzYkAiIiJqBVauXInevXsDABYuXAhnZ2cEBwcjMjKyVtnQ0FD07dsX1dXVUKvV6N+/v/4Smql07twZu3btgqOjIw4ePIhhw4ZBpVLp+wgAvXr1Qo8ePbB48eJa9bt27QpLS0sAQFRUFO6991589tlnuPvuu/Vnyx554gk8fvo0/u9C7Ut89anKPwFxi7NUpsCARERE1Ap06NABP//8M95880307dsXOp0OeXl5sLa2RnR0NNauXYthw4bpy0+ZMkX/dX2X125X79698cUXX0AulyMtLQ2jR4/Ggw8+iDfffBOenp7466+/EBAQgDVr1tQ5nhUrVsDb2xvFxcXIzMxEUVERAgICsH79evj6+qJarYaTTIbFHp4N7pNEJgNaYH0kiajrwiPdUllZGRwdHVFaWmpwPZeIiKgl/Pzzz4iIiICtrS0KCwthb29v7i41mrayEifvjYBQqxtcp8vWLbAJDm7yMRv6+c0zSERERG1Ibm4unnzySTzxxBMAgMmTJ7fJcAQAEqkU9tFDGlxe7usD6xsu8TUnBiQiIqI2pLi4GFu3bkVpaSlGjRqF119/3dxdajKptTVcnnsOkv/NVbqVDpMnt8j8I6AVBKTVq1fDx8cHCoUC4eHhyMrKqrfsunXr0K9fPzg5OcHJyQlRUVG1yl9fWt34deMEMh8fn1r7k5OTm22MREREphIZGQkhBMrLy7F161ZYW1ubu0u3xdLLC15LlwC3CEkdJk+Gw8MPQyqXt0i/zBqQtm3bhoSEBCQmJiI7OxuBgYGIjo7GhQsX6iyfnp6O0aNHY/fu3cjIyIC3tzeGDBmCP/74Q1+msLDQ4LV+/XpIJBKMGDHCoK2FCxcalJs2bVqzjpWIiIhqkyoUsH3gAfh+/hnsH3qoVlCyCQuF93v/hsuU51osHAFmnqQdHh6O0NBQrFq1CgCg0+ng7e2NadOmYc6cObesr9Vq4eTkhFWrVtU7g3/YsGEoLy9HWlqafpuPjw9mzJiBGTNmNLnvnKRNRERkWrqqKojqaqh++w3Q6mDZyQsyJydIZLKau9dMoNVP0q6ursbhw4cRFRX1d2ekUkRFRSEjI6NBbVy7dg1qtRrOzs517i8uLsY333yD8ePH19qXnJyMDh06IDg4GIsXL4amnqXNr1OpVCgrKzN4ERERkelIFQrIHBxgExQEm759YOnmBqlcbrJw1Bgts153HS5dugStVgs3NzeD7W5ubsjLy2tQG7Nnz4anp6dByLrRRx99BHt7ewwfPtxg+/Tp09GnTx84Ozvjp59+wty5c1FYWIglNzwLxlhSUhIWLFjQoH4RERFR22a2gHS7kpOTkZKSgvT0dCgUijrLrF+/HnFxcbX2JyQk6L/u3bs35HI5Jk+ejKSkJFjd8DyYG82dO9egXllZGby9vU0wEiIiImptzBaQXFxcIJPJaj1BuLi4GO7u7jet+/bbbyM5ORk//PCDwZLnN9q3bx/y8/Oxbdu2W/YlPDwcGo0GBQUF8Pf3r7OMlZVVveGJiIiI2hezzUGSy+Xo27evweRpnU6HtLQ0RERE1FvvrbfewqJFi5CamoqQkJB6y33wwQfo27cvAgMDb9mXnJwcSKVSuLq6Nm4QRERE1C6Z9RJbQkICxo4di5CQEISFhWHZsmWorKxEfHw8gJpny3h5eSEpKQkA8Oabb2LevHnYsmULfHx8UFRUBACws7ODnZ2dvt2ysjJ8+umneOedd2odMyMjA5mZmRg4cCDs7e2RkZGBmTNn4qmnnoKTk1MLjJqIiIhaO7MGpNjYWFy8eBHz5s1DUVERgoKCkJqaqp+4fe7cOUilf5/kWrNmDaqrq/XLq1+XmJiI+fPn69+npKRACIHRo0fXOqaVlRVSUlIwf/58qFQq+Pr6YubMmQbzi4iIiOjOxofVNhHXQSIiImp7Wv06SEREREStVZu9zd/crp9444KRREREbcf1z+1bXUBjQGqi8vJyAOBaSERERG1QeXk5HB0d693POUhNpNPp8Oeff8Le3h4SicRg3/VFJH///fd2PT/pThknwLG2R3fKOIE7Z6x3yjgBjvV2CCFQXl4OT09PgxvBjPEMUhNJpVJ06tTppmUcHBza/T9c4M4ZJ8Cxtkd3yjiBO2esd8o4AY61qW525ug6TtImIiIiMsKARERERGSEAakZWFlZITExsd0/u+1OGSfAsbZHd8o4gTtnrHfKOAGOtSVwkjYRERGREZ5BIiIiIjLCgERERERkhAGJiIiIyAgDEhEREZERBqQmuHLlCuLi4uDg4AClUonx48ejoqLipuWnTZsGf39/WFtbo3Pnzpg+fTpKS0sNyp07dw5Dhw6FjY0NXF1d8dJLL0Gj0TT3cG6qsWMFgPfeew+RkZFwcHCARCJBSUlJrTI+Pj6QSCQGr+Tk5GYaxa011zib0m5za0qfqqqq8Pzzz6NDhw6ws7PDiBEjUFxcbFDG+OcpkUiQkpLSnEOpZfXq1fDx8YFCoUB4eDiysrJuWv7TTz9FQEAAFAoFevXqhZ07dxrsF0Jg3rx58PDwgLW1NaKionDy5MnmHEKDmHqc48aNq/Wzi4mJac4hNFhjxnr8+HGMGDFC//+XZcuW3XabLcXU45w/f36tn2lAQEAzjqDhGjPWdevWoV+/fnBycoKTkxOioqJqlW+231NBjRYTEyMCAwPFzz//LPbt2ye6desmRo8eXW/5Y8eOieHDh4uvv/5anDp1SqSlpYnu3buLESNG6MtoNBpxzz33iKioKHHkyBGxc+dO4eLiIubOndsSQ6pXY8cqhBBLly4VSUlJIikpSQAQV69erVWmS5cuYuHChaKwsFD/qqioaKZR3FpzjbMp7Ta3pvTp2WefFd7e3iItLU0cOnRI3HvvveK+++4zKANAbNiwweBn+tdffzXnUAykpKQIuVwu1q9fL44fPy4mTpwolEqlKC4urrP8gQMHhEwmE2+99Zb49ddfxauvviosLS3FsWPH9GWSk5OFo6Oj+PLLL8Uvv/wiHnvsMeHr69ui4zLWHOMcO3asiImJMfjZXblypaWGVK/GjjUrK0vMmjVLbN26Vbi7u4ulS5fedpstoTnGmZiYKO6++26Dn+nFixebeSS31tixPvnkk2L16tXiyJEjIjc3V4wbN044OjqK8+fP68s01+8pA1Ij/frrrwKAOHjwoH7brl27hEQiEX/88UeD2/nkk0+EXC4XarVaCCHEzp07hVQqFUVFRfoya9asEQ4ODkKlUpluAI1wu2PdvXv3TQNSXb/U5tBc4zTVvxVTakqfSkpKhKWlpfj000/123JzcwUAkZGRod8GQHzxxRfN1vdbCQsLE88//7z+vVarFZ6eniIpKanO8iNHjhRDhw412BYeHi4mT54shBBCp9MJd3d3sXjxYv3+kpISYWVlJbZu3doMI2gYU49TiJqA9PjjjzdLf29HY8d6o/r+H3M7bTaX5hhnYmKiCAwMNGEvTeN2v/8ajUbY29uLjz76SAjRvL+nvMTWSBkZGVAqlQgJCdFvi4qKglQqRWZmZoPbKS0thYODAywsLPTt9urVC25ubvoy0dHRKCsrw/Hjx003gEYw1Vjrk5ycjA4dOiA4OBiLFy822+XE5hpnc3//WqpPhw8fhlqtRlRUlH5bQEAAOnfujIyMDIOyzz//PFxcXBAWFob169dDtNAya9XV1Th8+LBBH6VSKaKiomr18bqMjAyD8kDN79z18mfOnEFRUZFBGUdHR4SHh9fbZnNrjnFel56eDldXV/j7++O5557D5cuXTT+ARmjKWM3R5u1qzj6dPHkSnp6e8PPzQ1xcHM6dO3e73b0tphjrtWvXoFar4ezsDKB5f0/5sNpGKioqgqurq8E2CwsLODs7o6ioqEFtXLp0CYsWLcKkSZMM2r0xHAHQv29ou6ZmirHWZ/r06ejTpw+cnZ3x008/Ye7cuSgsLMSSJUtuq92maK5xNuf3ryX7VFRUBLlcDqVSabDdzc3NoM7ChQvx4IMPwsbGBt999x2mTJmCiooKTJ8+3eTjMHbp0iVotdo6f4fy8vLqrFPf79z1MV3/783KtLTmGCcAxMTEYPjw4fD19cXp06fxr3/9Cw899BAyMjIgk8lMP5AGaMpYzdHm7WquPoWHh+PDDz+Ev78/CgsLsWDBAvTr1w///e9/YW9vf7vdbhJTjHX27Nnw9PTUB6Lm/D1lQPqfOXPm4M0337xpmdzc3Ns+TllZGYYOHYqePXti/vz5t91eU7TUWG8mISFB/3Xv3r0hl8sxefJkJCUlmWw5+dYwzpbSGsb62muv6b8ODg5GZWUlFi9e3CIBiW7PqFGj9F/36tULvXv3RteuXZGeno5BgwaZsWfUVA899JD+6969eyM8PBxdunTBJ598gvHjx5uxZ02XnJyMlJQUpKenQ6FQNPvxGJD+58UXX8S4ceNuWsbPzw/u7u64cOGCwXaNRoMrV67A3d39pvXLy8sRExMDe3t7fPHFF7C0tNTvc3d3rzUz//pdQrdqt7FaYqyNFR4eDo1Gg4KCAvj7+5ukTXOPsyW/f805Vnd3d1RXV6OkpMTgLFJxcfFNxxEeHo5FixZBpVI1+zOUXFxcIJPJat1Zd7M+uru737T89f8WFxfDw8PDoExQUJAJe99wzTHOuvj5+cHFxQWnTp0yW0BqyljN0ebtaqk+KZVK3HXXXTh16pTJ2mys2xnr22+/jeTkZPzwww/o3bu3fnuz/p7e1gymO9D1Sa6HDh3Sb/v2229vOfG2tLRU3HvvvWLAgAGisrKy1v7rk7RvnMn/73//Wzg4OIiqqirTDqKBmjrW6242SdvYpk2bhFQqNcudM801zttttzk0pU/XJ2l/9tln+m15eXm1Jmkbe/3114WTk5PpOn8LYWFhYurUqfr3Wq1WeHl53XTy8iOPPGKwLSIiotYk7bffflu/v7S0tFVM0jblOOvy+++/C4lEIr766ivTdLqJGjvWG91sknZT22wuzTFOY+Xl5cLJyUksX778drp625oy1jfffFM4ODjU+f+b5vw9ZUBqgpiYGBEcHCwyMzPF/v37Rffu3Q1ukz5//rzw9/cXmZmZQoiaH1Z4eLjo1auXOHXqlMFtlxqNRgjx923+Q4YMETk5OSI1NVV07NixVdzm35ixCiFEYWGhOHLkiFi3bp0AIPbu3SuOHDkiLl++LIQQ4qeffhJLly4VOTk54vTp02LTpk2iY8eOYsyYMS0+vuuaY5wNadccmjLWZ599VnTu3Fn8+OOP4tChQyIiIkJERETo93/99ddi3bp14tixY+LkyZPi3XffFTY2NmLevHktNq6UlBRhZWUlPvzwQ/Hrr7+KSZMmCaVSqb8z9OmnnxZz5szRlz9w4ICwsLAQb7/9tsjNzRWJiYl13uavVCrFV199JY4ePSoef/zxVnGbvynHWV5eLmbNmiUyMjLEmTNnxA8//CD69OkjunfvbrY/zq5r7FhVKpU4cuSIOHLkiPDw8BCzZs0SR44cESdPnmxwm+bQHON88cUXRXp6ujhz5ow4cOCAiIqKEi4uLuLChQstPr4bNXasycnJQi6Xi88++8zgs7O8vNygTHP8njIgNcHly5fF6NGjhZ2dnXBwcBDx8fEGP6wzZ84IAGL37t1CiL/PMNT1OnPmjL5eQUGBeOihh4S1tbVwcXERL774on4ZAHNp7FiFqLm9tK6xbtiwQQghxOHDh0V4eLhwdHQUCoVC9OjRQ7zxxhtm/Z9xc4yzIe2aQ1PG+tdff4kpU6YIJycnYWNjI/7xj3+IwsJC/f5du3aJoKAgYWdnJ2xtbUVgYKBYu3at0Gq1LTk0sXLlStG5c2chl8tFWFiY+Pnnn/X7BgwYIMaOHWtQ/pNPPhF33XWXkMvl4u677xbffPONwX6dTidee+014ebmJqysrMSgQYNEfn5+Swzlpkw5zmvXrokhQ4aIjh07CktLS9GlSxcxceJEswaGGzVmrNf/7Rq/BgwY0OA2zcXU44yNjRUeHh5CLpcLLy8vERsbK06dOtWCI6pfY8bapUuXOseamJioL9Ncv6cSIVroPlwiIiKiNoLrIBEREREZYUAiIiIiMsKARERERGSEAYmIiIjICAMSERERkREGJCIiIiIjDEhERERERhiQiIiIiIwwIBEREREZYUAiIqpHVVUVxo0bh169esHCwgLDhg0zd5eIqIUwIBER1UOr1cLa2hrTp09HVFSUubtDRC2IAYmI7ig7duyAUqmEVqsFAOTk5EAikWDOnDn6MhMmTMBTTz0FW1tbrFmzBhMnToS7u7u5ukxEZsCARER3lH79+qG8vBxHjhwBAOzZswcuLi5IT0/Xl9mzZw8iIyPN00EiahUYkIjojuLo6IigoCB9IEpPT8fMmTNx5MgRVFRU4I8//sCpU6cwYMAA83aUiMyKAYmI7jgDBgxAeno6hBDYt28fhg8fjh49emD//v3Ys2cPPD090b17d3N3k4jMyMLcHSAiammRkZFYv349fvnlF1haWiIgIACRkZFIT0/H1atXefaIiHgGiYjuPNfnIS1dulQfhq4HpPT0dM4/IiIGJCK68zg5OaF3797YvHmzPgz1798f2dnZOHHihMEZpF9//RU5OTm4cuUKSktLkZOTg5ycHPN0nIhaDC+xEdEdacCAAcjJydEHJGdnZ/Ts2RPFxcXw9/fXl3v44Ydx9uxZ/fvg4GAAgBCiRftLRC1LIvhbTkRERGSAl9iIiIiIjDAgERERERlhQCIiIiIywoBEREREZIQBiYiIiMgIAxIRERGREQYkIiIiIiMMSERERERGGJCIiIiIjDAgERERERlhQCIiIiIy8v9J5Ac9pYMBAwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.optim import Adam\n",
    "from torch.distributions.uniform import Uniform\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "import lightning as L\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# one hot encoding for the words.\n",
    "# example: Troll2 is great! Gymkata\n",
    "inputs = torch.tensor([[1.,0.,0.,0.], [0.,1.,0.,0.], [0.,0.,1.,0.], [0.,0.,0.,1.]]) # Create the input tensor\n",
    "labels = torch.tensor([[0.,1.,0.,0.], [0.,0.,1.,0.], [0.,0.,0.,1.], [0.,1.,0.,0.]]) # Create the label tensor\n",
    "dataset = TensorDataset(inputs, labels) # Create the dataset\n",
    "dataloader = DataLoader(dataset, batch_size=1, shuffle=True) # Create the dataloader. for batch, shuffle and for debugging\n",
    "\n",
    "class WordEmbeddingFromScratch(L.LightningModule):\n",
    "    def __init__(self):\n",
    "        super().__init__() # get the init from the parent class which comes from the lightning module\n",
    "        min_value = -0.5 # minimum value for the uniform distribution\n",
    "        max_value = 0.5 # maximum value for the uniform distribution\n",
    "        \n",
    "        self.input1_w1 = nn.Parameter(Uniform(min_value, max_value).sample()) # Initialize the first weight for the first input with uniform distribution\n",
    "        self.input1_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input2_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input2_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input3_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input3_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input4_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.input4_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        \n",
    "        self.output1_w1 = nn.Parameter(Uniform(min_value, max_value).sample()) # Initialize the first weight for the first output with uniform distribution\n",
    "        self.output1_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output2_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output2_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output3_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output3_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output4_w1 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        self.output4_w2 = nn.Parameter(Uniform(min_value, max_value).sample())\n",
    "        \n",
    "        self.loss = nn.CrossEntropyLoss() # Initialize the loss function\n",
    "        \n",
    "    def forward(self, input):\n",
    "        input = input[0]\n",
    "        inputs_to_top_hidden = ((input[0] * self.input1_w1) + \n",
    "                                (input[1] * self.input2_w1) +\n",
    "                                (input[2] * self.input3_w1) +\n",
    "                                (input[3] * self.input4_w1)) # Calculate the inputs to the top hidden layer\n",
    "        inputs_to_bottom_hidden = ((input[0] * self.input1_w2) + \n",
    "                                   (input[1] * self.input2_w2) +\n",
    "                                   (input[2] * self.input3_w2) +\n",
    "                                   (input[3] * self.input4_w2)) # Calculate the inputs to the bottom hidden layer\n",
    "        output1 = ((inputs_to_top_hidden * self.output1_w1) + (inputs_to_bottom_hidden * self.output1_w2)) # Calculate the output for the first word\n",
    "        output2 = ((inputs_to_top_hidden * self.output2_w1) + (inputs_to_bottom_hidden * self.output2_w2))\n",
    "        output3 = ((inputs_to_top_hidden * self.output3_w1) + (inputs_to_bottom_hidden * self.output3_w2))\n",
    "        output4 = ((inputs_to_top_hidden * self.output4_w1) + (inputs_to_bottom_hidden * self.output4_w2))\n",
    "        output_presoftmax = torch.stack([output1, output2, output3, output4]) # Stack the outputs torch.stack makes backpropagation possible\n",
    "        return (output_presoftmax)\n",
    "    \n",
    "    def configure_optimizers(self):\n",
    "        return Adam(self.parameters(), lr=0.1)\n",
    "    \n",
    "    def training_step(self, batch, batch_idx):\n",
    "        input_i, label_i = batch # Get the input and label for the batch\n",
    "        output_i = self.forward(input_i) # Run the forward function\n",
    "        loss = self.loss(output_i, label_i[0]) # Calculate the loss with cross entropy loss\n",
    "        return loss\n",
    "    \n",
    "modelFromScratch = WordEmbeddingFromScratch() # Initialize the model\n",
    "print(\"Before optimization, the parameters are: \")\n",
    "for name, param in modelFromScratch.named_parameters():\n",
    "    print(name, param.data)\n",
    "data = {'w1': [modelFromScratch.input1_w1.item(),\n",
    "               modelFromScratch.input2_w1.item(),\n",
    "               modelFromScratch.input3_w1.item(),\n",
    "               modelFromScratch.input4_w1.item()],\n",
    "        'w2': [modelFromScratch.input1_w2.item(),\n",
    "               modelFromScratch.input2_w2.item(),\n",
    "               modelFromScratch.input3_w2.item(),\n",
    "               modelFromScratch.input4_w2.item()],\n",
    "        'token': ['Troll2', 'is', 'great!', 'Gymkata'],\n",
    "        'input': ['input1', 'input2', 'input3', 'input4']}\n",
    "df = pd.DataFrame(data)\n",
    "df\n",
    "sns.scatterplot(data=df, x='w1', y='w2', hue='token', s=100) # Plot the weights\n",
    "plt.text(df.w1[0], df.w2[0], df.token[0], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[1], df.w2[1], df.token[1], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[2], df.w2[2], df.token[2], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[3], df.w2[3], df.token[3], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "be926ccd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name         | Type             | Params\n",
      "--------------------------------------------------\n",
      "0 | loss         | CrossEntropyLoss | 0     \n",
      "  | other params | n/a              | 16    \n",
      "--------------------------------------------------\n",
      "16        Trainable params\n",
      "0         Non-trainable params\n",
      "16        Total params\n",
      "0.000     Total estimated model params size (MB)\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\trainer\\connectors\\data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.\n",
      "c:\\Users\\user\\anaconda3\\envs\\general\\lib\\site-packages\\lightning\\pytorch\\loops\\fit_loop.py:298: The number of training batches (4) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1:  50%|█████     | 2/4 [00:00<00:00, 285.78it/s, v_num=12]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 99: 100%|██████████| 4/4 [00:00<00:00, 333.37it/s, v_num=12]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=100` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 99: 100%|██████████| 4/4 [00:00<00:00, 235.32it/s, v_num=12]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAGwCAYAAAAHVnkYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDq0lEQVR4nO3deVxVdf7H8fdhuVwQuIiCooKCmlsupGhqJm5pU6nT3lguqW2mmVlqvymzmrTUzDZrWrQpNZs2y8bMKcE0d0UdcxfCfU1AlO3e8/uDvBPjEiB4uNzX8/G4j7znfM85Hy7Weff9fu/3GKZpmgIAAIBlfKwuAAAAwNsRyAAAACxGIAMAALAYgQwAAMBiBDIAAACLEcgAAAAsRiADAACwmJ/VBVxOLpdLBw4cUEhIiAzDsLocAABQDKZpKisrS7Vq1ZKPT+XsS/KqQHbgwAFFR0dbXQYAACiFvXv3qk6dOlaXUS68KpCFhIRIKvyFhoaGWlwNAAAojszMTEVHR7vv45WRVwWys8OUoaGhBDIAADxMZZ5uVDkHYgEAADwIgQwAAMBiBDIAAACLEcgAAAAsRiADAACwGIEMlqhXr54Mw9Azzzxzzr6BAwfKMAzVq1fvstcFAIAVCGRewDTNSzo+Ly+vjCoBAADnQyCrpM4UnFFOQY6+Tf1WH/78oT7f+bl27t+p226/TUFBQYqJidGMGTOUmJgowzCUmJgoqXCNF8Mw9NJLL+nmm29WcHCw7rvvPklSRkaGHnnkEdWtW1c2m0116tTRqFGjdPr0afd1Fy9erE6dOikyMlI2m02hoaHq1KmTFi5cKElKS0uTYRj65ZdfJEkTJkxwXxMAAG/lVQvDegOny6kcZ46mrp2qBXsW6EzBGfe+9NfTlbk2U5IUGBSoxx9//ILneeqpp2S32xUbGyubzaa8vDwlJiYqJSVFdrtdTZo00Y4dOzRt2jRt3LhR//73v2UYhrZs2aJVq1YpOjpaderU0c6dO7Vs2TL17t1ba9euVWRkpNq1a6cNGzYoLy9PtWvXrrSPwQAAoLjoIatEXKZL2QXZunPBnfrnjn8WCWO5R3LdYazmn2rqk6Wf6KdVPyk3N/e854qLi1NaWpo2b96sGTNmaO7cuUpJSZHNZtOmTZu0ceNGrVy5UpL0ww8/6IcffpAk/fnPf9aRI0e0e/durV+/Xunp6QoJCVFBQYE+/fRTRUVFaeXKlYqKipIkDRkyRCtXrnSfS5L7KQoOh6PsPyQAACogAlkl4nQ5NWrJKKVlpp2zL3f/f4NXUJsgPfjvB9XgigZq0aLFec81YMAAVa1aVZLk6+ur1atXSyqcT3bFFVfIMAy1atXK3f5soMrNzdXAgQMVGRkpX19fhYeHKysrS1Lhw92Lo27dukX+CQBAZceQZSVyIPuAVh1aVay2WflZ+nzn5zJ1/gn/NWrUOO92m82m+Pj4c7afDW833HCDdu3aJT8/PzVv3lx2u909POl0OotVG4EMAOBtCGSVxJn8M5qzdc4F99tr291/zlyfqaC4IL33/XvavGnzedv/7yT7hIQESZLT6dSbb76pq666SpKUk5Ojb775Rt26ddPx48e1a9cuSdKzzz6rcePGKS0tTY0bNz7n/EFBQZKk7Ozsc/Z17NhRW7dulb+//8V+ZAAAKg2GLCsJU+Z5hyrPskXaFNq6cG7WsQXHtHPcTi19fKlsNluxzn/XXXepRYsWcjqdSkhI0JVXXqlGjRopLCxMt956q06ePKnw8HD3BP3x48erefPmuuqqq+Tnd27uPxvSXn31VSUkJGjQoEHufePGjVOTJk3UrVu34v74AAB4NAJZJeJjXPzXWfve2gpNCJVhM+TMcSrqtig1bdpUkhQYGHjRYwMCApScnKwRI0YoOjpaO3bs0K+//qo2bdrob3/7m2rUqCHDMPTZZ58pISFBvr6+cjqdmj17tqpXr37O+Z5//nldffXV8vHx0dq1a7V58/l76gAA8AaGeamrhnqQzMxMORwOZWRkuL/JV1nkFOTo75v+rnc2v3PBNnnH8+QX4icfW2Fwq5tXV9+P+F45OTkaO3asJk6ceLnKBQCg2Crz/fssj+0hmzRpkgzD0MiRI60upUKw+9nVr0m/i/aSZa7N1PZR25U2JU1pU9L0/cjCMFajRg0NHz78MlYLAAB+zyMD2Zo1a/T2229fcMkGbxVsC9b19a6/4H57tF22SJtO7z6t7K3Zqh5eXYMGDdKqVatUq1aty1gpAAD4PY/7luWpU6fUr18/vfPOO3r++ecv2jY3N7fIwqeZmZnlXZ6lAnwD9GzHZ3X0zFGtPrT6nP3BTYMV/HSwIoMi9Y9e/1CNKjXk5+NxfwUAAKh0PK6HbNiwYbrhhhvUvXv3P2w7ceJEORwO9ys6OvoyVGgtm69Nb3V/S+PajlPd0KLreIXaQtW/aX993vtzRVaJJIwBAFBBeNQd+eOPP9b69eu1Zs2aYrUfN26cRo0a5X6fmZnpFaHM39dft15xq+5odId+yfpFJ86cUJB/kBqGNVSBWaBAv4t/oxIAAFxeHhPI9u7dq0ceeUSLFy+W3W7/4wNUuFRDQEBAOVdWMdl8C9cXi3PEKc4R597uLxZbBQCgovGYZS++/PJL/fnPf5avr697m9PplGEY8vHxUW5ubpF95+MNX5sFAKCy8Yb7t8f0kHXr1u2cxUMHDRqkxo0ba8yYMX8YxgAAACoqjwlkISEhuvLKK4tsq1KliqpVq3bOdgAAAE/icd+yBAAAqGw8pofsfJKSkqwuAQAA4JLRQwYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxjwlkM2bMUIsWLRQaGqrQ0FC1b99eCxcutLosAACAS+YxgaxOnTqaNGmS1q1bp7Vr16pr167q06ePtmzZYnVpAAAAl8QwTdO0uojSCg8P1+TJkzV48OBitc/MzJTD4VBGRoZCQ0PLuToAAFAWvOH+7Wd1AaXhdDr1z3/+U9nZ2Wrfvv0F2+Xm5io3N9f9PjMz83KUBwAAUCIeM2QpSZs3b1ZwcLACAgL0wAMP6IsvvlDTpk0v2H7ixIlyOBzuV3R09GWsFgAAoHg8asgyLy9P6enpysjI0Keffqp3331XycnJFwxl5+shi46OrtRdngAAVDbeMGTpUYHsf3Xv3l3169fX22+/Xaz23vALBQCgsvGG+7dHDVn+L5fLVaQHDAAAwBN5zKT+cePG6frrr1dMTIyysrI0Z84cJSUladGiRVaXBgAAcEk8JpAdOXJE/fv318GDB+VwONSiRQstWrRIPXr0sLo0AACAS+Ixgey9996zugQAAIBy4dFzyAAAAM4nLS1NhmHIMAwlJSVZXc4fIpABAIAyl5OTo2nTpqlDhw4KDg52h6NrrrlGL7/8sgYOHCjDMJSYmGh1qX8oKSnJXX9aWlq5XINABgCAl6tXr547cFzo9cwzzxT7fMePH9fVV1+tUaNGacWKFXI6ne59q1at0mOPPXbe48aOHav27dsrMjJSdrtdcXFxGj58uI4ePXqpP2KFRyADAMDLxcfHq127dmrXrp1q167t3t6qVSv39jp16hQ5Ji8vT5LkOnNGzuxsnV67VqeWL1funj16eNgwbdy4UZL0yCOPaP78+e7j1q1bp5kzZ+rgwYOSpOTkZO3cuVOS9OKLL2rVqlU6evSo8vLylJqaqtdff10NGjSQJC1evFhXXHGFqlSpon79+ik7O1vPP/+8IiIiFBUVpfHjx1/053z11VdlGIZ8fX314YcfSioMgc2aNVNYWJj8/f1Vq1YtDRgwwF3fM888oy5durjPERsbK8MwNHDgQEnStGnT1KpVK4WHh8vf318RERG6+eabtWPHjpL9EkwvkpGRYUoyMzIyrC4FAADL5Bc4zbwCp/nL8Wzzp11HzVV7jpsZp/PM07kF5vjx401JpiQzNTXVNE3T7Ny5synJvPvuu83Ro0ebERERZr26dc3c9HQzffTj5hM1o8w4m830NwyzimG4j2/ZooXpdDrNJUuWnHPO/v37u7eNGzfONE3T/L//+z+zQ4cOpiRz6NCh5i233OJuI8kMDg42GzVq5H7fpEkTMzAw0IyLi3Nv+/bbb03TNM3U1FT3tiVLlpjvvfeeaRiG6evra3700Ufuz6Jly5amw+Ewr7zySrNx48am8Vv9CQkJpmma5jvvvGM2adLEfa5WrVqZ7dq1M5999lnTNE2zT58+ZpUqVcwmTZqYV155penr62tKMuvUqWOeOXOm2L8TAhkAAF4kN99pLti43+z92o9m3TEL3K8GT35jPjx7nTl89LgLBjKbzWb6+/ubVzZrZl7ZsKG5Nf4q8xaHw90+xt+/SCDrVb+B6crLKxLIzr46duzo/nPt2rXNgoIC8/Dhw6aPj48pyVy2bJk5derUIse88847pmmaRY5dtmyZ6XQ6zbp165qSzDFjxpimWTSQPfDAA6aPj4/p6+trzpkzp8jnsWnTJtPpdLrfv/POO+7jdu3aZZqmed5AedaWLVvMvLw89/vFixe72/773/8u9u+FIUsAALxEfoFLzy7YomFzNmjjvoyi+5ymvt50ULNX/XLRc6xZs0Ybli7VvOAQ/XLypD7PKDzPPVWr6tu4+nqt9n+HNtenpero62/IdZGn6vj4+Gj//v1atGiRvvzyS7lcLjVo0ECtWrXSP/7xjyJte/XqJalwzpskVa1aVR07dpSPj4/q1q0rSTp8+PA513jrrbfkcrn08ssv66677iqyLyUlRQkJCe4vHgwdOtS978CBAxf9LCTpl19+UZcuXRQaGiofH58i66MW5/izCGQAAHiBnHyn/rEyTR+tTL9oO6frwo+47tKli5o3aqQTM2fJOH1aW3JydLb1DSGFz5hsZrfL+G3bUadTv86eLRkXjhuRkZGSpJkzZ+qzzz6TJN1yyy3q1q2bNm7cqGrVqrnbnn2OpZ+fX5H3kmQYhVc1z/OI7uDgYEnSjBkzdOzYMff2ZcuWacCAAVq/fr3sdrsSEhLUpEmT/34Wv/sywvns2bNHffv21fLlyyVJrVu3VqtWrYp9/O8RyAAA8AJ+vobeTt5TomNy8ooGiho1asjw99fJ34LT+YT4+ira319S4bjd9LRUZf20/ILto6KiJElfffWVlixZIsMwNGfOHK1atUpXX3217r333hLVfD5vvPGGatWqpW3btulPf/qTTp06JanwG59nA9zmzZu1evVq9e/f/5zjg4KC3H/Ozs52/3nDhg3uLzcsWrRIa9as0ZgxY0pVI4EMAIBKzjRNLd1+TEeyLjx0eD6G8b/vDTkzMuQ8cUJS0d6wb7IyJUlZTqfyXC73MTOOH9dfXnjhgtcIDQ1V69atlZeXp/z8fPn6+mrv3r269dZbtWTJkiJhqLRiYmK0cOFCORwOrVmzRn379lVubq5atGjhbtO8eXM1adJEkydPPuf4+vXry/+3kNm9e3ddffXV+vTTT9WsWTP5+vpKKhxObd68uYYPH16qGglkAABUcvlOUxv2/lri446eL8D9bkgwxmbTzQ6HJOnDX39Vrz27dd2e3TrkdMpX0u0Oh5oF2OX63THXXnut3nrrLcXExLi3PfTQQ+4/FxQUyDAMpaenKzExUe+++26J6z6fFi1a6IsvvpDNZtP333+vu+66S127dtWLL76oWrVq6cyZM2rcuLFmzJhxzrHVqlXTq6++qujoaB0+fFirVq3SoUOH1LhxY73//vuKjY1VXl6eqlevrrlz55aqPsM832BrJZWZmSmHw6GMjIwi484AAFRmuQVOvfbDLr3+w64/bHty2WxlLC8MFclr/6NrWzdTYmKikpOTNWDAAM18913tuLq9XL8N+zlNUx/8ekKfZ2Rob36+bIahFna7HqxWXW1+693aecvN6vNbL1lqaqrq1aungQMH6oMPPlDnzp01adIktW/f/g9rq8z3bwIZAACVnMs0tWDTAY2Ym1LsY2y+Pkp5uoeCAvyKnuvMGR2ZMkW/zp5TrPMYAQG6YsVP8jnP0OPWrVv13HPPaenSpdq/f79GjRqlqVOnntPOG+7fDFkCAFDJ+RiGejWLUmig3x83/k2vK2u6v7lY5FyBgao2eLCM3+ZU/RFH377nTkb7zeHDhzV37lxlZGTozjvv1PPPP1/s+iobAhkAAF7A6XLp7nZ1i9XWx5AeTKyvQJvveff7hoer1stTJb+LB7ygtgmq8X9Pyicw8Lz7ExMTZZqmsrKyNHfuXAVeoJ03IJABAOAFAm1+erTHFerWJPKi7QxDmnhzc8VWr3LBNj52u4I7dVK9jz5UULt25+z3rVZN1R96SNHvvlvsnjRvxxwyAAC8SL7Tpb8v3aN/rEjT4cyi36KMjw7TY9ddoTb1wmX3P3/v2O+ZTqdMp1POo0eVvXqNXDlnZIuOUZWr28ksKJCP3V4mNXvD/ZtABgCAl8nJd8rP19CqPSe05+gp2fx8lFAvXHWqBsnHkPx8K9YAmjfcv4s/uw8AAFQKZ3u/Ojaoro4NqltcDSTmkAEAAFiOQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMY8JZBMnTlRCQoJCQkIUGRmpvn37avv27VaXBQAAcMk8JpAlJydr2LBhWrlypRYvXqz8/Hxdd911ys7Otro0AACAS2KYpmlaXURpHD16VJGRkUpOTta1115brGMyMzPlcDiUkZGh0NDQcq4QAACUBW+4f/tZXUBpZWRkSJLCw8Mv2CY3N1e5ubnu95mZmeVeFwAAQEl5zJDl77lcLo0cOVIdO3bUlVdeecF2EydOlMPhcL+io6MvY5UAAADF45FDlg8++KAWLlyoZcuWqU6dOhdsd74esujo6Erd5QkAQGXDkGUF9PDDD2vBggVaunTpRcOYJAUEBCggIOAyVQYAAFA6HhPITNPU8OHD9cUXXygpKUmxsbFWlwQAAFAmPCaQDRs2THPmzNH8+fMVEhKiQ4cOSZIcDocCAwMtrg4AAKD0PGYOmWEY590+c+ZMDRw4sFjn8IYxaAAAKhtvuH97TA+Zh+RGAACAEvPIZS8AAAAqEwIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxTwqkC1dulQ33XSTatWqJcMw9OWXX1pdEgAAwCXzqECWnZ2tli1b6o033rC6FAAAgDLjZ3UBJXH99dfr+uuvL3b73Nxc5ebmut9nZmaWR1kAAACXxKN6yEpq4sSJcjgc7ld0dLTVJQEAAJyjUgeycePGKSMjw/3au3ev1SUBAACcw6OGLEsqICBAAQEBVpcBAABwUZW6hwwAAMATEMgAAAAs5lFDlqdOndKuXbvc71NTU5WSkqLw8HDFxMRYWBkAAEDpeVQgW7t2rbp06eJ+P2rUKEnSgAEDNGvWLIuqAgAAuDQeFcgSExNlmqbVZQAAAJQp5pABAABYjEAGAABgMQIZAACAxQhkAAAAFitxIPvXv/6lIUOG6IknntC2bduK7Pv111/VtWvXMisOAADAG5QokM2ZM0e9e/fWoUOHtGLFCsXHx2v27Nnu/Xl5eUpOTi7zIgEAACqzEi17MXnyZL388ssaMWKEJOmTTz7Rvffeq5ycHA0ePLhcCgQAAKjsShTIdu7cqZtuusn9/vbbb1dERIR69+6t/Px8/fnPfy7zAgEAACq7EgWy0NBQHT58WLGxse5tXbp00YIFC3TjjTdq3759ZV4gAABAZVeiOWRt27bVwoULz9neuXNnff3113rllVfKqi4AAACvUaJA9uijj8put593X2Jior7++mv179+/TAoDAADwFoZZiodD9u/fX126dNG1116r+vXrl0dd5SIzM1MOh0MZGRkKDQ21uhwAAFAM3nD/LtXCsDabTRMnTlTDhg0VHR2tu+++W++++6527txZ1vUBAABUeqXqITtr//79Wrp0qZKTk5WcnKwdO3YoKiqqwk7u94aEDQBAZeMN9+9LenRS1apVVa1aNVWtWlVhYWHy8/NTREREWdUGAADgFUoVyJ588kl16NBB1apV09ixY5WTk6OxY8fq0KFD2rBhQ1nXCAAAUKmVasjSx8dHERERevTRR3XzzTfriiuuKI/aypw3dHkCAFDZeMP9u0QLw561YcMGJScnKykpSVOnTpXNZlPnzp2VmJioxMREjwloAAAAFcElTeo/a+PGjZo2bZpmz54tl8slp9NZFrWVOW9I2AAAVDbecP8uVQ+ZaZrasGGDkpKSlJSUpGXLlikzM1MtWrRQ586dy7pGAACASq1UgSw8PFynTp1Sy5Yt1blzZw0dOlSdOnVSWFhYGZcHAABQ+ZUqkH300Ufq1KlTpe02BAAAuJxKFchuuOGGsq4DAADAa13SwrAAAAC4dAQyAAAAixHIAAAALEYgAwAAsBiBDAAAwGIEMgAAAIsRyAAAACxGIAMAALAYgQwAAMBiBDIAAACLEcgAAAAsRiADAACwGIEMAADAYgQyAAAAi3lcIHvjjTdUr1492e12tWvXTqtXr7a6JAAAgEviUYFs3rx5GjVqlMaPH6/169erZcuW6tmzp44cOWJ1aQAAAKVmmKZpWl1EcbVr104JCQl6/fXXJUkul0vR0dEaPny4xo4de0773Nxc5ebmut9nZmYqOjpaGRkZCg0NvWx1AwCA0svMzJTD4ajU92+P6SHLy8vTunXr1L17d/c2Hx8fde/eXStWrDjvMRMnTpTD4XC/oqOjL1e5AAAAxeYxgezYsWNyOp2qUaNGke01atTQoUOHznvMuHHjlJGR4X7t3bv3cpQKAABQIn5WF1CeAgICFBAQYHUZAAAAF+UxPWTVq1eXr6+vDh8+XGT74cOHVbNmTYuqAgAAuHQeE8hsNptat26t77//3r3N5XLp+++/V/v27S2sDAAA4NJ41JDlqFGjNGDAALVp00Zt27bVK6+8ouzsbA0aNMjq0gAAAErNowLZHXfcoaNHj+rpp5/WoUOH1KpVK3377bfnTPQHAADwJB61Dtml8oZ1TAAAqGy84f7tMXPIAAAAKisCGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAWI5ABAABYjEAGAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDF/KwuAACAsuJyuZSXl2d1GSghf39/+fr6Wl2GpQhkAIBKIS8vT6mpqXK5XFaXglIICwtTzZo1ZRiG1aVYgkAGAPB4pmnq4MGD8vX1VXR0tHx8mJHjKUzT1OnTp3XkyBFJUlRUlMUVWYNABgDweAUFBTp9+rRq1aqloKAgq8tBCQUGBkqSjhw5osjISK8cvuR/IQAAHs/pdEqSbDabxZWgtM4G6fz8fIsrsQaBDABQaXjr/KPKwNt/dwQyAAAAi3lMIPvb3/6mDh06KCgoSGFhYVaXAwBAuUlMTNTIkSOtLgOXkccEsry8PN1222168MEHrS4FAIBiI1yhODzmW5YTJkyQJM2aNcvaQgAAAMqYx/SQlUZubq4yMzOLvAAAuFwGDhyo5ORkTZ8+XYZhyDAMpaWlKTk5WW3btlVAQICioqI0duxYFRQUXPA833zzjRwOh2bPni1J2rt3r26//XaFhYUpPDxcffr0UVpaWpHr9u3bV1OmTFFUVJSqVaumYcOGee03GD1BpQ5kEydOlMPhcL+io6OtLgkA4EWmT5+u9u3ba+jQoTp48KAOHjwof39//elPf1JCQoI2btyoGTNm6L333tPzzz9/3nPMmTNHd911l2bPnq1+/fopPz9fPXv2VEhIiH788UctX75cwcHB6tWrV5HHRi1ZskS7d+/WkiVL9MEHH2jWrFmMMlVglgaysWPHuv+P4UKvbdu2lfr848aNU0ZGhvu1d+/eMqweAICLczgcstlsCgoKUs2aNVWzZk29+eabio6O1uuvv67GjRurb9++mjBhgqZOnXrOY5/eeOMNPfTQQ/r666914403SpLmzZsnl8uld999V82bN1eTJk00c+ZMpaenKykpyX1s1apV3de48cYbdcMNN+j777+/nD8+SsDSOWSPPfaYBg4ceNE2cXFxpT5/QECAAgICSn08AABlbevWrWrfvn2Rdbc6duyoU6dOad++fYqJiZEkffrppzpy5IiWL1+uhIQEd9uNGzdq165dCgkJKXLenJwc7d692/2+WbNmRVa8j4qK0ubNm8vrx8IlsjSQRUREKCIiwsoSAACokOLj47V+/Xq9//77atOmjTvAnTp1Sq1bt3bPJ/u9399T/f39i+wzDIMHr1dgHvMty/T0dJ04cULp6elyOp1KSUmRJDVo0EDBwcHWFgcAwAXYbDb3o50kqUmTJvrss89kmqY7ZC1fvlwhISGqU6eOu139+vU1depUJSYmytfXV6+//rok6aqrrtK8efMUGRmp0NDQy/vDoNx4zKT+p59+WvHx8Ro/frxOnTql+Ph4xcfHa+3atVaXBgDABdWrV0+rVq1SWlqajh07poceekh79+7V8OHDtW3bNs2fP1/jx4/XqFGj5ONT9LZ8xRVXaMmSJfrss8/ca5n169dP1atXV58+ffTjjz8qNTVVSUlJGjFihPbt22fBT4iy4DGBbNasWTJN85xXYmKi1aUBAHBBo0ePlq+vr5o2baqIiAjl5+frX//6l1avXq2WLVvqgQce0ODBg/XXv/71vMc3atRIP/zwg+bOnavHHntMQUFBWrp0qWJiYnTzzTerSZMmGjx4sHJycugx82CGaZqm1UVcLpmZmXI4HMrIyOAvLQBUIjk5OUpNTVVsbKzsdrvV5aAULvY79Ib7t8f0kAEAAFRWBDIAAACLEcgAAAAsRiADAACwGIEMAADAYgQyAAAAixHIAAAALEYgAwAAsJjHPMsSAIDylpPvlI9hyM/XUIHTlMs0Zff3tboseAF6yAAAXi8n36mMM/l658c9+vOby3XtS0v05zeX650f9yjjTL5y8p1/fJIKICkpSYZh6OTJk5IKHzsYFhZmaU0oHgIZAMCr5RW49I8Vv6jN84s19bsd2nIgU/t+PaMtBzI19bsdavP8Yv1jxS/KK3CV6XUNw7jo65lnninT60nS559/rh49eigiIkKhoaFq3769Fi1aVObXQckxZAkA8Fo5+U79Y8UveuFfWy/YJt9p6oV/bZVhSPdcXbfMhjAPHjzo/vO8efP09NNPa/v27e5twcHB7j+bpimn0yk/v0u7bS9dulQ9evTQCy+8oLCwMM2cOVM33XSTVq1apfj4+Es6Ny4NPWQAAK+Vm+/U5EXbitX2pW+3KbcMe8lq1qzpfjkcDhmG4X6/bds2hYSEaOHChWrdurUCAgK0bNky5ebmasSIEYqMjJTdbtc111yjNWvWFPuar7zyip544gklJCSoYcOGeuGFF9SwYUN9/fXXZfZzoXQIZAAAr5ST79Q/Vv6ifKdZrPb5TlMfrki7rPPJxo4dq0mTJmnr1q1q0aKFnnjiCX322Wf64IMPtH79ejVo0EA9e/bUiRMnSnV+l8ulrKwshYeHl3HlKCkCGQDAK/kYhr79z6ESHbPwP4fkYxjlVNG5nn32WfXo0UP169dXQECAZsyYocmTJ+v6669X06ZN9c477ygwMFDvvfdeqc4/ZcoUnTp1SrfffnsZV46SYg4ZAMAr+fkayjiTX6JjMnPy5ed7+QJZmzZt3H/evXu38vPz1bFjR/c2f39/tW3bVlu3XngO3IXMmTNHEyZM0Pz58xUZGVkm9aL06CEDAHilAqcpR6B/iY4JtfuroJhDnGWhSpUq5XLejz/+WEOGDNEnn3yi7t27l8s1UDIEMgCAV3KZpnpdWbNEx1x/ZU25zMsXyH6vfv36stlsWr58uXtbfn6+1qxZo6ZNmxb7PHPnztWgQYM0d+5c3XDDDeVRKkqBQAYA8Ep2f1/1v7qu/Is5BOnva+ie9vUsW7m/SpUqevDBB/X444/r22+/1c8//6yhQ4fq9OnTGjx4cLHOMWfOHPXv319Tp05Vu3btdOjQIR06dEgZGRnlXD3+CIEMAOC1Avx99XjPxsVqO6ZXYwX4WXvbnDRpkm655Rbdc889uuqqq7Rr1y4tWrRIVatWLdbxf//731VQUKBhw4YpKirK/XrkkUfKuXL8EcM0Lep7tUBmZqYcDocyMjIUGhpqdTkAgDKSk5Oj1NRUxcbGym63l+jYvAKXPliRppe+3XbeJTD8fQ090auxBrSvJ5vFgawyu9jv0Bvu33zLEgDg1Wx+Prrn6rq6vU20PlyRpoX/OaTMnHyF2v11/ZU1dU/7egrw8yGMoVwRyAAAXs/u7yu7v6+GdIrTfdfWl5+voQKnKZdpWjZnDN6FQAYAwG9+H75sfpdvvTGA/lcAAACLEcgAAAAsRiADAACwGIEMAADAYkzqBwDgrPwcyfCRfP0kZ4FkuiT/kq1rBpQGgQwAgPwzUkGutOYd6eevpJwMye6QmvaWEoZKfgGSf6DVVaISI5ABALxbQa605l3p+wmSM7/ovkObpOQXpW7jpbb3FQazMpSYmKhWrVrplVdeKdPzwvMQyAAA3iv/TGEY++6vF27jzP9tvyElDC7TnrLPP/9c/v7+ZXY+eC4m9QMAvFdBTmHPWHF8/0xhb1oZCg8PV0hISJmeE56JQAYA8E75OdLqd88dprwQZ35hb1p+TpmVkJiYqJEjR0qS3nzzTTVs2FB2u101atTQrbfeWmbXQcXHkCUAwDsZPtLWr0p2zNavpA4jyryUtWvXasSIEfrwww/VoUMHnThxQj/++GOZXwcVF4EMAOCdfP0Kv01ZEjkZhceVsfT0dFWpUkU33nijQkJCVLduXcXHx5f5dVBxecSQZVpamgYPHqzY2FgFBgaqfv36Gj9+vPLy8qwuDQDgqZwFhUtblITdUXhcGevRo4fq1q2ruLg43XPPPZo9e7ZOnz5d5tdBxeURgWzbtm1yuVx6++23tWXLFk2bNk1vvfWWnnzySatLAwB4KtNVuM5YSTTpXXhcGQsJCdH69es1d+5cRUVF6emnn1bLli118uTJMr8WKiaPCGS9evXSzJkzdd111ykuLk69e/fW6NGj9fnnn1tdGgDAU/nbpYQhkm8xl53w9S9sX04r9/v5+al79+566aWXtGnTJqWlpemHH34ol2uh4vHYOWQZGRkKDw+/aJvc3Fzl5v73K8qZmZnlXRYAwJP42QsXfb3YOmRndXumzBeGPWvBggXas2ePrr32WlWtWlX/+te/5HK51KhRo3K5Hioej+gh+1+7du3Sa6+9pvvvv/+i7SZOnCiHw+F+RUdHX6YKAQAewT+wcAX+6/524Z4yX//C/W2Hltvjk8LCwvT555+ra9euatKkid566y3NnTtXzZo1K5froeIxTNM0rbr42LFj9eKLL160zdatW9W4cWP3+/3796tz585KTEzUu+++e9Fjz9dDFh0drYyMDIWGhl5a8QCACiMnJ0epqamKjY2V3V6KIUX3syzfLVza4uyzLJv0Lhym5FmW5e5iv8PMzEw5HI5Kff+2NJAdPXpUx48fv2ibuLg42Ww2SdKBAweUmJioq6++WrNmzZKPT8k6+LzhFwoA3uiSA9lZ+TmF65P5+hV+m9J0lducMRTl7YHM0jlkERERioiIKFbb/fv3q0uXLmrdurVmzpxZ4jAGAMAf+n348rNZVwe8jkdM6t+/f78SExNVt25dTZkyRUePHnXvq1mzpoWVAQAAXDqPCGSLFy/Wrl27tGvXLtWpU6fIPgtHXAEAAMqER4z7DRw4UKZpnvcFAADg6TwikAEAAFRmBDIAAACLecQcMgAALofcglz5GD7y9fGV0+WUy3QpoJxW5wd+j0AGAPB6OQU5ynPm6eNtH2tx+mJl5WUpxBaiHjE9dGfjO2Xztcnux3pkKD8EMgCAVzsbxKZvmK4CV0GRfdtObNOMTTP0SPwj+kuTv8jmy9pkKB/MIQMAeK2cghzN2TpHU9dNPSeMnVXgKtDUdVM1d9tc5RTkXOYKy09SUpIMw9DJkyfP2ZeYmKiRI0de9pq8GYEMAOC1cp25mr5herHavrL+FeU588q5oj+Wl2d9DSh7BDIAgFfKLcjVx9s+vmDP2P8qcBVo3vZ5yi3ILdM6srKy1K9fP1WpUkVRUVGaNm1akR6qevXq6bnnnlP//v0VGhqq++67T5K0bNkyderUSYGBgYqOjtaIESOUnZ3tPu+HH36oNm3aKCQkRDVr1tRf/vIXHTlyRJKUlpamLl26SJKqVq0qwzA0cODAMv25UDIEMgCAV/IxfPTv9H+X6JjFvyyWYRhlWseoUaO0fPlyffXVV1q8eLF+/PFHrV+/vkibKVOmqGXLltqwYYOeeuop7d69W7169dItt9yiTZs2ad68eVq2bJkefvhh9zH5+fl67rnntHHjRn355ZdKS0tzh67o6Gh99tlnkqTt27fr4MGDmj69eD2FKB9M6gcAeCVfH19l5WWV6JisvCz5+ZTdrTMrK0sffPCB5syZo27dukmSZs6cqVq1ahVp17VrVz322GPu90OGDFG/fv3cvWgNGzbUq6++qs6dO2vGjBmy2+2699573e3j4uL06quvKiEhQadOnVJwcLDCw8MlSZGRkQoLCyuznwmlQw8ZAMArOV1OhdhCSnRMiC2k2EOcxbFnzx7l5+erbdu27m0Oh0ONGjUq0q5NmzZF3m/cuFGzZs1ScHCw+9WzZ0+5XC6lpqZKktatW6ebbrpJMTExCgkJUefOnSVJ6enpZVY/yg49ZAAAr+QyXeoR00PbTmwr9jE96vaw5DnKVapUKfL+1KlTuv/++zVixIhz2sbExCg7O1s9e/ZUz549NXv2bEVERCg9PV09e/bkSwEVFIEMAOCVAvwCdEfjOzRj04xi9Xr5+fjpjkZ3lOnK/XFxcfL399eaNWsUExMjScrIyNCOHTt07bXXXvC4q666Sj///LMaNGhw3v2bN2/W8ePHNWnSJEVHR0uS1q5dW6SNzVa4pprT6Tzn+KSkpNL8OLgEDFkCALxWgG+AHol/pFhtH73q0TJfGDYkJEQDBgzQ448/riVLlmjLli0aPHiwfHx8LvrlgTFjxuinn37Sww8/rJSUFO3cuVPz5893T+qPiYmRzWbTa6+9pj179uirr77Sc889V+QcdevWlWEYWrBggY4ePapTp06593Xr1k0TJ04s058VF0cgAwB4LbufXX9p8heNbjP6gpP1/Xz8NLrNaN3Z+M5yeXzSyy+/rPbt2+vGG29U9+7d1bFjRzVp0kR2+4Wv1aJFCyUnJ2vHjh3q1KmT4uPj9fTTT7u/DBAREaFZs2bpn//8p5o2bapJkyZpypQpRc5Ru3ZtTZgwQWPHjlWNGjWKfENz9+7dOnz4cJn/rLgww7RiMNwimZmZcjgcysjIUGhoqNXlAADKSE5OjlJTUxUbG3vRIHPB4397luW87fO0+JffPcuybg/d0eiOy/osy+zsbNWuXVtTp07V4MGDL8s1K4KL/Q694f7NHDIAgNez+9ll97Orf9P+GtBsgPx8/FTgKpBpmmU6Z+x8NmzYoG3btqlt27bKyMjQs88+K0nq06dPuV4XFQuBDACA3/w+fF3OB4lPmTJF27dvl81mU+vWrfXjjz+qevXql+36sB6BDAAAC8XHx2vdunVWlwGLMakfAADAYgQyAAAAixHIAAAALEYgAwAAsBiT+gEA+I0rN7dwhXw/P6mgcNkLn4DyXfYCkOghK1P16tWTYRh65plnrC4FAFACrpwcOTMzdfz9mUq9407t7nGdUu+4U8ffnylnZqZcOTlWl4hKjkBWhuLj49WuXTvVqVPH6lIAAMXkysvTr3PmakfHa3Rs+nTlbt2q/P37lbt1q45Nn64dHa/Rr3PmypWXZ3WpZS4pKUmGYejkyZNWl+L1CGRlxTT1xRdfaOXKlRoyZIjV1QAAisGVk6NfP5qtIy+9JOXnn79Rfr6OvPSSfp09u9x6yg4dOqRHHnlEDRo0kN1uV40aNdSxY0fNmDFDp0+fLpdrlpXExESNHDnS6jI8HoGstFwuqSBHyjwgrZslrXxD9WpHFg5ZPvV/cjqdGjdunOLi4mS32xUeHq42bdpo8uTJVlcOAPiNmZurI9OmFavtkZenySyHXrI9e/YoPj5e3333nV544QVt2LBBK1as0BNPPKEFCxbo3//+d5lfExUPgaw08nOkIz9Lc+6UpjWVvn5EWvR/Uvaxwv17luiNV6dp0qRJSk9PV6NGjVStWjVt3rxZ33zzjbW1AwAkFU7gPzFnzoV7xv5Xfn7h0GVubpnW8dBDD8nPz09r167V7bffriZNmiguLk59+vTRN998o5tuukn33nuvbrzxxv8pJ1+RkZF67733JBX2VA0fPlwjR45U1apVVaNGDb3zzjvKzs7WoEGDFBISogYNGmjhwoUXrOX06dO6/vrr1bFjR508eVLHjx/XXXfdpdq1aysoKEjNmzfX3Llz3e0HDhyo5ORkTZ8+XYZhyDAMpaWlyel0avDgwYqNjVVgYKAaNWqk6dOnl+nnVtkQyEoqP0fat1p6t5u0Z4lkmue22b9OO+dPkSQNGjhQGzdu1M6dO3X8+HF6yACggjAMQ1nfLS7RMVnffScZRpnVcPz4cX333XcaNmyYqlSpct42hmFoyJAh+vbbb3Xw4EH39gULFuj06dO644473Ns++OADVa9eXatXr9bw4cP14IMP6rbbblOHDh20fv16XXfddbrnnnvOOwx68uRJ9ejRQy6XS4sXL1ZYWJhycnLUunVrffPNN/rPf/6j++67T/fcc49Wr14tSZo+fbrat2+voUOH6uDBgzp48KCio6PlcrlUp04d/fOf/9TPP/+sp59+Wk8++aQ++eSTMvvsKhsCWUkVnJHm3lk4XHkRN9bJkmFI7773nmrXrq0uXbro+eefV3h4+GUqFABwUX5+cmVmlugQZ2amDL+yWzFq165dMk1TjRo1KrK9evXqCg4OVnBwsMaMGaMOHTqoUaNG+vDDD91tZs6cqdtuu03BwcHubS1bttRf//pXNWzYUOPGjZPdblf16tU1dOhQNWzYUE8//bSOHz+uTZs2FbneoUOH1LlzZ0VFRenrr79WUFCQJKl27doaPXq0WrVqpbi4OA0fPly9evVyByuHwyGbzaagoCDVrFlTNWvWlK+vr/z9/TVhwgS1adNGsbGx6tevnwYNGkQguwgCWUnkn5FWvyPlZf9h054N/LT+vmA9+dhwxcfHa8eOHXrxxRfVsWNHnTp16jIUCwC4qIIC+YSGlugQ39BQmQUF5VTQf61evVopKSlq1qyZcn8bIh0yZIhmzpwpSTp8+LAWLlyoe++9t8hxLVq0+G+tvr6qVq2amjdv7t5Wo0YNSdKRI0eKHNejRw81aNBA8+bNk81mc293Op167rnn1Lx5c4WHhys4OFiLFi1Senr6H/4Mb7zxhlq3bq2IiAgFBwfr73//e7GO81YEspLwsxdO4C+GTYedigiS/tYrXAu++KfWrVsnqfBfou3bt5djkQCA4jBNUyHXXVeiY0Kuu+78U1VKqUGDBjIM45z7QlxcnBo0aKDAwED3tv79+2vPnj1asWKFPvroI8XGxqpTp05FjvP39y/y3jCMItuM34ZbXS5XkXY33HCDli5dqp9//rnI9smTJ2v69OkaM2aMlixZopSUFPXs2VN5f/Dlho8//lijR4/W4MGD9d133yklJUWDBg36w+O8GSv1l4QrX8rcX6ymn2zJ1ws/5qnOPyYrIma+0vfukyQFBQWpfv365VklAKAYfAICFP6Xu3TszTeLN7Hf319V/3JXma7cX61aNfXo0UOvv/66hg8ffsF5ZGfb9u3bVzNnztSKFSs0aNCgMqtj0qRJCg4OVrdu3ZSUlKSmTZtKkpYvX64+ffro7rvvllQY5Hbs2OHeL0k2m01Op7PI+ZYvX64OHTrooYcecm/bvXt3mdVbGdFDVhJG8T+ua+v6qVcDP7lM6T9bfpZpmuratasWLlyosLCw8qsRAFBsRkCAIh99tFhtI0eNkvG74byy8uabb6qgoEBt2rTRvHnztHXrVm3fvl0fffSRtm3bJl9fX3fbIUOG6IMPPtDWrVs1YMCAMq1jypQp6tevn7p27apt27ZJkho2bKjFixfrp59+0tatW3X//ffr8OHDRY6rV6+eVq1apbS0NB07dkwul0sNGzbU2rVrtWjRIu3YsUNPPfWU1qxZU6b1Vjb0kJVURCPp6PmHHNNGhhR5f119P6nbeOnqhyR/++WoDgBQAj52u6re3U8yCtcZO29Pmb+/Ikc9qqr9/iKfcghk9evX14YNG/TCCy9o3Lhx2rdvnwICAtS0aVONHj26SC9T9+7dFRUVpWbNmqlWrVplXsu0adPkdDrVtWtXJSUl6a9//av27Nmjnj17KigoSPfdd5/69u2rjIwM9zGjR4/WgAED1LRpU505c0apqam6//77tWHDBt1xxx0yDEN33XWXHnrooYsuueHtDNMsw8HwCi4zM1MOh0MZGRkKLeFETklSQa60/kPpX48Vr72vvzR6lxQYVvJrAQCKLScnR6mpqYqNjZXdXvL/AXbl5Mj87RFKWd99J2dmpnxDQxVy3XWq+pe7ZNhs8inFecvaqVOnVLt2bc2cOVM333yz1eWUqYv9Di/5/u0B6CErCb8AKb6f9OMUKevgH7dv1a/wGABAheZjt0t2u8IHDVT4vYNk+PkVfpvSNMt0zlhpuVwuHTt2TFOnTlVYWJh69+5tdUkoYx4zh6x3796KiYmR3W5XVFSU7rnnHh04cODyF2L4SgO+lqpEXLxdg+7SnyZL/oEXbwcAqDB8AgLkY7PJ8PGRj81WIcKYJKWnp6tGjRqaM2eO3n//ffmV4VpoqBg8JpB16dJFn3zyibZv367PPvtMu3fv1q233nr5C/GzSWEx0kMrpHb3SwH/03Ua0Ui6cZp018eSb9nPNQAAeJ969erJNE3t3btX3bp1s7oclAOPnUP21VdfqW/fvsrNzT1n3ZWzcnNz3QvqSYVj0NHR0WU3Bp13WvLxLXyuZf5pqUqkFB5b+OBxP8IYAFwulzqHDNZjDpkHOnHihGbPnq0OHTpcMIxJ0sSJEzVhwoTyK8RW+GgJ1Yovut1j+h0BoHLx0D4GiN+dR0WHMWPGqEqVKqpWrZrS09M1f/78i7YfN26cMjIy3K+9e/depkoBAJfT2bW6WAnec5194PnFOloqM0uHLMeOHasXX3zxom22bt2qxo0bS5KOHTumEydO6JdfftGECRPkcDi0YMEC96Mg/og3dHkCgDcyTVPp6enKz89XrVq15OPjUf0NXs00TZ0+fVpHjhxRWFiYoqKizmnjDfdvSwPZ0aNHdfz48Yu2iYuLK/Kg07P27dun6Oho/fTTT2rfvn2xrucNv1AA8FZ5eXlKTU095zmN8AxhYWGqWbPmeTtZvOH+bekcsoiICEVE/MHyERdw9l+430/aBwB4L5vNpoYNGzJs6YH8/f2LPCLKG3nEpP5Vq1ZpzZo1uuaaa1S1alXt3r1bTz31lOrXr1/s3jEAQOXn4+PDtyzhkTxikD0oKEiff/65unXrpkaNGmnw4MFq0aKFkpOTFVBBFu0DAAAoLY/oIWvevLl++OEHq8sAAAAoFx7RQwYAAFCZeUQPWVk5+4XSzMxMiysBAADFdfa+XZkXj/WqQJaVlSVJio6OtrgSAABQUllZWXI4HFaXUS489lmWpeFyuXTgwAGFhIQUezHZy+3s8zb37t1baddaKQk+j3PxmRTF51EUn8e5+EyK8sTPwzRNZWVlVepFf72qh8zHx0d16tSxuoxiCQ0N9Zh/US4HPo9z8ZkUxedRFJ/HufhMivK0z6Oy9oydVTljJgAAgAchkAEAAFiMQFbBBAQEaPz48Sx4+xs+j3PxmRTF51EUn8e5+EyK4vOomLxqUj8AAEBFRA8ZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECWQXWu3dvxcTEyG63KyoqSvfcc48OHDhgdVmWSUtL0+DBgxUbG6vAwEDVr19f48ePV15entWlWeZvf/ubOnTooKCgIIWFhVldjiXeeOMN1atXT3a7Xe3atdPq1autLskSS5cu1U033aRatWrJMAx9+eWXVpdkqYkTJyohIUEhISGKjIxU3759tX37dqvLstSMGTPUokUL94Kw7du318KFC60uC78hkFVgXbp00SeffKLt27frs88+0+7du3XrrbdaXZZltm3bJpfLpbfffltbtmzRtGnT9NZbb+nJJ5+0ujTL5OXl6bbbbtODDz5odSmWmDdvnkaNGqXx48dr/fr1atmypXr27KkjR45YXdpll52drZYtW+qNN96wupQKITk5WcOGDdPKlSu1ePFi5efn67rrrlN2drbVpVmmTp06mjRpktatW6e1a9eqa9eu6tOnj7Zs2WJ1aRDLXniUr776Sn379lVubq78/f2tLqdCmDx5smbMmKE9e/ZYXYqlZs2apZEjR+rkyZNWl3JZtWvXTgkJCXr99dclFT6vNjo6WsOHD9fYsWMtrs46hmHoiy++UN++fa0upcI4evSoIiMjlZycrGuvvdbqciqM8PBwTZ48WYMHD7a6FK9HD5mHOHHihGbPnq0OHToQxn4nIyND4eHhVpcBC+Tl5WndunXq3r27e5uPj4+6d++uFStWWFgZKqKMjAxJ4r8Xv3E6nfr444+VnZ2t9u3bW10ORCCr8MaMGaMqVaqoWrVqSk9P1/z5860uqcLYtWuXXnvtNd1///1WlwILHDt2TE6nUzVq1CiyvUaNGjp06JBFVaEicrlcGjlypDp27Kgrr7zS6nIstXnzZgUHBysgIEAPPPCAvvjiCzVt2tTqsiAC2WU3duxYGYZx0de2bdvc7R9//HFt2LBB3333nXx9fdW/f39VtlHmkn4mkrR//3716tVLt912m4YOHWpR5eWjNJ8HgAsbNmyY/vOf/+jjjz+2uhTLNWrUSCkpKVq1apUefPBBDRgwQD///LPVZUHMIbvsjh49quPHj1+0TVxcnGw22znb9+3bp+joaP3000+Vqou5pJ/JgQMHlJiYqKuvvlqzZs2Sj0/l+v+K0vwd8cY5ZHl5eQoKCtKnn35aZK7UgAEDdPLkSa/uTWYO2X89/PDDmj9/vpYuXarY2Firy6lwunfvrvr16+vtt9+2uhSv52d1Ad4mIiJCERERpTrW5XJJknJzc8uyJMuV5DPZv3+/unTpotatW2vmzJmVLoxJl/Z3xJvYbDa1bt1a33//vTt4uFwuff/993r44YetLQ6WM01Tw4cP1xdffKGkpCTC2AW4XK5Kd0/xVASyCmrVqlVas2aNrrnmGlWtWlW7d+/WU089pfr161eq3rGS2L9/vxITE1W3bl1NmTJFR48ede+rWbOmhZVZJz09XSdOnFB6erqcTqdSUlIkSQ0aNFBwcLC1xV0Go0aN0oABA9SmTRu1bdtWr7zyirKzszVo0CCrS7vsTp06pV27drnfp6amKiUlReHh4YqJibGwMmsMGzZMc+bM0fz58xUSEuKeV+hwOBQYGGhxddYYN26crr/+esXExCgrK0tz5sxRUlKSFi1aZHVpkCQTFdKmTZvMLl26mOHh4WZAQIBZr14984EHHjD37dtndWmWmTlzpinpvC9vNWDAgPN+HkuWLLG6tMvmtddeM2NiYkybzWa2bdvWXLlypdUlWWLJkiXn/bswYMAAq0uzxIX+WzFz5kyrS7PMvffea9atW9e02WxmRESE2a1bN/O7776zuiz8hjlkAAAAFqt8E3AAAAA8DIEMAADAYgQyAAAAixHIAAAALEYgAwAAsBiBDAAAwGIEMgAAAIsRyAAAACxGIAMAALAYgQxApZKTk6OBAweqefPm8vPzcz94HAAqMgIZgErF6XQqMDBQI0aMUPfu3a0uBwCKhUAGoMJbsGCBwsLC5HQ6JUkpKSkyDENjx451txkyZIjuvvtuValSRTNmzNDQoUNVs2ZNq0oGgBIhkAGo8Dp16qSsrCxt2LBBkpScnKzq1asrKSnJ3SY5OVmJiYnWFAgAl4hABqDCczgcatWqlTuAJSUl6dFHH9WGDRt06tQp7d+/X7t27VLnzp2tLRQASolABsAjdO7cWUlJSTJNUz/++KNuvvlmNWnSRMuWLVNycrJq1aqlhg0bWl0mAJSKn9UFAEBxJCYm6v3339fGjRvl7++vxo0bKzExUUlJSfr111/pHQPg0eghA+ARzs4jmzZtmjt8nQ1kSUlJzB8D4NEIZAA8QtWqVdWiRQvNnj3bHb6uvfZarV+/Xjt27CjSQ/bzzz8rJSVFJ06cUEZGhlJSUpSSkmJN4QBQDAxZAvAYnTt3VkpKijuQhYeHq2nTpjp8+LAaNWrkbvenP/1Jv/zyi/t9fHy8JMk0zctaLwAUl2HyXygAAABLMWQJAABgMQIZAACAxQhkAAAAFiOQAQAAWIxABgAAYDECGQAAgMUIZAAAABYjkAEAAFiMQAYAAGAxAhkAAIDFCGQAAAAW+3/dCmwIEmk4uwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0., 1., 0., 0.], grad_fn=<RoundBackward1>)\n"
     ]
    }
   ],
   "source": [
    "trainer = L.Trainer(max_epochs=100) # Initialize the trainer\n",
    "trainer.fit(modelFromScratch, dataloader) # Train the model\n",
    "data = {'w1': [modelFromScratch.input1_w1.item(),\n",
    "               modelFromScratch.input2_w1.item(),\n",
    "               modelFromScratch.input3_w1.item(),\n",
    "               modelFromScratch.input4_w1.item()],\n",
    "        'w2': [modelFromScratch.input1_w2.item(),\n",
    "               modelFromScratch.input2_w2.item(),\n",
    "               modelFromScratch.input3_w2.item(),\n",
    "               modelFromScratch.input4_w2.item()],\n",
    "        'token': ['Troll2', 'is', 'great!', 'Gymkata'],\n",
    "        'input': ['input1', 'input2', 'input3', 'input4']}\n",
    "df = pd.DataFrame(data)\n",
    "df\n",
    "sns.scatterplot(data=df, x='w1', y='w2', hue='token', s=100) # Plot the weights\n",
    "plt.text(df.w1[0], df.w2[0], df.token[0], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[1], df.w2[1], df.token[1], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[2], df.w2[2], df.token[2], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.text(df.w1[3], df.w2[3], df.token[3], horizontalalignment='left', size='medium', color='black', weight='semibold')\n",
    "plt.show()\n",
    "softmax = nn.Softmax(dim=0) # Initialize the softmax function. dim=0 means the softmax will be calculated for each row and dim=1 means the softmax will be calculated for each column\n",
    "print(torch.round(softmax(modelFromScratch(torch.tensor([[1., 0., 0., 0.]]))), decimals=2)) # Predict the next word for the first word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29913ebe",
   "metadata": {},
   "outputs": [],
   "source": [
    "class WordEmbeddingWithLinearLayer(L.LightningModule):\n",
    "    def __init__(self):\n",
    "        super().__init__() # get the init from the parent class which comes from the lightning module\n",
    "        self.input_to_hidden = nn.Linear(in_features=4, out_features=2, bias=False) # Initialize the linear layer for the input to hidden layer\n",
    "        self.hidden_to_output = nn.Linear(in_features=2, out_features=4, bias=False)\n",
    "        self.loss = nn.CrossEntropyLoss() # Initialize the loss function\n",
    "    def forward(self, input):\n",
    "        hidden = self.input_to_hidden(input) # Calculate the inputs to the top hidden layer\n",
    "        output_values = self.hidden_to_output(hidden) # Calculate the output for the first word\n",
    "        return (output_values)\n",
    "    def configure_optimizers(self):\n",
    "        return Adam(self.parameters(), lr=0.1)\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        input_i, label_i = batch # Get the input and label for the batch\n",
    "        output_i = self.forward(input_i) # Run the forward function\n",
    "        loss = self.loss(output_i, label_i[0]) # Calculate the loss with cross entropy loss\n",
    "        return loss\n",
    "modelLinear= WordEmbeddingWithLinearLayer() # Initialize the model\n",
    "data = {'w1': modelLinear.input_to_hidden.weight.detach()[0].numpy(),\n",
    "        'w2': modelLinear.input_to_hidden.weight.detach()[1].numpy(),\n",
    "        'token': ['Troll2', 'is', 'great!', 'Gymkata'],\n",
    "        'input': ['input1', 'input2']}\n",
    "word_embedding = nn.Embedding.from_pretrained(modelLinear.input_to_hidden.weight.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a9db74",
   "metadata": {},
   "source": [
    "# LeNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0560a7c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 10])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class LeNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.relu=nn.ReLU()\n",
    "        self.pool=nn.AvgPool2d(kernel_size=(2,2), stride=(2,2))\n",
    "        self.conv1=nn.Conv2d(in_channels=1, out_channels=6, kernel_size=(5,5), stride=(1,1), padding=(0,0))\n",
    "        self.conv2=nn.Conv2d(in_channels=6, out_channels=16, kernel_size=(5,5), stride=(1,1), padding=(0,0))\n",
    "        self.conv3=nn.Conv2d(in_channels=16, out_channels=120, kernel_size=(4,4), stride=(1,1), padding=(0,0))\n",
    "        self.linear1=nn.Linear(120, 84)\n",
    "        self.linear2=nn.Linear(84, 10)\n",
    "    def forward(self, x):\n",
    "        x=self.relu(self.conv1(x))\n",
    "        x=self.pool(x)\n",
    "        x=self.relu(self.conv2(x))\n",
    "        x=self.pool(x)\n",
    "        x=self.relu(self.conv3(x)) # num_examples x 120 x 1 x 1 -> num_examples x 120\n",
    "        x=x.reshape(x.shape[0], -1)\n",
    "        x=self.relu(self.linear1(x))\n",
    "        x=self.linear2(x)\n",
    "        return x\n",
    "\n",
    "x=torch.randn(64, 1, 28, 28)\n",
    "model=LeNet()\n",
    "print(model(x).shape)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c0943995",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking accuracy on training data\n",
      "Got 58692/60000 with accuracy 97.82\n",
      "Checking accuracy on test data\n",
      "Got 9785/10000 with accuracy 97.85\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "# Hyperparameters\n",
    "learning_rate=0.001\n",
    "batch_size=64\n",
    "num_epochs=2\n",
    "# Load Data\n",
    "train_dataset=datasets.MNIST(root='dataset/', train=True, transform=transforms.ToTensor(), download=True)\n",
    "train_loader=DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)\n",
    "test_dataset=datasets.MNIST(root='dataset/', train=False, transform=transforms.ToTensor(), download=True)\n",
    "test_loader=DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True)\n",
    "# Initialize network\n",
    "model=LeNet().to(device)\n",
    "# Loss and Optimizer\n",
    "criterion=nn.CrossEntropyLoss()\n",
    "optimizer=optim.Adam(model.parameters(), lr=learning_rate)\n",
    "# Train networks\n",
    "for epoch in range(num_epochs):\n",
    "    for batch_idx, (data, targets) in enumerate(train_loader):\n",
    "        data=data.to(device=device)\n",
    "        targets=targets.to(device=device)\n",
    "        # Forward\n",
    "        scores=model(data)\n",
    "        loss=criterion(scores, targets)\n",
    "        # Backward\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "# Check accuracy on training & test to see how good our model\n",
    "def check_accuracy(loader, model):\n",
    "    if loader.dataset.train:\n",
    "        print(\"Checking accuracy on training data\")\n",
    "    else:\n",
    "        print(\"Checking accuracy on test data\")\n",
    "    num_correct=0\n",
    "    num_samples=0\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for x, y in loader:\n",
    "            x=x.to(device=device)\n",
    "            y=y.to(device=device)\n",
    "\n",
    "            scores=model(x)\n",
    "            _, predictions=scores.max(1)\n",
    "            num_correct+=(predictions==y).sum()\n",
    "            num_samples+=predictions.size(0)\n",
    "    model.train()\n",
    "    print(f'Got {num_correct}/{num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}')\n",
    "    \n",
    "check_accuracy(train_loader, model)\n",
    "check_accuracy(test_loader, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761a2d1f-2f07-4319-9d85-901d68818610",
   "metadata": {},
   "source": [
    "<a id='4'></a>\n",
    "# 4. Computer Vision methods\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da00b209-bedf-412a-8917-e416bc656160",
   "metadata": {},
   "source": [
    "<a id='4.1'></a>\n",
    "## 4.1 Object Detection\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e223733-12af-4778-a27a-0ad4349613c6",
   "metadata": {},
   "source": [
    "## 4.2 Object Tracking"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47958b1c-79e1-4408-8fef-f4a89ce284f4",
   "metadata": {},
   "source": [
    "<a id='4.2'></a>\n",
    "## 4.3 Object Segmentation\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6d13eff-c4ca-4fa3-a81f-3fc53c542677",
   "metadata": {},
   "source": [
    "<a id='4.3'></a>\n",
    "## 4.4 Style Transfer\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aef2f51-3c66-4c97-9cd5-34354fd74bd8",
   "metadata": {},
   "source": [
    "<a id='4.4'></a>\n",
    "## 4.5 Super-Resolution with GAN, Autoencoder\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889b7f99-d495-4a75-94f3-099377f5353c",
   "metadata": {},
   "source": [
    "<a id='4.5'></a>\n",
    "## 4.6 3D reconstruction methods\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acb244db-a4c4-4f57-9c1f-aaa6eddbd59a",
   "metadata": {},
   "source": [
    "<a id='4.5.1'></a>\n",
    "### 4.6.1 Stereopsis\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "458fe914-4460-4b53-878d-de788d2b6d32",
   "metadata": {},
   "source": [
    "<a id='4.5.2'></a>\n",
    "### 4.6.2 Multiview Stereo\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "972bee41-c2ab-4d95-a50b-a5f168bce3f8",
   "metadata": {},
   "source": [
    "<a id='4.6.3'></a>\n",
    "### 4.5.3 Structure from Motion\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42bb223d-e739-4da7-9a2d-0a99680883fe",
   "metadata": {},
   "source": [
    "## 4.7 Super-Resolution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5e6c129-ea49-467b-b56f-6ead29dbec5d",
   "metadata": {},
   "source": [
    "## 4.8 Frame-Interpolation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44e23e2-f7eb-4e46-90c1-480eb47bb604",
   "metadata": {},
   "source": [
    "## 4.9 Optical Character Recognition (OCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9cb0340-3826-4dcb-97fa-8fb77fff723b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d7ce25f3",
   "metadata": {},
   "source": [
    "## 4.10 Image Captioning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c9372ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision.models as models\n",
    "\n",
    "class EncoderCNN(nn.Module):\n",
    "    def __init__(self, embed_size, train_CNN=False):\n",
    "        super(EncoderCNN, self).__init__()\n",
    "        self.train_CNN=train_CNN\n",
    "        self.inception=models.inception_v3(pretrained=True, aux_logits=False)\n",
    "        self.inception.fc=nn.Linear(self.inception.fc.in_features, embed_size)\n",
    "        self.relu=nn.ReLU()\n",
    "        self.dropout=nn.Dropout(0.5)\n",
    "        \n",
    "    def forward(self, images):\n",
    "        features=self.inception(images)\n",
    "        for name, param in self.inception.named_parameters():\n",
    "            if \"fc.weight\" in name or \"fc.bias\" in name:\n",
    "                param.requires_grad=True\n",
    "            else:\n",
    "                param.requires_grad=self.train_CNN\n",
    "        return self.dropout(self.relu(features))\n",
    "\n",
    "class DecoderRNN(nn.Module):\n",
    "    def __init__(self, embed_size, hidden_size, vocab_size, num_layers):\n",
    "        super(DecoderRNN, self).__init__()\n",
    "        self.embed=nn.Embedding(vocab_size, embed_size)\n",
    "        self.lstm=nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)\n",
    "        self.linear=nn.Linear(hidden_size, vocab_size)\n",
    "        self.dropout=nn.Dropout(0.5)\n",
    "        \n",
    "    def forward(self, features, captions):\n",
    "        embeddings=self.dropout(self.embed(captions))\n",
    "        embeddings=torch.cat((features.unsqueeze(1), embeddings), 1)\n",
    "        hiddens, _=self.lstm(embeddings)\n",
    "        outputs=self.linear(hiddens)\n",
    "        return outputs\n",
    "    \n",
    "class CNNtoRNN(nn.Module):\n",
    "    def __init__(self, embed_size, hidden_size, vocab_size, num_layers):\n",
    "        super(CNNtoRNN, self).__init__()\n",
    "        self.encoder=EncoderCNN(embed_size)\n",
    "        self.decoder=DecoderRNN(embed_size, hidden_size, vocab_size, num_layers)\n",
    "        \n",
    "    def forward(self, images, captions):\n",
    "        features=self.encoder(images)\n",
    "        outputs=self.decoder(features, captions)\n",
    "        return outputs\n",
    "    \n",
    "    def caption_image(self, image, vocabulary, max_length=50):\n",
    "        result_caption=[]\n",
    "        with torch.no_grad():\n",
    "            x=self.encoder(image).unsqueeze(0)\n",
    "            states=None\n",
    "            for _ in range(max_length):\n",
    "                hiddens, states=self.decoder.lstm(x, states)\n",
    "                output=self.decoder.linear(hiddens)\n",
    "                predicted=output.argmax(2)\n",
    "                result_caption.append(predicted.item())\n",
    "                x=self.decoder.embed(predicted).unsqueeze(0)\n",
    "                if vocabulary.itos[predicted.item()]==\"<EOS>\":\n",
    "                    break\n",
    "        return [vocabulary.itos[idx] for idx in result_caption]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088c8b94",
   "metadata": {},
   "source": [
    "## ESRGAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "745a1d62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 3, 192, 192])\n",
      "torch.Size([5, 1])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "class ConvBlock(nn.Module):\n",
    "    def __init__(self, in_channels, out_channels, use_act, **kwargs):\n",
    "        super().__init__()\n",
    "        self.cnn = nn.Conv2d(in_channels, out_channels, **kwargs, bias=True)\n",
    "        self.act=nn.LeakyReLU(0.2, inplace=True) if use_act else nn.Identity()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.act(self.cnn(x))\n",
    "    \n",
    "class UpsampleBlock(nn.Module):\n",
    "    def __init__(self, in_c, scale_factor=2):\n",
    "        super().__init__()\n",
    "        self.upsample=nn.Upsample(scale_factor=scale_factor, mode='nearest')\n",
    "        self.conv=nn.Conv2d(in_c, in_c, 3, 1, 1, bias=True)\n",
    "        self.act=nn.LeakyReLU(0.2, inplace=True)\n",
    "    def forward(self, x):\n",
    "        return self.act(self.conv(self.upsample(x)))\n",
    "\n",
    "class DenseResidualBlock(nn.Module):\n",
    "    def __init__(self, in_channels, channels=32, residual_beta=0.2):\n",
    "        super().__init__()\n",
    "        self.residual_beta=residual_beta\n",
    "        self.blocks=nn.ModuleList()\n",
    "        \n",
    "        for i in range(5):\n",
    "            self.blocks.append(ConvBlock(in_channels+channels*i, channels if i <=3 else in_channels, kernel_size=3, stride=1, padding=1, use_act=True if i <=3 else False))\n",
    "            \n",
    "    def forward(self, x):\n",
    "        new_inputs=x\n",
    "        for block in self.blocks:\n",
    "            out=block(new_inputs)\n",
    "            new_inputs=torch.cat([new_inputs, out], dim=1)\n",
    "        return self.residual_beta*out+x\n",
    "    \n",
    "class RRDB(nn.Module):\n",
    "    def __init__(self, in_channels, residual_beta=0.2):\n",
    "        super().__init__()\n",
    "        self.residual_beta=residual_beta\n",
    "        self.rrdb=nn.Sequential(*[DenseResidualBlock(in_channels) for _ in range(3)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.residual_beta*self.rrdb(x)+x\n",
    "    \n",
    "class Generator(nn.Module):\n",
    "    def __init__(self, in_channels=3, num_channels=64, num_blocks=23):\n",
    "        super().__init__()\n",
    "        self.initial=nn.Conv2d(in_channels, num_channels, kernel_size=3, stride=1, padding=1, bias=True)\n",
    "        self.residuals=nn.Sequential(*[RRDB(num_channels) for _ in range(num_blocks)])\n",
    "        self.conv=nn.Conv2d(num_channels, num_channels, kernel_size=3, stride=1, padding=1)\n",
    "        self.upsamples=nn.Sequential(UpsampleBlock(num_channels), UpsampleBlock(num_channels), UpsampleBlock(num_channels))\n",
    "        self.final=nn.Sequential(nn.Conv2d(num_channels, num_channels, 3,1,1,bias=True),\n",
    "                                 nn.LeakyReLU(0.2, inplace=True),\n",
    "                                 nn.Conv2d(num_channels, in_channels, 3,1,1,bias=True))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        initial=self.initial(x)\n",
    "        x=self.conv(self.residuals(initial))+initial\n",
    "        x=self.upsamples(x)\n",
    "        return self.final(x)\n",
    "\n",
    "class Discriminator(nn.Module):\n",
    "    def __init__(self, in_channels=3, features=[64, 64, 128, 128, 256, 256, 512, 512]):\n",
    "        super().__init__()\n",
    "        blocks=[]\n",
    "        for idx, feature in enumerate(features):\n",
    "            blocks.append(ConvBlock(in_channels, feature, kernel_size=3, stride=1 +idx%2, padding=1, use_act=True))\n",
    "            in_channels=feature\n",
    "        self.blocks=nn.Sequential(*blocks)\n",
    "        self.classifier=nn.Sequential(nn.AdaptiveAvgPool2d((6,6)), \n",
    "                                      nn.Flatten(), \n",
    "                                      nn.Linear(512*6*6, 1024), \n",
    "                                      nn.LeakyReLU(0.2, inplace=True), \n",
    "                                      nn.Linear(1024, 1))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x=self.blocks(x)\n",
    "        return self.classifier(x)\n",
    "    \n",
    "def initialize_weights(model, scale=0.1):\n",
    "    for m in model.modules():\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "            nn.init.kaiming_normal_(m.weight.data)\n",
    "            m.weight.data *= scale\n",
    "        elif isinstance(m, nn.Linear):\n",
    "            nn.init.kaiming_normal_(m.weight.data)\n",
    "            m.weight.data *= scale\n",
    "            \n",
    "def test():\n",
    "    gen=Generator()\n",
    "    disc=Discriminator()\n",
    "    low_res=24\n",
    "    x=torch.rand((5,3,low_res, low_res))\n",
    "    gen_out=gen(x)\n",
    "    disc_out=disc(gen_out)\n",
    "    print(gen_out.shape)\n",
    "    print(disc_out.shape)\n",
    "    \n",
    "test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b3c138f-2965-4449-a016-3124209b9b87",
   "metadata": {},
   "source": [
    "<a id='5'></a>\n",
    "# 5. Tensorflow 2 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42d86488-76cf-44c4-ae88-5ec9bbe69a7b",
   "metadata": {},
   "source": [
    "<a id='5.1'></a>\n",
    "## 5.1 Layers (Dense, Flatten, Convolution, Dropout, )\n",
    "<img src=\"img/common_layer.png\" alt=\"drawing\" width=\"800\"/>\n",
    "<a id='5.1.1'></a>\n",
    "### 5.1.1 Dense Layer\n",
    "\n",
    "It connects each node in the layer to every node in the previous layer. The dense layer is used in a feedforward neural network. It is used when the input data is a vector or when a neural network has multiple layers and we want to reduce the dimensionality between layers.\n",
    "<a id='5.1.2'></a>\n",
    "### 5.1.2 Flatten Layer\n",
    "\n",
    "A flatten layer is used to convert a high dimensional matrix into a 1-dimensional vector. It is used to convert the high-dimensional tensors into 1D tensors. It is used after the convolution layer when we need to pass the output of convolution layer to dense layer.\n",
    "<a id='5.1.3'></a>\n",
    "### 5.1.3 Convolution Layer\n",
    "\n",
    "A convolution layer is used in convolutional neural networks (CNNs) to perform feature extraction and image classification tasks. Convolution layers apply filters to local regions of the input data to extract features. It is used for image classification problems. The Convolution layer reduces the spatial dimensions of the input tensor, but increases the number of channels.\n",
    "<a id='5.1.4'></a>\n",
    "### 5.1.4 Dropout Layer\n",
    "\n",
    "A dropout layer is used to prevent overfitting in neural networks. It randomly drops out (turns off) some of the nodes during training, forcing the network to learn to make predictions based on the remaining nodes. It is used before the final dense layer of the Neural Network.\n",
    "<a id='5.1.5'></a>\n",
    "### 5.1.5 Recurrent Layer\n",
    "\n",
    "A recurrent layer is used in recurrent neural networks (RNNs) to process sequential data, such as time series or text data. Recurrent layers allow the network to maintain information from previous time steps and make predictions based on that information.\n",
    "<a id='5.1.6'></a>\n",
    "### 5.1.6 Embedding Layer\n",
    "\n",
    "An embedding layer is used to convert categorical variables into a continuous vector representation. It is commonly used in natural language processing tasks.\n",
    "\n",
    "### 5.1.7 Lambda Layer\n",
    "\n",
    "In TensorFlow, the Lambda layer is a layer that can be used to create a custom layer with a user-defined function. The Lambda layer allows you to define a function that takes as input the output of the previous layer and applies some operation on it to produce the output of the current layer. This layer is useful when you want to apply a custom transformation to the input of a neural network, without having to write a custom layer from scratch.In TensorFlow, the Lambda layer is a layer that can be used to create a custom layer with a user-defined function. The Lambda layer allows you to define a function that takes as input the output of the previous layer and applies some operation on it to produce the output of the current layer. This layer is useful when you want to apply a custom transformation to the input of a neural network, without having to write a custom layer from scratch.\n",
    "\n",
    "### 5.1.8 Other Custom Layer\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516486c1-a4a5-47bc-b9c5-b6a183b9873b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import warnings\n",
    "from tensorflow.keras.layers import Layer\n",
    "\n",
    "class SimpleQuadratic(Layer):\n",
    "    def __init__(self, units=32, activation=None):\n",
    "        '''Initializes the class and sets up the internal variables'''\n",
    "        super(SimpleQuadratic, self).__init__()\n",
    "        self.units=units\n",
    "        self.activation=tf.keras.activations.get(activation)\n",
    "    \n",
    "    def build(self, input_shape):\n",
    "        '''Create the state of the layer (weights)'''\n",
    "        a_init=tf.random_normal_initializer()\n",
    "        self.a=tf.Variable(name='a', initial_value=a_init(shape=(input_shape[-1], self.units), dtype='float32'), trainable=True)\n",
    "        b_init=tf.random_normal_initializer()\n",
    "        self.b=tf.Variable(name='b', initial_value=b_init(shape=(input_shape[-1], self.units), dtype='float32'), trainable=True)\n",
    "        c_init=tf.zeros_initializer()\n",
    "        self.c=tf.Variable(name='c', initial_value=c_init(shape=(self.units,), dtype='float32'), trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        '''Defines the computation from inputs to outputs'''\n",
    "        x_squared=tf.math.square(inputs)\n",
    "        x_squared_times_a=tf.matmul(x_squared, self.a)\n",
    "        x_times_b=tf.matmul(inputs, self.b)\n",
    "        return self.activation(x_squared_times_a+x_times_b+self.c)\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "input1 = np.array([[0.1, 0.2, 0.3, 0.4]]) # Input data\n",
    "input2 = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]])\n",
    "input_dense = tf.keras.Input(shape=(4,)) # Define inputs for each layer\n",
    "input_flatten = tf.keras.Input(shape=(4,))\n",
    "input_conv1d = tf.keras.Input(shape=(8, 1))\n",
    "input_dropout = tf.keras.Input(shape=(4,))\n",
    "input_recurrent = tf.keras.Input(shape=(8, 1))\n",
    "input_embedding = tf.keras.Input(shape=(4,))\n",
    "\n",
    "dense = tf.keras.layers.Dense(units=1, activation='relu')(input_dense) # Define Dense layer\n",
    "flatten = tf.keras.layers.Flatten()(input_flatten) # Define Flatten layer\n",
    "conv1d = tf.keras.layers.Conv1D(filters=2, kernel_size=2, activation='relu')(input_conv1d) # Define Conv1D layer\n",
    "dropout = tf.keras.layers.Dropout(rate=0.5)(input_dropout) # Define Dropout layer\n",
    "recurrent = tf.keras.layers.SimpleRNN(units=1, activation='relu')(input_recurrent) # Define SimpleRNN layer\n",
    "embedding = tf.keras.layers.Embedding(input_dim=4, output_dim=1)(input_embedding) # Define Embedding layer\n",
    "Lambda_L = tf.keras.layers.Lambda(lambda x: tf.abs(x))(input_flatten)\n",
    "custom = SimpleQuadratic(units=1)(input_dense)\n",
    "\n",
    "model_dense = tf.keras.Model(inputs=input_dense, outputs=dense) # Create model instances\n",
    "model_flatten = tf.keras.Model(inputs=input_flatten, outputs=flatten)\n",
    "model_conv1d = tf.keras.Model(inputs=input_conv1d, outputs=conv1d)\n",
    "model_dropout = tf.keras.Model(inputs=input_dropout, outputs=dropout)\n",
    "model_recurrent = tf.keras.Model(inputs=input_recurrent, outputs=recurrent)\n",
    "model_embedding = tf.keras.Model(inputs=input_embedding, outputs=embedding)\n",
    "model_lambda = tf.keras.Model(inputs=input_flatten, outputs=Lambda_L)\n",
    "model_custom = tf.keras.Model(inputs=input_dense, outputs=custom)\n",
    "\n",
    "output_dense = model_dense.predict(np.array(input1)) # Predict and compare outputs of each model\n",
    "output_flatten = model_flatten.predict(np.array(input1))\n",
    "output_conv1d = model_conv1d.predict(np.array(input2).reshape(1, 8, 1))\n",
    "output_dropout = model_dropout.predict(np.array(input1))\n",
    "output_recurrent = model_recurrent.predict(np.array(input2))\n",
    "output_embedding = model_embedding.predict(np.array(input1))\n",
    "output_lambda = model_lambda.predict(np.array(input1))\n",
    "output_custom = model_custom.predict(np.array(input1))\n",
    "\n",
    "print(\"Dense layer output shape:\", output_dense, output_dense.shape) # shape of the output from each layer\n",
    "print(\"Flatten layer output shape:\", output_flatten, output_flatten.shape)\n",
    "print(\"Convolution layer output shape:\", output_conv1d, output_conv1d.shape)\n",
    "print(\"Dropout layer output shape:\", output_dropout, output_dropout.shape)\n",
    "print(\"Recurrent layer output shape:\", output_recurrent, output_recurrent.shape)\n",
    "print(\"Embedding layer output shape:\", output_embedding, output_embedding.shape)\n",
    "print(\"Lambda layer output shape:\", output_lambda, output_lambda.shape)\n",
    "print(\"Simple Custom layer output shape:\", output_custom, output_custom.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bd71e6a-5caf-4a1a-aa64-0583cf4b9c85",
   "metadata": {},
   "source": [
    "<a id='5.2'></a>\n",
    "## 5.2 Activations (ReLU, Sigmoid, LeakyReLU, Hyperbolic Tangent)\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ecb419d-e6be-45d8-acbb-75d97c161c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.keras as keras\n",
    "# Leaky ReLU\n",
    "# TensorFlow2\n",
    "layer = tf.keras.layers.LeakyReLU()\n",
    "output = layer([-3.0, -1.0, 0.0, 2.0])\n",
    "print(list(output.numpy()))\n",
    "layer = tf.keras.layers.LeakyReLU(alpha=0.1)\n",
    "output = layer([-3.0, -1.0, 0.0, 2.0])\n",
    "print(list(output.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c91ade5-1581-4e78-b229-e55e9fb20a8b",
   "metadata": {},
   "source": [
    "## 5.3 Loss functions\n",
    "1. Mean Squared Error (MSE):\\\n",
    "MSE calculates the average squared difference between the true and predicted values. It is commonly used in regression tasks.\n",
    "\n",
    "2. Mean Absolute Error (MAE):\\\n",
    "MAE calculates the average absolute difference between the true and predicted values. Like MSE, it is used for regression tasks but is less sensitive to outliers.\n",
    "\n",
    "3. Binary Cross-Entropy:\\\n",
    "Binary Cross-Entropy measures the performance of a binary classification model. It calculates the cross-entropy loss between the true binary labels and the predicted probabilities. It is used for binary classification tasks.\n",
    "\n",
    "4. Categorical Cross-Entropy:\\\n",
    "Categorical Cross-Entropy measures the performance of a multi-class classification model. It calculates the cross-entropy loss between the true one-hot encoded class labels and the predicted class probabilities. It is used for multi-class classification tasks with one-hot encoded labels.\n",
    "\n",
    "5. Sparse Categorical Cross-Entropy:\\\n",
    "Sparse Categorical Cross-Entropy is similar to Categorical Cross-Entropy but takes integer labels instead of one-hot encoded labels. It is used for multi-class classification tasks with integer labels.\n",
    "\n",
    "6. Hinge:\\\n",
    "Hinge loss is used for binary classification tasks within Support Vector Machines (SVMs). It calculates the hinge loss between the true and predicted values.\n",
    "\n",
    "7. Huber:\\\n",
    "Huber loss is used for regression tasks. It's a combination of MSE and MAE. It is less sensitive to outliers than MSE and provides a smoother transition between MSE and MAE. It has a parameter, delta, that determines the transition point between the two error calculations. By default, delta is set to 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6402cd92-44bb-4ee8-a9e9-7839694486f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import backend as K\n",
    "from tensorflow.keras.losses import Loss\n",
    "class MinLossCallback(tf.keras.callbacks.Callback):\n",
    "    def on_train_begin(self, logs=None):\n",
    "        self.min_loss = float('inf')\n",
    "        self.min_loss_epoch = 0\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        current_loss = logs.get('loss')\n",
    "        if current_loss < self.min_loss:\n",
    "            self.min_loss = current_loss\n",
    "            self.min_loss_epoch = epoch\n",
    "# Regression targets\n",
    "input_regression = np.array([1.0,  2.0, 3.0, 4.0, 5.0, 6.0], dtype=float)\n",
    "output_regression = np.array([1.0, 4.0, 9.0, 16.0, 25.0, 36.0], dtype=float)\n",
    "\n",
    "# Binary classification targets\n",
    "input_binary = np.array([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5], dtype=float)\n",
    "output_binary = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1], dtype=float)\n",
    "\n",
    "# Multiclass classification targets\n",
    "input_multiclass = np.array([-5, 0, 5], dtype=float)\n",
    "output_multiclass = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=float)\n",
    "\n",
    "# Sparse multiclass classification targets\n",
    "input_multiclass = np.array([-5, 0, 5], dtype=float)\n",
    "output_sparse_multiclass = np.array([0, 1, 2], dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f1367ff-5bce-4aa9-90cd-9a07f50c3013",
   "metadata": {},
   "outputs": [],
   "source": [
    "mse = tf.keras.losses.MeanSquaredError()\n",
    "model_mse = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model_mse.compile(optimizer='sgd', loss=mse)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_mse.fit(input_regression, output_regression, epochs=500,verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum Mean Squared Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' MSE predict: {model_mse.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9583721-fefd-4053-9cd6-98a25aa5337a",
   "metadata": {},
   "outputs": [],
   "source": [
    "mae = tf.keras.losses.MeanAbsoluteError()\n",
    "model_mae = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model_mae.compile(optimizer='sgd', loss=mae)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_mae.fit(input_regression, output_regression, epochs=500,verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum Mean Absolute Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' MAE predict: {model_mae.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcc467cd-84d4-4b13-ac6a-c85363146370",
   "metadata": {},
   "outputs": [],
   "source": [
    "binary_crossentropy = tf.keras.losses.BinaryCrossentropy()\n",
    "model_bc = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1]),keras.layers.Dense(units=1, activation='sigmoid')])\n",
    "model_bc.compile(optimizer='sgd', loss=binary_crossentropy)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_bc.fit(input_binary, output_binary, epochs=500, verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum BC Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' BC predict: {model_bc.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f61d7f5c-a09b-467b-a833-b22c3b9313cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_crossentropy = tf.keras.losses.CategoricalCrossentropy()\n",
    "model_cc = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1]), keras.layers.Dense(units=3, activation='softmax')])\n",
    "model_cc.compile(optimizer='sgd', loss=categorical_crossentropy)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_cc.fit(input_multiclass, output_multiclass, epochs=500, verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum CC Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' CC predict: {model_cc.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef388da-4406-4a73-b0ea-9ba0bb1139ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "sparse_categorical_crossentropy = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "model_scc = tf.keras.Sequential([keras.layers.Dense(units=10, input_shape=[1]), keras.layers.Dense(units=3, activation='softmax')])\n",
    "model_scc.compile(optimizer='sgd', loss=sparse_categorical_crossentropy)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_scc.fit(input_multiclass, output_sparse_multiclass, epochs=500, verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum SCC Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' SCC predict: {model_scc.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efa33b0d-30ad-4e73-9a83-b9992448bfcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "hinge = tf.keras.losses.Hinge()\n",
    "model_hinge = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1]),keras.layers.Dense(units=1, activation='sigmoid')])\n",
    "model_hinge.compile(optimizer='sgd', loss=hinge)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_hinge.fit(input_binary, output_binary, epochs=500, verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum Hinge Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' Hinge predict: {model_hinge.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63d95582-e09b-4407-84be-858b1831179b",
   "metadata": {},
   "outputs": [],
   "source": [
    "huber = tf.keras.losses.Huber()\n",
    "model_huber = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model_huber.compile(optimizer='sgd', loss=huber)\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_huber.fit(input_regression, output_regression, epochs=500,verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum Huber Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' Huber predict: {model_huber.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7203155-e3ce-4d61-bf1c-be4ed03ff930",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyHuberLoss(Loss):\n",
    "    # initialize instance attributes\n",
    "    def __init__(self, threshold=1):\n",
    "        super().__init__()\n",
    "        self.threshold = threshold\n",
    "    # compute loss\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) <= self.threshold\n",
    "        small_error_loss = tf.square(error) / 2\n",
    "        big_error_loss = self.threshold * (tf.abs(error) - (0.5 * self.threshold))\n",
    "        return tf.where(is_small_error, small_error_loss, big_error_loss)\n",
    "model_custom = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model_custom.compile(optimizer='sgd', loss=MyHuberLoss(threshold=1.02))\n",
    "min_loss_callback = MinLossCallback()\n",
    "model_custom.fit(input_regression, output_regression, epochs=500, verbose=0, callbacks=[min_loss_callback])\n",
    "print(f\"Minimum custom Huber Error: {min_loss_callback.min_loss} at epoch {min_loss_callback.min_loss_epoch}\", f' custom huber predict: {model_custom.predict([7.0])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b3afe68-cb45-4109-a64a-f97497364399",
   "metadata": {},
   "source": [
    "<a id='5.3'></a>\n",
    "## 5.4 Batch and Batch Normalization\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67d64d58-daa4-431c-924c-eab072ec7408",
   "metadata": {},
   "source": [
    "<a id='5.5'></a>\n",
    "## 5.5 Functional and Sequential API with Shape of Networks in TensorFlow2\n",
    "\n",
    "* [List](#0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1acd0611-7ab0-47b3-8b99-7840a31ac81c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Flatten, Dense, Dropout, Lambda\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "from tensorflow.keras.datasets import fashion_mnist\n",
    "from tensorflow.keras.utils import plot_model\n",
    "from tensorflow.keras import backend as K\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image, ImageFont, ImageDraw\n",
    "import random\n",
    "def build_model_with_sequential():\n",
    "    # instantiate a Sequential class and linearly stack the layers of your model\n",
    "    seq_model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "                                            tf.keras.layers.Dense(128, activation=tf.nn.relu),\n",
    "                                            tf.keras.layers.Dropout(0.1),\n",
    "                                            tf.keras.layers.Dense(128, activation=tf.nn.relu),\n",
    "                                            tf.keras.layers.Dropout(0.1),\n",
    "                                            tf.keras.layers.Dense(128, activation=tf.nn.relu)])\n",
    "    return seq_model\n",
    "def build_model_with_functional():\n",
    "    # instantiate the input Tensor\n",
    "    input_layer = Input(shape=(28, 28,), name='base_input')\n",
    "    # stack the layers using the syntax: new_layer()(previous_layer)\n",
    "    flatten_layer = Flatten(name='flatten_input')(input_layer)\n",
    "    first_dense = Dense(128, activation='relu')(flatten_layer)\n",
    "    first_drop = Dropout(0.1, name='first_dropout')(first_dense)\n",
    "    second_dense = Dense(128, activation='relu', name='second_base_dense')(first_drop)\n",
    "    second_drop = Dropout(0.1, name='second_dropout')(second_dense)\n",
    "    output_layer = Dense(128, activation='relu')(second_drop)\n",
    "    # declare inputs and outputs\n",
    "    func_model = Model(inputs=input_layer, outputs=output_layer)\n",
    "    return func_model\n",
    "model_f = build_model_with_functional()\n",
    "model_s = build_model_with_sequential()\n",
    "# Plot model graph\n",
    "plot_model(model_s, show_shapes=True, show_layer_names=True, to_file='img/model_s.png')\n",
    "plot_model(model_f, show_shapes=True, show_layer_names=True, to_file='img/model_f.png')\n",
    "print(model_s.summary())\n",
    "print(model_f.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff69517-ff73-4224-b033-d7111a14ea6a",
   "metadata": {},
   "source": [
    "### 5.5.1 Multi-input layer networks (Y-shape)\n",
    "\n",
    "<img src=\"img/Y_shape_model.png\" alt=\"drawing\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb22e2ff-3e70-446f-ad6c-37c2beebf19f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_pairs(x, digit_indices):\n",
    "    pairs = []\n",
    "    labels = []\n",
    "    n = min([len(digit_indices[d]) for d in range(10)]) - 1\n",
    "    for d in range(10):\n",
    "        for i in range(n):\n",
    "            z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]\n",
    "            pairs += [[x[z1], x[z2]]]\n",
    "            inc = random.randrange(1, 10)\n",
    "            dn = (d + inc) % 10\n",
    "            z1, z2 = digit_indices[d][i], digit_indices[dn][i]\n",
    "            pairs += [[x[z1], x[z2]]]\n",
    "            labels += [1, 0]  \n",
    "    return np.array(pairs), np.array(labels)\n",
    "def create_pairs_on_set(images, labels):\n",
    "    \n",
    "    digit_indices = [np.where(labels == i)[0] for i in range(10)]\n",
    "    pairs, y = create_pairs(images, digit_indices)\n",
    "    y = y.astype('float32')\n",
    "    return pairs, y\n",
    "def show_image(image):\n",
    "    plt.figure()\n",
    "    plt.imshow(image)\n",
    "    plt.colorbar()\n",
    "    plt.grid(False)\n",
    "    plt.show()\n",
    "def eucl_dist(vects):\n",
    "    x,y = vects\n",
    "    sum_square=K.sum(K.square(x-y), axis=1, keepdims=True)\n",
    "    return K.sqrt(K.maximum(sum_square, K.epsilon()))\n",
    "def eucl_dist_out_shape(shape):\n",
    "    shape1, shape2 = shapes\n",
    "    return (shape1[0], 1)\n",
    "def contrastive_loss_with_margin(margin):\n",
    "    def contrastive_loss(y_true, y_pred):\n",
    "        square_pred = K.square(y_pred)\n",
    "        margin_square = K.square(K.maximum(margin - y_pred, 0))\n",
    "        return (y_true * square_pred + (1 - y_true) * margin_square)\n",
    "    return contrastive_loss\n",
    "def compute_accuracy(y_true, y_pred):\n",
    "    pred = y_pred.ravel() < 0.5\n",
    "    return np.mean(pred == y_true)\n",
    "def plot_metrics(metric_name, title, ylim=5):\n",
    "    plt.title(title)\n",
    "    plt.ylim(0,ylim)\n",
    "    plt.plot(history.history[metric_name],color='blue',label=metric_name)\n",
    "    plt.plot(history.history['val_' + metric_name],color='green',label='val_' + metric_name)\n",
    "def visualize_images():\n",
    "    plt.rc('image', cmap='gray_r')\n",
    "    plt.rc('grid', linewidth=0)\n",
    "    plt.rc('xtick', top=False, bottom=False, labelsize='large')\n",
    "    plt.rc('ytick', left=False, right=False, labelsize='large')\n",
    "    plt.rc('axes', facecolor='F8F8F8', titlesize=\"large\", edgecolor='white')\n",
    "    plt.rc('text', color='a8151a')\n",
    "    plt.rc('figure', facecolor='F0F0F0')# Matplotlib fonts\n",
    "def display_images(left, right, predictions, labels, title, n):\n",
    "    plt.figure(figsize=(17,3))\n",
    "    plt.title(title)\n",
    "    plt.yticks([])\n",
    "    plt.xticks([])\n",
    "    plt.grid(None)\n",
    "    left = np.reshape(left, [n, 28, 28])\n",
    "    left = np.swapaxes(left, 0, 1)\n",
    "    left = np.reshape(left, [28, 28*n])\n",
    "    plt.imshow(left)\n",
    "    plt.figure(figsize=(17,3))\n",
    "    plt.yticks([])\n",
    "    plt.xticks([28*x+14 for x in range(n)], predictions)\n",
    "    for i,t in enumerate(plt.gca().xaxis.get_ticklabels()):\n",
    "        if predictions[i] > 0.5: t.set_color('red') # bad predictions in red\n",
    "    plt.grid(None)\n",
    "    right = np.reshape(right, [n, 28, 28])\n",
    "    right = np.swapaxes(right, 0, 1)\n",
    "    right = np.reshape(right, [28, 28*n])\n",
    "    plt.imshow(right)\n",
    "# load the dataset\n",
    "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
    "# prepare train and test sets\n",
    "train_images = train_images.astype('float32')\n",
    "test_images = test_images.astype('float32')\n",
    "# normalize values\n",
    "train_images = train_images / 255.0\n",
    "test_images = test_images / 255.0\n",
    "# create pairs on train and test sets\n",
    "tr_pairs, tr_y = create_pairs_on_set(train_images, train_labels)\n",
    "ts_pairs, ts_y = create_pairs_on_set(test_images, test_labels)\n",
    "\n",
    "# create the left input and point to the base network\n",
    "input_a = Input(shape=(28,28,), name=\"left_input\")\n",
    "vect_output_a = model_f(input_a)\n",
    "# create the right input and point to the base network\n",
    "input_b = Input(shape=(28,28,), name=\"right_input\")\n",
    "vect_output_b = model_f(input_b)\n",
    "# measure the similarity of the two vector outputs\n",
    "output = Lambda(eucl_dist, name=\"output_layer\", output_shape=eucl_dist_out_shape)([vect_output_a, vect_output_b])\n",
    "# specify the inputs and output of the model\n",
    "y_model = Model([input_a, input_b], output)\n",
    "\n",
    "# plot model graph\n",
    "plot_model(y_model, show_shapes=True, show_layer_names=True, to_file='img/Y_shape_model.png')\n",
    "rms = RMSprop()\n",
    "y_model.compile(loss=contrastive_loss_with_margin(margin=1), optimizer=rms)\n",
    "history = y_model.fit([tr_pairs[:,0], tr_pairs[:,1]], tr_y, epochs=20, batch_size=128, validation_data=([ts_pairs[:,0], ts_pairs[:,1]], ts_y), verbose=0)\n",
    "\n",
    "loss = y_model.evaluate(x=[ts_pairs[:,0],ts_pairs[:,1]], y=ts_y)\n",
    "y_pred_train = y_model.predict([tr_pairs[:,0], tr_pairs[:,1]], verbose=0)\n",
    "train_accuracy = compute_accuracy(tr_y, y_pred_train)\n",
    "y_pred_test = y_model.predict([ts_pairs[:,0], ts_pairs[:,1]], verbose=0)\n",
    "test_accuracy = compute_accuracy(ts_y, y_pred_test)\n",
    "\n",
    "print(\"Loss = {}, Train Accuracy = {} Test Accuracy = {}\".format(loss, train_accuracy, test_accuracy))\n",
    "plot_metrics(metric_name='loss', title=\"Loss\", ylim=0.2)\n",
    "y_pred_train = np.squeeze(y_pred_train)\n",
    "indexes = np.random.choice(len(y_pred_train), size=10)\n",
    "display_images(tr_pairs[:, 0][indexes], tr_pairs[:, 1][indexes], y_pred_train[indexes], tr_y[indexes], \"clothes and their dissimilarity\", 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "241d57a7-cd9c-4fc7-a30a-11564a9296f5",
   "metadata": {},
   "source": [
    "### 5.5.2 Deep Wide Layer networks\n",
    "\n",
    "<img src=\"img/WideDeep_model.png\" alt=\"drawing\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "404f047a-3a2b-4c06-92c7-ab3df88a9cf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, Input, concatenate\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "class DeepAndWideModel:\n",
    "    def __init__(self):\n",
    "        self.model = self.build_model()\n",
    "    def build_model(self):\n",
    "        input_a = Input(shape=[1], name=\"Wide_Input\")\n",
    "        input_b = Input(shape=[1], name=\"Deep_Input\")\n",
    "\n",
    "        hidden_1 = Dense(30, activation=\"relu\")(input_b)\n",
    "        hidden_2 = Dense(30, activation=\"relu\")(hidden_1)\n",
    "\n",
    "        concat = concatenate([input_a, hidden_2])\n",
    "        output = Dense(1, name=\"Output\")(concat)\n",
    "\n",
    "        aux_output = Dense(1, name=\"aux_Output\")(hidden_2)\n",
    "\n",
    "        model = Model(inputs=[input_a, input_b], outputs=[output, aux_output])\n",
    "        return model\n",
    "    def visualize_model(self):\n",
    "        plot_model(self.model, show_shapes=True, show_layer_names=True, to_file='img/WideDeep_model.png')\n",
    "deep_and_wide_model = DeepAndWideModel()\n",
    "deep_and_wide_model.visualize_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d68420da-824a-48ab-9505-b3749cd10470",
   "metadata": {},
   "source": [
    "### 5.5.3 Residual Networks\n",
    "\n",
    "<img src=\"img/Resnet_model.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92525918-695b-4061-adea-aa3f5a3b9f93",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Input, Conv2D, BatchNormalization, Activation, Add, MaxPool2D, GlobalAveragePooling2D, Dense\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "def identity_block(input_tensor, filters, kernel_size):\n",
    "    x = Conv2D(filters, kernel_size, padding='same')(input_tensor)\n",
    "    x = BatchNormalization()(x)\n",
    "    x = Activation('relu')(x)\n",
    "    \n",
    "    x = Conv2D(filters, kernel_size, padding='same')(x)\n",
    "    x = BatchNormalization()(x)\n",
    "    \n",
    "    x = Add()([x, input_tensor])\n",
    "    x = Activation('relu')(x)\n",
    "    return x\n",
    "\n",
    "def resnet_model(input_shape, num_classes):\n",
    "    inputs = Input(shape=input_shape)\n",
    "    \n",
    "    x = Conv2D(64, 7, padding='same')(inputs)\n",
    "    x = BatchNormalization()(x)\n",
    "    x = Activation('relu')(x)\n",
    "    x = MaxPool2D((3, 3))(x)\n",
    "    # insert the identity blocks in the middle of the network\n",
    "    x = identity_block(x, 64, 3)\n",
    "    x = identity_block(x, 64, 3)\n",
    "    \n",
    "    x = GlobalAveragePooling2D()(x)\n",
    "    outputs = Dense(num_classes, activation='softmax')(x)\n",
    "    \n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    return model\n",
    "\n",
    "resnet = resnet_model((28, 28, 1), 10)\n",
    "#plot_model(resnet, show_shapes=True, show_layer_names=True, to_file='img/Resnet_model.png')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2405a8f-9116-464d-8e15-7030eea7a1a6",
   "metadata": {},
   "source": [
    "### 5.5.4 VGG Networks\n",
    "\n",
    "It is primarily made up of a series of Conv2D layers followed by a softmax activated layers to classify the image. As you can see, this will be a handful and the code will look huge if you specify each layer individually.\\\n",
    "<img src=\"img/VGG.png\" alt=\"drawing\"/>\\\n",
    "<img src=\"img/VGG_model.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eb64a9d-694e-42f4-96f4-09e35e4b817c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "def vgg_block(input_tensor, filters, kernel_size, repetitions):\n",
    "    x = input_tensor\n",
    "    # Define a conv2D_0, conv2D_1, etc based on the number of repetitions\n",
    "    for _ in range(repetitions):\n",
    "        # Define a Conv2D layer, specifying filters, kernel_size, activation and padding.\n",
    "        x = Conv2D(filters, kernel_size, activation='relu', padding='same')(x)\n",
    "    # Define the max pool layer that will be added after the Conv2D blocks\n",
    "    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(x)\n",
    "    return x\n",
    "\n",
    "def vgg_model(input_shape, num_classes):\n",
    "    inputs = Input(shape=input_shape)\n",
    "    # Creating blocks of VGG with the following \n",
    "    # (filters, kernel_size, repetitions) configurations\n",
    "    x = vgg_block(inputs, 64, 3, 2)\n",
    "    x = vgg_block(x, 128, 3, 2)\n",
    "    x = vgg_block(x, 256, 3, 3)\n",
    "    x = vgg_block(x, 512, 3, 3)\n",
    "    x = vgg_block(x, 512, 3, 3)\n",
    "    \n",
    "    # Classification head\n",
    "    # Define a Flatten layer\n",
    "    x = Flatten()(x)\n",
    "    # Create a Dense layer with 256 units and ReLU as the activation function\n",
    "    x = Dense(256, activation='relu')(x)\n",
    "    # Finally add the softmax classifier using a Dense layer\n",
    "    outputs = Dense(num_classes, activation='softmax')(x)\n",
    "    \n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    return model\n",
    "\n",
    "vgg = vgg_model((224, 224, 3), 2)\n",
    "#plot_model(vgg, show_shapes=True, show_layer_names=True, to_file='img/vgg_model.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ac7d0376",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1000])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "vgg_types = {\n",
    "    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],\n",
    "    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],\n",
    "    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],\n",
    "    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M']\n",
    "}\n",
    "# then flaten and 4096x4096x1000 linear layers\n",
    "\n",
    "class vgg_net(nn.Module):\n",
    "    def __init__(self,in_channels, num_classes):\n",
    "        super(vgg_net,self).__init__()\n",
    "        self.in_channels=in_channels\n",
    "        self.conv_layers=self.create_conv_layers(vgg_types['VGG19'])\n",
    "        self.fcs=nn.Sequential(\n",
    "            nn.Linear(512*7*7, 4096),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(p=0.5),\n",
    "            nn.Linear(4096, 4096),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(p=0.5),\n",
    "            nn.Linear(4096, num_classes)\n",
    "            )\n",
    "    def forward(self,x):\n",
    "        x=self.conv_layers(x)\n",
    "        x=x.reshape(x.shape[0], -1)\n",
    "        x=self.fcs(x)\n",
    "        return x\n",
    "    \n",
    "    def create_conv_layers(self,architecture):\n",
    "        layers=[]\n",
    "        in_channels=self.in_channels\n",
    "        for x in architecture:\n",
    "            if type(x)==int:\n",
    "                out_channels=x\n",
    "                layers+=[nn.Conv2d(in_channels=in_channels,out_channels=out_channels,\n",
    "                                   kernel_size=(3,3),stride=(1,1),padding=(1,1)),\n",
    "                        nn.BatchNorm2d(x),\n",
    "                        nn.ReLU()]\n",
    "                in_channels=x\n",
    "            elif x=='M':\n",
    "                layers+=[nn.MaxPool2d(kernel_size=(2,2),stride=(2,2))]\n",
    "        return nn.Sequential(*layers)\n",
    "\n",
    "device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model=vgg_net(in_channels=3, num_classes=1000).to(device)\n",
    "x=torch.randn(1,3,224,224).to(device)\n",
    "print(model(x).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db76f14c-ac36-4177-a711-7ae7b15af4b2",
   "metadata": {},
   "source": [
    "### 5.5.5 Siamese networks\n",
    "\n",
    "<img src=\"img/Siamese_model.png\" alt=\"drawing\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5659ca57-d5d8-46a9-bdc0-cf49b5c9c13b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Lambda\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras import backend as K\n",
    "\n",
    "def create_base_network(input_shape):\n",
    "    inputs = Input(shape=input_shape)\n",
    "    x = Conv2D(64, (3, 3), activation='relu')(inputs)\n",
    "    x = MaxPooling2D(pool_size=(2, 2))(x)\n",
    "    x = Conv2D(128, (3, 3), activation='relu')(x)\n",
    "    x = MaxPooling2D(pool_size=(2, 2))(x)\n",
    "    x = Flatten()(x)\n",
    "    x = Dense(256, activation='relu')(x)\n",
    "    return Model(inputs=inputs, outputs=x)\n",
    "\n",
    "def euclidean_distance(vectors):\n",
    "    x, y = vectors\n",
    "    sum_square = K.sum(K.square(x - y), axis=1, keepdims=True)\n",
    "    return K.sqrt(K.maximum(sum_square, K.epsilon()))\n",
    "\n",
    "input_shape = (105, 105, 1)\n",
    "base_network = create_base_network(input_shape)\n",
    "\n",
    "input_a = Input(shape=input_shape, name='input_a')\n",
    "input_b = Input(shape=input_shape, name='input_b')\n",
    "\n",
    "processed_a = base_network(input_a)\n",
    "processed_b = base_network(input_b)\n",
    "\n",
    "distance = Lambda(euclidean_distance, name='euclidean_distance')([processed_a, processed_b])\n",
    "\n",
    "siamese_net = Model(inputs=[input_a, input_b], outputs=distance)\n",
    "\n",
    "#plot_model(siamese_net, show_shapes=True, show_layer_names=True, to_file='img/siamese_model.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58187426-ce21-4b4b-b36d-7520e7cf0f7f",
   "metadata": {},
   "source": [
    "## 5.6 Callbacks\n",
    "callback is a set of functions that can be applied at various stages during the training process of a model. Callbacks allow you to customize the behavior of your model during training, evaluation, or inference. You can use callbacks for various purposes, such as monitoring model performance, saving the model at specific intervals, stopping training early if the model stops improving, or adjusting learning rates.\\\n",
    "1. Monitoring model performance: Track the model's performance using metrics such as accuracy or loss, and log them for later analysis.\n",
    "2. Model checkpointing: Save the model at regular intervals or when it achieves its best performance, allowing you to resume training from a saved model or use the best model for inference.\n",
    "3. Early stopping: Stop training when the model stops improving, preventing overfitting and reducing the training time.\n",
    "4. Adjusting learning rate: Dynamically change the learning rate during training based on the model's performance or schedule.\n",
    "5. Custom behavior: Implement custom behavior during training, such as logging specific variables or performing custom operations at specific intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7b58abf-1634-4a83-bd26-704e8a8d36c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\n",
    "\n",
    "# Define callbacks\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=5)\n",
    "model_checkpoint = ModelCheckpoint('best_model.h5', save_best_only=True)\n",
    "reduce_lr_on_plateau = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3)\n",
    "\n",
    "# Train the model with callbacks\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=100, callbacks=[early_stopping, model_checkpoint, reduce_lr_on_plateau])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16169126-1c2d-4148-9210-c342d7baa953",
   "metadata": {},
   "source": [
    "<a id='5.5'></a>\n",
    "## 5.7 Gradient Tape in TensorFlow2\n",
    "\n",
    "* [List](#0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48ef6dc8-fc77-49f6-9053-2e6d2a2a0360",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "852e02ba-1f98-4350-b813-84cf3d947ed4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74c3bb6b-5b34-4abf-bbe3-3581da2e8496",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1394627-22b1-4090-a6e2-9022dd3f3a73",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "10426561-c028-4cac-8b44-234101ca6919",
   "metadata": {},
   "source": [
    "<a id='7'></a>\n",
    "# 7. Reinforcement Learning\n",
    "\n",
    "* [List](#0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa5f4200-0b05-43bf-9913-09fc1e394bcc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9019553d-1330-404c-bd4a-c23fa124b175",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb7a2462-abdd-4b01-b399-837155ddc8ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d890ac8-c7ea-4250-a4df-03284806fd13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "left = cv2.cvtColor(cv2.imread('img/left.png'),cv2.COLOR_BGR2GRAY)\n",
    "plt.imshow(left,cmap='gray')\n",
    "print(left.shape)\n",
    "left2=left[270:275,223:228]\n",
    "print(left2)\n",
    "plt.imshow(left2,cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc1c9487-874c-4fce-9e9f-82ccb3fb5726",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "left = cv2.cvtColor(cv2.imread('img/left.png'),cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(left)\n",
    "print(left.shape)\n",
    "left2=left[270:275,223:228,:]\n",
    "print(left2)\n",
    "plt.imshow(left2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f81dc2f6-f2fa-445e-9238-9bfa7ea3cfb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "def unitvec(n):\n",
    "    x=n[0]\n",
    "    y=n[1]\n",
    "    z=n[2]\n",
    "    mag=np.sqrt(x**2+y**2+z**2)\n",
    "    if x ==0 and y == 0 and z == 0:\n",
    "        return 0,0,0,(0,0,0)\n",
    "    return np.array([0,x/mag]), np.array([0,y/mag]), np.array([0,z/mag]) ,(x/255,y/255,z/255)\n",
    "# Create figure and 3D axes\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "\n",
    "# Plot the line\n",
    "ax.plot(unitvec(left2[4,4,:])[0], unitvec(left2[4,4,:])[1], unitvec(left2[4,4,:])[2],c=unitvec(left2[4,4,:])[3], label='4,4')\n",
    "ax.plot(unitvec(left2[3,4,:])[0], unitvec(left2[3,4,:])[1], unitvec(left2[3,4,:])[2],c=unitvec(left2[3,4,:])[3], label='3,4')\n",
    "ax.plot(unitvec(left2[2,2,:])[0], unitvec(left2[2,2,:])[1], unitvec(left2[2,2,:])[2],c=unitvec(left2[2,2,:])[3], label='2,2')\n",
    "ax.plot(unitvec(left2[0,1,:])[0], unitvec(left2[0,1,:])[1], unitvec(left2[0,1,:])[2],c=unitvec(left2[0,1,:])[3], label='0,1')\n",
    "ax.plot(unitvec(left2[0,0,:])[0], unitvec(left2[0,0,:])[1], unitvec(left2[0,0,:])[2],c=unitvec(left2[0,0,:])[3], label='0,0')\n",
    "ax.plot(unitvec([255,255,255])[0], unitvec([255,255,255])[1], unitvec([255,255,255])[2],c=unitvec([255,255,255])[3], label='mid')\n",
    "\n",
    "# Set axis labels and title\n",
    "ax.set_xlabel('X')\n",
    "ax.set_ylabel('Y')\n",
    "ax.set_zlabel('Z')\n",
    "ax.set_title('3D Line Plot')\n",
    "ax.legend()\n",
    "ax.view_init(30, 50, 0)\n",
    "# Show the plot\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e643c9a8-c8fb-45af-b0ee-49c70f3fc789",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "\n",
    "def upscale_video(input_video, output_video, scale_factor=2):\n",
    "    # Open the input video\n",
    "    video = cv2.VideoCapture(input_video)\n",
    "    fps = int(video.get(cv2.CAP_PROP_FPS))\n",
    "    width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH) * scale_factor)\n",
    "    height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT) * scale_factor)\n",
    "\n",
    "    # Create the output video\n",
    "    fourcc = cv2.VideoWriter_fourcc(*\"mp4v\")\n",
    "    out = cv2.VideoWriter(output_video, fourcc, fps, (width, height))\n",
    "\n",
    "    while True:\n",
    "        ret, frame = video.read()\n",
    "        if not ret:\n",
    "            break\n",
    "\n",
    "        # Upscale the frame using INTER_CUBIC interpolation\n",
    "        upscaled_frame = cv2.resize(frame, (width, height), interpolation=cv2.INTER_CUBIC)\n",
    "        out.write(upscaled_frame)\n",
    "\n",
    "    video.release()\n",
    "    out.release()\n",
    "\n",
    "input_video = \"vid/race_car.mp4\"\n",
    "output_video = \"vid/upscaled_video.mp4\"\n",
    "\n",
    "upscale_video(input_video, output_video)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89f07ae2-a868-484c-af48-6d63629bfebb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c48b264-5944-4eac-b3bf-487d05283802",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "186169a2-09ec-49a2-b542-5273162f280b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0209e58-9515-427e-9098-48a39bead56b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c299ac-b482-4446-87f2-0b19554635c3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c0b2f8e-09b1-4a66-b3ea-33ed5acd66b8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "general",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}