Maximum_likelihood_estimation (2).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Objectives\n",
        "\n",
        "* Introduce the Maximum likelihood estimation\n",
        "* MLE as a general principle to estimate the parameters of a NN\n",
        "* Example of using MLE to estimate the parameters of a model : logistic regression\n"
      ],
      "metadata": {
        "id": "hFn2I6EECBC6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Maximum likelihood estimation\n",
        "\n",
        "Maximum likelihood estimation (MLE) is a popular statistical method for estimating the parameters of a probability distribution from a set of observations or data. The basic idea behind MLE is to find the values of the parameters that maximize the likelihood of observing the data given the probability distribution. In other words, MLE seeks to find the set of parameters that make the observed data most probable.\n",
        "\n",
        "To illustrate MLE in Python, let's consider an example. Suppose we have a set of observations of a random variable that follows a normal distribution with unknown mean and variance. Our goal is to estimate the values of the mean and variance that best fit the observed data. We can use MLE to find the maximum likelihood estimates of these parameters.\n",
        "\n",
        "We can start by defining the probability density function (PDF) of the normal distribution, which depends on the mean and variance parameters. In Python, we can use the scipy.stats.norm module to define the PDF:"
      ],
      "metadata": {
        "id": "7ThTI-XDQHNU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from scipy.stats import norm\n",
        "\n",
        "def likelihood(x, mu, sigma):\n",
        "    return np.prod(norm.pdf(x, loc=mu, scale=sigma))"
      ],
      "metadata": {
        "id": "K0d7BX-SQoLX"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we can define the likelihood function, which is the product of the PDF values for each observed data point. In Python, we can define the likelihood function as:"
      ],
      "metadata": {
        "id": "csqCcoH_QsEO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def log_likelihood(x, mu, sigma):\n",
        "    return np.sum(norm.logpdf(x, loc=mu, scale=sigma))"
      ],
      "metadata": {
        "id": "iXEJ39YHQxGH"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we can use an optimization algorithm such as scipy.optimize.minimize to find the maximum likelihood estimates of the mean and variance parameters. Here's an example code that uses MLE to estimate the mean and variance of a set of data:"
      ],
      "metadata": {
        "id": "JgbaE6tFQ6AB"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "zqeBtIiPQ4R4"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy import stats\n",
        "import numpy as np\n",
        "from scipy.optimize import minimize\n",
        "\n",
        "# Set random seed for reproducibility\n",
        "np.random.seed(1)\n",
        "\n",
        "# Number of data points\n",
        "n = 20\n",
        "\n",
        "mean = 0\n",
        "\n",
        "std = 3 \n",
        "\n",
        "# Generate a random sample from a normal distribution\n",
        "sample_data = np.random.normal(loc=mean, scale=std, size=n)\n",
        "\n",
        "def negative_log_likelihood(params):\n",
        "    # Extract the mean and standard deviation from the parameter vector\n",
        "    mean = params[0]\n",
        "    sd = params[1]\n",
        "\n",
        "    # Calculate the negative log-likelihood\n",
        "    nll = -np.sum(stats.norm.logpdf(sample_data, loc=mean, scale=sd))\n",
        "\n",
        "    return nll\n",
        "\n",
        "# Initial parameter values\n",
        "initParams = [1, 1]\n",
        "\n",
        "# Minimize the negative log-likelihood using the Nelder-Mead method\n",
        "results = minimize(negative_log_likelihood, initParams, method='Nelder-Mead')\n",
        "\n",
        "# Extract the estimated parameters from the optimization results\n",
        "estimated_mean, estimated_std = results.x\n",
        "\n",
        "# Print the estimated parameters\n",
        "\n",
        "print(\"original Mean:\", mean)\n",
        "print(\"Orignal Standard Deviation:\", std)\n",
        "\n",
        "print(\"Estimated Mean:\", estimated_mean)\n",
        "print(\"Estimated Standard Deviation:\", estimated_std)\n"
      ],
      "metadata": {
        "id": "-Uv3g-09Qox4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c1befcd4-1c70-4f02-e2dd-c1b447e7fa3a"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Estimated Mean: -0.40005354794526626\n",
            "Estimated Standard Deviation: 3.299413469903742\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this code, we import the required libraries and set the random seed for reproducibility.\n",
        "\n",
        "Next, we generate a random sample (sample_data) of size n from a normal distribution with a mean of 0 and a standard deviation of 3.\n",
        "\n",
        "\n",
        "\n",
        "The gaussian function takes a parameter vector as input and calculates the negative log-likelihood by summing the logarithm of the probability density function of the normal distribution (stats.norm.logpdf) for each data point in sample_data.\n",
        "\n",
        "\n",
        "\n",
        "We initialize the parameter values (initParams) as [1, 1].\n",
        "\n",
        "\n",
        "Using the minimize function from SciPy's optimization module, we minimize the negative log-likelihood function (gaussian) using the Nelder-Mead method.\n",
        "\n",
        "\n",
        "\n",
        "The estimated mean and standard deviation are extracted from the optimization results (results.x).\n",
        "\n",
        "\n",
        "Finally, we print the estimated mean and standard deviation.\n",
        "\n",
        "When you run this code, you will see the estimated mean and standard deviation printed in the console. These values represent the MLE estimates of the mean and standard deviation based on the observed data."
      ],
      "metadata": {
        "id": "sgOl_TZ_CKeo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MLE and estimating the parameters of a general neural network"
      ],
      "metadata": {
        "id": "LsdFAAsxCU_P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now lets us see how to use MLE to estimate the parameters of a general neural network. We will give the general steps and then we provide example using python and a simple logistic regression model.\n",
        "\n",
        "Neural Network Architecture: Consider a neural network with parameters θ that consists of an input layer, one or more hidden layers, and an output layer. Each layer contains nodes or neurons, and the connections between nodes are weighted by parameters θ.\n",
        "\n",
        "(1) Likelihood Function: Let's assume we have a training dataset D = {(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)}, where x_i represents the input data and y_i represents the corresponding target output. The likelihood function L(θ|D) represents the joint probability of observing all the target outputs y given the input data x and the model parameters θ.\n",
        "\n",
        "(2) Output Distribution: In the context of a neural network, the output values are typically modeled using a probability distribution, such as a Gaussian (normal) distribution. The choice of the distribution depends on the nature of the problem and the desired behavior of the network.\n",
        "\n",
        "(3) Log-Likelihood Function: Taking the logarithm of the likelihood function, we obtain the log-likelihood function ℓ(θ|D), which simplifies the calculations and allows us to work with sums instead of products:\n",
        "\n",
        "ℓ(θ|D) = log L(θ|D)\n",
        "\n",
        "(4) Loss Function: The negative log-likelihood function becomes the loss function for training the neural network. We denote it as J(θ|D) and define it as the negative of the log-likelihood:\n",
        "\n",
        "J(θ|D) = -ℓ(θ|D)\n",
        "\n",
        "The goal is to minimize this loss function by adjusting the network parameters θ.\n",
        "\n",
        "(5) Optimization: Optimization algorithms, such as Stochastic Gradient Descent (SGD) or its variants, are used to minimize the loss function J(θ|D) and find the optimal parameter values. During training, the algorithm iteratively adjusts the parameters θ in the direction that reduces the loss function, guided by the gradients of the loss with respect to the parameters.\n",
        "\n",
        "(6) Backpropagation: Backpropagation is used to compute the gradients of the loss function J(θ|D) with respect to the network parameters θ. It calculates the gradients by propagating the errors from the output layer to the input layer of the network. The chain rule is applied to compute the gradients at each layer based on the gradients of the subsequent layers.\n",
        "\n",
        "(7) Parameter Updates: The gradients computed through backpropagation are used to update the network parameters θ. The specific update rule depends on the chosen optimization algorithm. Typically, the parameters are adjusted in the direction that minimizes the loss function using a learning rate α, which controls the step size taken in the parameter space during optimization.\n",
        "\n",
        "(8) Convergence: The training process continues until a stopping criterion is met, such as reaching a maximum number of iterations or achieving satisfactory performance on a validation set. At this point, the estimated parameters θ represent the maximum likelihood estimates for the given neural network architecture."
      ],
      "metadata": {
        "id": "Xui8Z_d1Cqun"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MLE using logistic regression using explicit computations\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "riRNpBpMHla9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "# Step 1: Define the Logistic Regression Model\n",
        "class LogisticRegression:\n",
        "    def __init__(self, learning_rate=0.01, num_iterations=1000):\n",
        "        self.learning_rate = learning_rate\n",
        "        self.num_iterations = num_iterations\n",
        "        self.weights = None\n",
        "        self.bias = None\n",
        "\n",
        "    def sigmoid(self, z):\n",
        "        return 1 / (1 + np.exp(-z))\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        num_samples, num_features = X.shape\n",
        "\n",
        "        # Initialize the weights and bias to zeros\n",
        "        self.weights = np.zeros((num_features, 1))\n",
        "        self.bias = 0\n",
        "\n",
        "        # Gradient Descent\n",
        "        for iteration in range(self.num_iterations):\n",
        "            # Forward pass\n",
        "            linear_model = np.dot(X, self.weights) + self.bias\n",
        "            predictions = self.sigmoid(linear_model)\n",
        "\n",
        "            # Compute gradients\n",
        "            # this step is equivalent to the step of minimizing the nll\n",
        "            # nll = -sum(y * log(p) + (1-y) * log(1-p))\n",
        "            # nll represents the negative log-likelihood.\n",
        "            # y is the actual label (0 or 1) for each sample.\n",
        "            # p is the predicted probability for the corresponding sample.\n",
        "            # see remark (1) below  \n",
        "\n",
        "            # this step is \"manual differentiation\" will be replaced by automatic differentiation later using the full\n",
        "            # functionality of Pytorch\n",
        "            dw = (1 / num_samples) * np.dot(X.T, (predictions - y))\n",
        "            db = (1 / num_samples) * np.sum(predictions - y)\n",
        "              \n",
        "            # Update parameters\n",
        "            # this step using \"manual update of the parameters\" will be replaced by better method using specialized optimizers to update the parameters\n",
        "            self.weights -= self.learning_rate * dw.reshape(-1, 1)\n",
        "            self.bias -= self.learning_rate * db\n",
        "\n",
        "    def predict(self, X):\n",
        "        linear_model = np.dot(X, self.weights) + self.bias\n",
        "        predictions = self.sigmoid(linear_model)\n",
        "        return np.round(predictions)\n",
        "\n",
        "# Step 2: Generate Made-Up Data\n",
        "np.random.seed(1)\n",
        "num_samples = 100\n",
        "X = np.random.randn(num_samples, 1)\n",
        "y = np.random.randint(0, 2, size=(num_samples, 1))\n",
        "\n",
        "# Step 3: Create and Train the Logistic Regression Model\n",
        "model = LogisticRegression()\n",
        "model.fit(X, y)\n",
        "\n",
        "# Step 4: Make Predictions on New Data\n",
        "X_test = np.random.randn(10, 1)\n",
        "predictions = model.predict(X_test)\n",
        "\n",
        "print(\"Predictions:\")\n",
        "for i in range(len(X_test)):\n",
        "    print(f\"Input: {X_test[i].item()}, Predicted Class: {predictions[i].item()}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O3k4adYtHuiu",
        "outputId": "084c5074-9676-46e1-c75e-6c71d46240e0"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predictions:\n",
            "Input: 0.5205763370733708, Predicted Class: 0.0\n",
            "Input: -1.1443413896231427, Predicted Class: 0.0\n",
            "Input: 0.8018610318713447, Predicted Class: 0.0\n",
            "Input: 0.04656729842414554, Predicted Class: 0.0\n",
            "Input: -0.18656977190734877, Predicted Class: 0.0\n",
            "Input: -0.10174587252914521, Predicted Class: 0.0\n",
            "Input: 0.8688861570058679, Predicted Class: 0.0\n",
            "Input: 0.7504116398650081, Predicted Class: 0.0\n",
            "Input: 0.5294653243527092, Predicted Class: 0.0\n",
            "Input: 0.13770120999738608, Predicted Class: 0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Remark: The relation between nll and db and dw\n",
        "\n",
        "Recall\n",
        "\n",
        "\n",
        "\n",
        "*   nll = -[y * log(p) + (1-y) * log(1-p)]  \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "and \n",
        "\n",
        "dw = (1 / num_samples) * np.dot(X.T, (predictions - y))         \n",
        "\n",
        "\n",
        "db = (1 / num_samples) * np.sum(predictions - y)               \n",
        "\n",
        "\n",
        "but how exactly do we go from nll to db and dw ?"
      ],
      "metadata": {
        "id": "N39Z5hFnNcYB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To derive the gradients dw and db, we need to compute the partial derivatives of the negative log-likelihood with respect to the weights and bias. Let's start with the weight gradient dw."
      ],
      "metadata": {
        "id": "Z8q7s9zkODic"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "d_nll / dw = -[y * (1/p) * dp/dw + (1-y) * (1/(1-p)) * dp/dw]\n",
        "           = -[y * (1/p) * p * (1-p) * x + (1-y) * (1/(1-p)) * (-p) * (1-p) * x]\n",
        "           = -[y * (1-p) * x - (1-y) * p * x]\n",
        "           = x * [p - y]\n"
      ],
      "metadata": {
        "id": "TuulD4j8KUU5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Compute the weight gradient dw by summing the derivatives over all the samples and dividing by the number of samples:\n",
        "\n"
      ],
      "metadata": {
        "id": "l3TPDXNrOGfE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dw = (1/N) * sum(x * [p - y])\n"
      ],
      "metadata": {
        "id": "AdV-pJE9OIDj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's derive the bias gradient db.\n",
        "\n",
        "Compute the partial derivative of the negative log-likelihood with respect to the bias b"
      ],
      "metadata": {
        "id": "J7hFfKjJOJ_j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "d_nll / db = -[y * (1/p) * dp/db + (1-y) * (1/(1-p)) * dp/db]\n",
        "           = -[y * (1/p) * p * (1-p) + (1-y) * (1/(1-p)) * (-p) * (1-p)]\n",
        "           = -[y * (1-p) - (1-y) * p]\n",
        "           = p - y\n"
      ],
      "metadata": {
        "id": "s2YmTSkZONQL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Compute the bias gradient db by summing the derivatives over all the samples and dividing by the number of samples:"
      ],
      "metadata": {
        "id": "3t_RqRk8ORQj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "db = (1/N) * sum(p - y)\n"
      ],
      "metadata": {
        "id": "aiejgO29OR3M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the code, these gradients are computed within the fit method of the LogisticRegression class using matrix operations to handle multiple samples efficiently. The gradients dw and db are used to update the weights and bias iteratively during gradient descent, which minimizes the negative log-likelihood and improves the logistic regression model's predictions."
      ],
      "metadata": {
        "id": "1Ks7OzZNOXED"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MLE using logistic regression using Pytroch modern functionalities\n",
        "\n",
        "In this example, we build a logistic regression model using PyTorch but this time we use nn.Module. We define a simple neural network with a single linear layer and a sigmoid activation function. We use the binary cross-entropy loss (BCELoss) as the log-likelihood loss function. We then generate some random made-up data and train the model using gradient descent (SGD) optimization. Finally, we retrieve the estimated parameters of the model.\n",
        "\n",
        "Note that logistic regression is a simple linear model, and using a neural network for this purpose is not necessary. However, this example demonstrates how the steps of MLE can be applied in the context of neural networks."
      ],
      "metadata": {
        "id": "SS1PjrLSOcMp"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "TCxtwd3ZOPGi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import numpy as np\n",
        "\n",
        "# Step 1: Define the Model Architecture\n",
        "class LogisticRegression(nn.Module):\n",
        "    def __init__(self, input_size):\n",
        "        super(LogisticRegression, self).__init__()\n",
        "        self.linear = nn.Linear(input_size, 1)\n",
        "        self.sigmoid = nn.Sigmoid()\n",
        "\n",
        "    def forward(self, x):\n",
        "        out = self.linear(x)\n",
        "        out = self.sigmoid(out)\n",
        "        return out\n",
        "\n",
        "# Step 2: Create the Model and Define the Likelihood Function\n",
        "input_size = 1  # Number of input features\n",
        "model = LogisticRegression(input_size)\n",
        "\n",
        "# Step 3: Define Loss Function (Log-Likelihood)\n",
        "criterion = nn.BCELoss()\n",
        "\n",
        "# Step 4: Define Optimization Algorithm\n",
        "learning_rate = 0.01\n",
        "optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
        "\n",
        "# Step 5: Generate Made-Up Data\n",
        "np.random.seed(1)\n",
        "num_samples = 100\n",
        "X = np.random.randn(num_samples, input_size)\n",
        "y = np.random.randint(0, 2, size=(num_samples, 1))\n",
        "X = torch.from_numpy(X).float()\n",
        "y = torch.from_numpy(y).float()\n",
        "\n",
        "# Step 6: Training Loop\n",
        "num_epochs = 100\n",
        "for epoch in range(num_epochs):\n",
        "    # Forward Pass\n",
        "    outputs = model(X)\n",
        "    \n",
        "    # the power of automatic differntiation : we do not need to differentiate \n",
        "    \n",
        "\n",
        "    # Compute Loss\n",
        "    loss = criterion(outputs, y)\n",
        "    \n",
        "    # the optimizer: we do not need to update the parameters manually\n",
        "\n",
        "    # Backward Pass and Optimization\n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    \n",
        "    # Print Loss for Monitoring\n",
        "    if (epoch+1) % 10 == 0:\n",
        "        print(f\"Epoch: {epoch+1}, Loss: {loss.item()}\")\n",
        "\n",
        "# Step 7: Retrieve the Estimated Parameters\n",
        "estimated_params = model.state_dict()\n",
        "print(\"Estimated Parameters:\")\n",
        "for param_name, param_value in estimated_params.items():\n",
        "    print(f\"{param_name}: {param_value.item()}\")\n",
        "\n",
        "# Step 8: Generate Test Data\n",
        "X_test = np.random.randn(10, input_size)\n",
        "X_test = torch.from_numpy(X_test).float()\n",
        "\n",
        "# Step 9: Make Predictions on Test Data\n",
        "model.eval()\n",
        "with torch.no_grad():\n",
        "    outputs_test = model(X_test)\n",
        "    predictions = (outputs_test >= 0.5).float()\n",
        "\n",
        "print(\"\\nTest Predictions:\")\n",
        "for i in range(len(X_test)):\n",
        "    print(f\"Input: {X_test[i].item()}, Predicted Probability: {outputs_test[i].item()}, Predicted Class: {predictions[i].item()}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EsRMxtFCCJzR",
        "outputId": "9a32f32d-2c66-4469-d30d-2ada22e36aec"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 10, Loss: 0.7045889496803284\n",
            "Epoch: 20, Loss: 0.7033473253250122\n",
            "Epoch: 30, Loss: 0.7021617293357849\n",
            "Epoch: 40, Loss: 0.7010295391082764\n",
            "Epoch: 50, Loss: 0.699948251247406\n",
            "Epoch: 60, Loss: 0.6989158391952515\n",
            "Epoch: 70, Loss: 0.6979299783706665\n",
            "Epoch: 80, Loss: 0.6969885230064392\n",
            "Epoch: 90, Loss: 0.696089506149292\n",
            "Epoch: 100, Loss: 0.6952308416366577\n",
            "Estimated Parameters:\n",
            "linear.weight: 0.045152705162763596\n",
            "linear.bias: 0.05085201933979988\n",
            "\n",
            "Test Predictions:\n",
            "Input: 0.5205763578414917, Predicted Probability: 0.5185807943344116, Predicted Class: 1.0\n",
            "Input: -1.1443413496017456, Predicted Probability: 0.499795526266098, Predicted Class: 0.0\n",
            "Input: 0.801861047744751, Predicted Probability: 0.521750807762146, Predicted Class: 1.0\n",
            "Input: 0.04656729847192764, Predicted Probability: 0.5132355690002441, Predicted Class: 1.0\n",
            "Input: -0.18656976521015167, Predicted Probability: 0.5106053948402405, Predicted Class: 1.0\n",
            "Input: -0.10174587368965149, Predicted Probability: 0.5115624070167542, Predicted Class: 1.0\n",
            "Input: 0.8688861727714539, Predicted Probability: 0.5225059390068054, Predicted Class: 1.0\n",
            "Input: 0.7504116296768188, Predicted Probability: 0.5211711525917053, Predicted Class: 1.0\n",
            "Input: 0.5294653177261353, Predicted Probability: 0.518680989742279, Predicted Class: 1.0\n",
            "Input: 0.13770121335983276, Predicted Probability: 0.5142635107040405, Predicted Class: 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "C3zS1PQP_js4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MLE : the mother of all loss functions\n",
        "\n",
        "  * Maximum Likelihood Estimation (MLE): \\\\\n",
        "  MLE is a statistical principle used to estimate the parameters of a model by maximizing the likelihood of observing the given data. In the context of neural networks, MLE aims to find the parameter values that maximize the probability of observing the target outputs for the given inputs.\n",
        "  \n",
        "  * Cross-Entropy Loss (CE): \\\\\n",
        "  Cross-Entropy loss is a commonly used loss function in classification problems, where the target outputs are discrete and represented as one-hot encoded vectors. CE measures the dissimilarity between the predicted class probabilities and the true class probabilities.\n",
        "  \n",
        "  Mathematically, for a single training example with input $x$ and target output $y$, the CE loss is given by:\n",
        "  \n",
        "   $$CE = -\\sum y_i \\log(p_i)$$\n",
        "  \n",
        "  Here, $y_i$ is the true class label (1 for the correct class, 0 for other classes), $p_i$ is the predicted probability for class $i$, and the sum is taken over all classes. The goal is to minimize this loss, which is equivalent to maximizing the log-likelihood of the observed data.\n",
        "  \n",
        "* Mean Squared Error (MSE) Loss: \\\\\n",
        "  MSE loss is commonly used in regression problems, where the target outputs are continuous values. It measures the average squared difference between the predicted outputs and the true outputs.\n",
        "  \n",
        "  Mathematically, for a single training example with input $x$ and target output $y$, the MSE loss is given by:\n",
        "  \n",
        "$$MSE = \\frac{1}{n} \\sum (y_i - \\hat{y}_i)^2\\$$\n",
        "  \n",
        "  Here, $y_i$ is the true output, $\\hat{y}_i$ is the predicted output, and the sum is taken over all output dimensions. The goal is to minimize this loss, which is also equivalent to maximizing the log-likelihood of the observed data assuming a Gaussian distribution.\n",
        "  \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "VRQ3xsJ3D4Sf"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ioeZ3Jr4ERAe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Relation between MLE, Cross-Entropy, and Regression Loss\n",
        "\n",
        "In the context of classification problems, Cross-Entropy loss is often used as the loss function. By minimizing the Cross-Entropy loss, we are effectively maximizing the likelihood of the observed data, which aligns with the principles of Maximum Likelihood Estimation (MLE).\n",
        "\n",
        "Similarly, in regression problems, minimizing the regression loss (e.g., MSE) is also aligned with the principles of MLE. Minimizing the MSE loss corresponds to maximizing the likelihood of the observed data assuming a Gaussian distribution.\n",
        "\n",
        "In both cases, MLE provides the theoretical foundation for estimating parameters in neural networks, while Cross-Entropy loss and Regression loss serve as practical implementations of MLE in specific problem domains.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "EfSR7aL4F8ye"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "# Derivation on the relationship between MLE, Cross-Entropy  and MSE \n",
        "## Cross-Entropy Loss (Classification)\n",
        "\n",
        " In classification problems, the target outputs are discrete and represented as one-hot encoded vectors. Let's consider a single training example with input x and target output y. The predicted class probabilities for this example are denoted as p_i, where i represents the class index.\n",
        "\n",
        " The likelihood function for this example can be written as:\n",
        " $L(y|x, p) = \\prod_{i} p_i^{y_i}$\n",
        " where y_i is the true class label (1 for the correct class, 0 for other classes) and p_i is the predicted probability for class i.\n",
        "\n",
        " Taking the logarithm of the likelihood function, we have:\n",
        " $\\log L(y|x, p) = \\sum_i y_i \\log p_i$\n",
        "\n",
        " Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. Therefore, the loss function used for classification problems is the Cross-Entropy loss, given by:\n",
        " $CE = -\\sum_i y_i \\log p_i$\n",
        "\n",
        "## Regression Loss (Mean Squared Error)\n",
        "\n",
        " In regression problems, the target outputs are continuous values. Let's consider a single training example with input x and target output y. The predicted output for this example is denoted as \\hat{y}.\n",
        "\n",
        " The likelihood function assuming a Gaussian distribution can be written as:\n",
        " $L(y|x, \\hat{y}) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}} \\exp\\left(-\\frac{(y - \\hat{y})^2}{2\\sigma^2}\\right)$\n",
        "\n",
        " Taking the logarithm of the likelihood function, we have:\n",
        " $\\log L(y|x, \\hat{y}) = -\\frac{(y - \\hat{y})^2}{2\\sigma^2} - \\frac{1}{2}\\log(2\\pi\\sigma^2)$\n",
        "\n",
        "Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. Therefore, the loss function used for regression problems is the Mean Squared Error (MSE), given by:\n",
        "$MSE = \\frac{1}{n} \\sum_i (y_i - \\hat{y}_i)^2$\n"
      ],
      "metadata": {
        "id": "ZPCiK8DFHn4V"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "3ssZHJf1F9Fd"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "5ot5-o05HhgO"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}