Market Regime Classififcation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7b45b578",
   "metadata": {},
   "source": [
    "# Market Regime Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52aae14c",
   "metadata": {},
   "source": [
    "Following the financial crisis, quantitative value strategies, exspcially in equities, have experienced a decade of underperformance, making it challenging for investors to maintain commitment to them. This notebook aims to identify corresponding regimes and implement tactical allocation changes by evaluating the performance of three well-known supervised methods for binary (Value or Momentum) regime classification. Due to the results of [Fernández-Delgado et al. (2014)](https://jmlr.org/papers/v15/delgado14a.html), we focus on the following methods:\n",
    "\n",
    "- Logistic Regression (base line model)\n",
    "- Random Forest\n",
    "- Support Vector Machine\n",
    "- Multi-layer Perceptron\n",
    "\n",
    "Even though these are not sequence models, time series problems can be reformulated by lagging the feature time series, extracting reasonable feature to capture corresponding aspects and forming (X, y) tuples with the target variable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53ae3392",
   "metadata": {},
   "source": [
    "### Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ec352d",
   "metadata": {},
   "source": [
    "These are the necessary libraries we rely on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fd252a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# General libraries for data manipulation and visualization\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from datetime import datetime\n",
    "import missingno as msno\n",
    "from arch.bootstrap import CircularBlockBootstrap, optimal_block_length\n",
    "from scipy.cluster import hierarchy\n",
    "import time\n",
    "\n",
    "# Sklearn libraries for model selection, metrics, and models\n",
    "from sklearn.model_selection import train_test_split, TimeSeriesSplit, RandomizedSearchCV\n",
    "from sklearn.metrics import (confusion_matrix, roc_curve, roc_auc_score, accuracy_score, precision_score,\n",
    "                             recall_score, f1_score, classification_report, log_loss, make_scorer)\n",
    "from sklearn.ensemble import RandomForestClassifier, BaggingClassifier\n",
    "from sklearn.linear_model import LogisticRegression, LinearRegression\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.tree import plot_tree\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.inspection import permutation_importance\n",
    "from sklearn.feature_selection import RFECV, SequentialFeatureSelector\n",
    "from sklearn.experimental import enable_halving_search_cv\n",
    "from sklearn.model_selection import HalvingRandomSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.base import clone\n",
    "\n",
    "# Skopt for hyperparameter tuning\n",
    "from skopt import BayesSearchCV\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "\n",
    "# SciPy libraries for statistical functions\n",
    "from scipy.stats import randint, uniform, reciprocal\n",
    "from scipy.stats import rv_continuous\n",
    "\n",
    "# Numpy random state\n",
    "from numpy.random import RandomState"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4ec87e5",
   "metadata": {},
   "source": [
    "## Data Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22847d87",
   "metadata": {},
   "source": [
    "The monthly return data used here are from the paper [How Do Factor Premia Vary Over Time? A Century of Evidence](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3400998) by Ilmanen et al. (2019). It explores the following four factor strategies across liquid asset classes:\n",
    "\n",
    "1. Value\n",
    "2. Momentum\n",
    "3. Carry\n",
    "4. Defensive\n",
    "\n",
    "We focus on **Value** and **Momentum**, aggregated across all liquid asset classes, for two reasons. First, they can be considered antagonists due to their most pronounced negative pairwise correlation, which results in the highest potential added value through timing them. Second, aggregation across asset classes helps lower the impact of asset-specific factors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fabe7bb",
   "metadata": {},
   "source": [
    "### Price based Features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5306458e",
   "metadata": {},
   "source": [
    "From the return data set, a few features can be engineered to capture several posibly informative aspects:\n",
    "\n",
    "- **Value & Momentum Returns**: They caputure the momentum of these factors, embodying the persistence of absolute returns.\n",
    "\n",
    "- **Value-Momentum Spread**: Designed to exploit potential mean-reversion tendencies, this feature captures divergences in returns that may trigger counter-movements, such as those driven by portfolio rebalancing activities.\n",
    "\n",
    "- **Value & Momentum 12M returns**: These annual cumulative returns provide a window into traders' portfolios over longer periods of time. They also capture trend effects over this time horizon.\n",
    "\n",
    "- **Value-Momentum 12M Spread**: This feature serves as another measure of mean reversion and extends the horizon to 12 months, allowing for the identification of longer-term cyclical behaviour.\n",
    "\n",
    "- **Value & Momentum 36M Returns**: Cumulative returns over this three-year period are an important criterion for the evaluation of investments of institutional investors. This feature caputres that aspect.\n",
    "\n",
    "- **Value-Momentum 36M Spread**: This feature extends the mean reversion capture window to 36 months, providing a longer-term perspective on the cyclicality of these strategies.\n",
    "\n",
    "- **Rolling Value & Momentum Means**: These rolling averages over the past year provide insight into current trends and shifts in expected returns, helping to identify potential strategic pivots.\n",
    "\n",
    "- **Rolling Mean Difference**: The difference between the rolling means of Value & Momentum provides a comparative perspective on the current expectations of each strategy.\n",
    "\n",
    "- **Value & Momentum Volatilities**: The annualized standard deviations of returns captures the inherent uncertainty associated with each strategy.\n",
    "\n",
    "- **Volatility Difference**: The difference in volatility between Value & Momentum captures the comparative uncertainty associated with each strategy.\n",
    "\n",
    "- **Value & Momentum Skewness**: These 12-month rolling statistics of return asymmetry captures asymmetries in risk and return profiles.\n",
    "\n",
    "- **Skewness Difference**: The difference between the skewness of Value & Momentum returns indicates potential profit opportunities associated with the asymmetry of each strategy's return distribution.\n",
    "\n",
    "- **Value & Momentum Excess Kurtosis**: Measures tail risk, highlighting the potential for extreme outcomes and capturing the degree of outlier risk in each strategy.\n",
    "\n",
    "- **Kurtosis Difference**: This measure of the divergence between the tail risks of value and momentum strategies helps to identify which may be more prone to extreme returns.\n",
    "\n",
    "- **Correlation**: This feature captures the interdependence between value and momentum trading returns, helping to capture potential portfolio overlap and diversification implications for investors.\n",
    "\n",
    "- **Value & Momentum Drawdowns**: These measures encapsulate the most significant losses investors would have suffered prior to a rebound, providing a tangible picture of the losses investors suffer.\n",
    "\n",
    "- **Drawdown Difference**: This feature compares the drawdowns of value and momentum strategies to capture which strategy may be causing investors more pain at this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0eba770",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load the dataset from a csv file, specify the separator as \";\"\n",
    "data = pd.read_csv('return based data.csv', sep = \";\")\n",
    "\n",
    "# Convert the 'Date' column to datetime format (day first)\n",
    "data['Date'] = pd.to_datetime(data['Date'], dayfirst=True)\n",
    "\n",
    "# Set 'Date' as the index of the DataFrame\n",
    "data.set_index('Date', inplace=True)\n",
    "\n",
    "# Remove the last two columns of the DataFrame\n",
    "data = data.iloc[:, :-2]\n",
    "\n",
    "# Remove the last row of the DataFrame\n",
    "data = data.iloc[:-1]\n",
    "\n",
    "# Convert the 'Regime' column to integer type\n",
    "data['Regime'] = data['Regime'].astype(int)\n",
    "\n",
    "# Display the DataFrame\n",
    "display(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79eb36c",
   "metadata": {},
   "source": [
    "First, we visualize the cumulative performance of both strategies to get a first impression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b78c06dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot((1 + data['Value Returns']).cumprod(), label='Value', color = 'red')\n",
    "plt.plot((1 + data['Momentum Returns']).cumprod(), label='Momentum', color = 'black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "# Determine the color for the background\n",
    "background_color = ['lightgrey' if value > momentum else 'lightcoral' for value, momentum in zip(value_returns, momentum_returns)]\n",
    "\n",
    "# Plot the data with colored background\n",
    "plt.plot(value_returns, label='Value', color='red')\n",
    "plt.plot(momentum_returns, label='Momentum', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Return\")\n",
    "plt.yscale('log')\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Value vs. Momentum\")\n",
    "\n",
    "# Set the background color\n",
    "plt.gca().set_facecolor(background_color)\n",
    "\n",
    "# Add legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef64d2e",
   "metadata": {},
   "source": [
    "As an example for a feature variable, the 3-year cumulative return difference is plotted here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5a1929b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(data[\"Value-Momentum 36M Spread\"], color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"return spread\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Value-Momentum 36M Spread\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c27d049",
   "metadata": {},
   "source": [
    "### Economic Features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a145d757",
   "metadata": {},
   "source": [
    "As a next step, we supplement the most important non-price based financial macroecomoic features, such as the **[Federal Funds Effective Rate](https://fred.stlouisfed.org/series/DFF)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "897b7fd0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Read the 'Federal Funds Effective Rate.csv' file using pandas read_csv function\n",
    "DFF = pd.read_csv('Federal Funds Effective Rate.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime format using the pandas to_datetime function\n",
    "DFF['DATE'] = pd.to_datetime(DFF['DATE'])\n",
    "\n",
    "# Set the 'DATE' column as the index of the DataFrame using the set_index function\n",
    "DFF = DFF.set_index('DATE')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "DFF.index.name = 'Date'\n",
    "\n",
    "# Rename the 'DFF' column to 'Effective Federal Funds Rate' using the rename function\n",
    "DFF = DFF.rename(columns={'DFF': 'Effective Federal Funds Rate'})\n",
    "\n",
    "# Merge the 'data' DataFrame with the 'DFF' DataFrame using the merge function. \n",
    "# The merge is performed on the indices of the two DataFrames (left_index=True and right_index=True)\n",
    "# and any rows that do not have a match in the other DataFrame are kept (how='outer')\n",
    "combined_df = data.merge(DFF, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Normalize the index to remove any time component from the datetime objects\n",
    "combined_df.index = pd.to_datetime(combined_df.index).normalize()\n",
    "\n",
    "# Display the combined DataFrame using the display function\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fb14366",
   "metadata": {},
   "source": [
    "As a next step, we add the **[Federal Reserve Assets](https://fred.stlouisfed.org/series/WALCL)** to our set of features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8d2ac2b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Read the 'Federal Reserve Assets.csv' file into a DataFrame\n",
    "WALCL = pd.read_csv('Federal Reserve Assets.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime format\n",
    "WALCL['DATE'] = pd.to_datetime(WALCL['DATE'])  \n",
    "\n",
    "# Set the 'DATE' column as the index of the DataFrame\n",
    "WALCL = WALCL.set_index('DATE')  \n",
    "\n",
    "# Rename the index to 'Date'\n",
    "WALCL.index.name = 'Date'  \n",
    "\n",
    "# Rename the 'WALCL' column to 'Federal Reserve Assets'\n",
    "WALCL = WALCL.rename(columns={'WALCL': 'Federal Reserve Assets'})\n",
    "\n",
    "# Merge the current DataFrame with the 'WALCL' DataFrame. The merge is performed on the indices\n",
    "# of the two DataFrames (left_index=True and right_index=True), and rows that do not have a match \n",
    "# in the other DataFrame are kept (how='outer')\n",
    "combined_df = combined_df.merge(WALCL, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the combined DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1bd769b",
   "metadata": {},
   "source": [
    "As a next step, we add the **[Mortage Rates](https://fred.stlouisfed.org/graph/?g=QGRW)** to our set of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9eec65ed",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load the CSV file 'Mortage Rates.csv' into a pandas DataFrame\n",
    "MR = pd.read_csv('Mortage Rates.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime format\n",
    "MR['DATE'] = pd.to_datetime(MR['DATE'])\n",
    "\n",
    "# Set 'DATE' column as the index of the DataFrame\n",
    "MR = MR.set_index('DATE')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "MR.index.name = 'Date'\n",
    "\n",
    "# Rename the column 'MORTGAGE30US' to '30Y Mortage Rates'\n",
    "MR = MR.rename(columns={'MORTGAGE30US': '30Y Mortage Rates'})\n",
    "\n",
    "# Rename the column 'MORTGAGE15US' to '15Y Mortage Rates'\n",
    "MR = MR.rename(columns={'MORTGAGE15US': '15Y Mortage Rates'})\n",
    "\n",
    "# Rename the column 'MORTGAGE5US' to '5/1Y Mortage Rates'\n",
    "MR = MR.rename(columns={'MORTGAGE5US': '5/1Y Mortage Rates'})\n",
    "\n",
    "# Merge 'combined_df' DataFrame with 'MR' DataFrame using their indices\n",
    "# Rows with indices that are not common in both DataFrames are filled with NaNs ('outer' join)\n",
    "combined_df = combined_df.merge(MR, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a858fe47",
   "metadata": {},
   "source": [
    "As a next step, we add the **[Treasury Yields](https://fred.stlouisfed.org/graph/?g=I2cQ)** to our set of features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07f8b1bb",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load the CSV file 'Treasury Yields.csv' into a pandas DataFrame\n",
    "TY = pd.read_csv('Treasury Yields.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime format\n",
    "TY['DATE'] = pd.to_datetime(TY['DATE'])\n",
    "\n",
    "# Set 'DATE' column as the index of the DataFrame\n",
    "TY = TY.set_index('DATE')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "TY.index.name = 'Date'\n",
    "\n",
    "# Rename the column 'DGS3MO' to '3M Treasury Yields'\n",
    "TY = TY.rename(columns={'DGS3MO': '3M Treasury Yields'})\n",
    "\n",
    "# Rename the column 'DGS2' to '2Y Treasury Yields'\n",
    "TY = TY.rename(columns={'DGS2': '2Y Treasury Yields'})\n",
    "\n",
    "# Rename the column 'DGS5' to '5Y Treasury Yields'\n",
    "TY = TY.rename(columns={'DGS5': '5Y Treasury Yields'})\n",
    "\n",
    "# Rename the column 'DGS10' to '10Y Treasury Yields'\n",
    "TY = TY.rename(columns={'DGS10': '10Y Treasury Yields'})\n",
    "\n",
    "# Merge 'combined_df' DataFrame with 'TY' DataFrame using their indices\n",
    "# Rows with indices that are not common in both DataFrames are filled with NaNs ('outer' join)\n",
    "combined_df = combined_df.merge(TY, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d272e5",
   "metadata": {},
   "source": [
    "As a next step, we add the **[Yield Spreads](https://fred.stlouisfed.org/graph/?g=I2c3)** to our set of features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b11f83ea",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load the 'Yield Spreads.csv' file into a DataFrame\n",
    "YS = pd.read_csv('Yield Spreads.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime\n",
    "YS['DATE'] = pd.to_datetime(YS['DATE'])\n",
    "\n",
    "# Set the 'DATE' column as the index of the DataFrame\n",
    "YS = YS.set_index('DATE')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "YS.index.name = 'Date'\n",
    "\n",
    "# Rename the 'T10Y2Y' column to '10-2Y Yield Spreads'\n",
    "YS = YS.rename(columns={'T10Y2Y': '10-2Y Yield Spreads'})\n",
    "\n",
    "# Rename the 'T10Y3M' column to '10-3M Yield Spreads'\n",
    "YS = YS.rename(columns={'T10Y3M': '10-3M Yield Spreads'})\n",
    "\n",
    "# Merge the 'YS' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(YS, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c7b4692",
   "metadata": {},
   "source": [
    "As a next step, we add the **[Financial Stress](https://fred.stlouisfed.org/graph/?g=12QTs)** indicators to our set of features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c648f2cd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load the 'Financial Stress.csv' file into a DataFrame\n",
    "FS = pd.read_csv('Financial Stress.csv', sep = \",\")\n",
    "\n",
    "# Convert the 'DATE' column to datetime format\n",
    "FS['DATE'] = pd.to_datetime(FS['DATE'])\n",
    "\n",
    "# Set the 'DATE' column as the index of the DataFrame\n",
    "FS = FS.set_index('DATE')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "FS.index.name = 'Date'\n",
    "\n",
    "# Rename the 'NFCI' column to 'Chicago Fed National Financial Stress'\n",
    "FS = FS.rename(columns={'NFCI': 'Chicago Fed National Financial Stress'})\n",
    "\n",
    "# Rename the 'STLFSI4' column to 'St. Louis Fed Financial Stress'\n",
    "FS = FS.rename(columns={'STLFSI4': 'St. Louis Fed Financial Stress'})\n",
    "\n",
    "# Merge the 'FS' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(FS, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8667cc4",
   "metadata": {},
   "source": [
    "Now, we add the **[Sentiment Index](http://people.stern.nyu.edu/jwurgler/)** from Baker and Wurgler (2006)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a07e5054",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Define a function to parse dates in the format \"%Y%m\"\n",
    "def date_parser(date):\n",
    "    return datetime.strptime(date, \"%Y%m\")\n",
    "\n",
    "# Load the 'Sentiment.csv' file into a DataFrame, parsing 'yearmo' as dates using the defined date_parser function\n",
    "SENT = pd.read_csv(\"Sentiment.csv\", sep=';', parse_dates=['yearmo'], date_parser=date_parser)\n",
    "\n",
    "# Convert the 'yearmo' column to datetime format\n",
    "SENT['yearmo'] = pd.to_datetime(SENT['yearmo'])\n",
    "\n",
    "# Set the 'yearmo' column as the index of the DataFrame\n",
    "SENT = SENT.set_index('yearmo')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "SENT.index.name = 'Date'\n",
    "\n",
    "# Rename the 'SENT' column to 'Sentiment'\n",
    "SENT = SENT.rename(columns={'SENT': 'Sentiment'})\n",
    "\n",
    "# Select only the first column of the DataFrame (i.e., 'Sentiment')\n",
    "SENT = SENT.iloc[:, 0:1]\n",
    "\n",
    "# Merge the 'SENT' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(SENT, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92b5701d",
   "metadata": {},
   "source": [
    "Now, we add the **[Total return cyclically adjusted price-to-earnings ratio](http://www.econ.yale.edu/~shiller/data.htm)** to out features, which I call **CAPE**.\n",
    "\n",
    "It is actually a modified version of the original CAPE that takes into account changes in payout policy, shifting from dividends to buybacks, which has taken place in recent years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5de5fef8",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Define a function to parse dates in two different formats: \"%Y.%m\" and \"%Y.%-m\"\n",
    "def date_parser(date):\n",
    "    try:\n",
    "        return datetime.strptime(date, \"%Y.%m\")  # Try to parse the date with the first format\n",
    "    except ValueError:  # If the first format fails, try the second format\n",
    "        return datetime.strptime(date, \"%Y.%-m\")\n",
    "\n",
    "# Load the 'CAPE.csv' file into a DataFrame, parsing 'Date' as dates using the defined date_parser function\n",
    "CAPE = pd.read_csv(\"CAPE.csv\", parse_dates=['Date'], date_parser=date_parser, sep=';')\n",
    "\n",
    "# Convert the 'Date' column to datetime format\n",
    "CAPE['Date'] = pd.to_datetime(CAPE['Date'])\n",
    "\n",
    "# Set the 'Date' column as the index of the DataFrame\n",
    "CAPE = CAPE.set_index('Date')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "CAPE.index.name = 'Date'\n",
    "\n",
    "# Merge the 'CAPE' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(CAPE, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f112a748",
   "metadata": {},
   "source": [
    "Here we add the total returns of the US-American stock market, i.e. the **[Beta](http://www.econ.yale.edu/~shiller/data.htm)** Factor, as major risk factor to our set of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b11b37dd",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Define a function to parse dates in two different formats: \"%Y.%m\" and \"%Y.%-m\"\n",
    "def date_parser(date):\n",
    "    try:\n",
    "        return datetime.strptime(date, \"%Y.%m\")  # Try to parse the date with the first format\n",
    "    except ValueError:  # If the first format fails, try the second format\n",
    "        return datetime.strptime(date, \"%Y.%-m\")\n",
    "\n",
    "# Load the 'Beta.csv' file into a DataFrame, parsing 'Date' as dates using the defined date_parser function\n",
    "BETA = pd.read_csv(\"Beta.csv\", parse_dates=['Date'], date_parser=date_parser, sep=';')\n",
    "\n",
    "# Convert the 'Date' column to datetime format\n",
    "BETA['Date'] = pd.to_datetime(BETA['Date'])\n",
    "\n",
    "# Set the 'Date' column as the index of the DataFrame\n",
    "BETA = BETA.set_index('Date')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "BETA.index.name = 'Date'\n",
    "\n",
    "# Select only the second column of the DataFrame (i.e., 'Return')\n",
    "BETA = BETA.iloc[:, 1:2]\n",
    "\n",
    "# Rename the 'Return' column to 'Beta'\n",
    "BETA = BETA.rename(columns={'Return': 'Beta'})\n",
    "\n",
    "# Merge the 'BETA' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(BETA, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5282cb52",
   "metadata": {},
   "source": [
    "Here, we add the **[Geopolitical Risk Index](http://www.policyuncertainty.com/gpr.html)** developed by Dario Caldara and Matteo Iacoviello at the Federal Reserve Board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4869824",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Define a function to parse dates in the format \"%d.%m.%Y\"\n",
    "def date_parser(date):\n",
    "    return pd.to_datetime(date, format=\"%d.%m.%Y\")\n",
    "\n",
    "# Load the 'Geopolitical Risk.csv' file into a DataFrame, parsing 'month' as dates using the defined date_parser function\n",
    "GEO = pd.read_csv(\"Geopolitical Risk.csv\", sep=';', parse_dates=['month'], date_parser=date_parser)\n",
    "\n",
    "# Convert the 'month' column to datetime format\n",
    "GEO['month'] = pd.to_datetime(GEO['month'])\n",
    "\n",
    "# Set the 'month' column as the index of the DataFrame\n",
    "GEO = GEO.set_index('month')\n",
    "\n",
    "# Rename the index to 'Date'\n",
    "GEO.index.name = 'Date'\n",
    "\n",
    "# Select only the fourth column of the DataFrame (i.e., 'GPRH')\n",
    "GEO = GEO.iloc[:, 3:4]\n",
    "\n",
    "# Rename the 'GPRH' column to 'Geopolitical Risk'\n",
    "GEO = GEO.rename(columns={'GPRH': 'Geopolitical Risk'})\n",
    "\n",
    "# Merge the 'GEO' DataFrame into the 'combined_df' DataFrame, aligning on their indexes\n",
    "# The 'outer' method ensures that all data is retained, even if a match isn't found in the other DataFrame\n",
    "combined_df = combined_df.merge(GEO, left_index=True, right_index=True, how='outer')\n",
    "\n",
    "# Display the merged DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81630b8b",
   "metadata": {},
   "source": [
    "Because there are some dots in the data frame, we need to replace them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d79dec40",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Replace any instances of '.' in the 'combined_df' DataFrame with NaN (representing missing data)\n",
    "combined_df = combined_df.replace('.', np.nan)\n",
    "\n",
    "# Display the updated DataFrame\n",
    "display(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e8b9a2",
   "metadata": {},
   "source": [
    "Due to the varying frequency of all our features (some are available weekly, others monthly), we need to consider how to merge them and how to handle missing data values.\n",
    "\n",
    "First, it is obvious to perform a forward fill, since in our model the last available value of a feature is the one that is available at the current time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d7e822c",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Iterate over each column in the 'combined_df' DataFrame\n",
    "for col in combined_df.columns:\n",
    "    # Convert the column's data to numeric values, replacing any errors (like non-numeric strings) with NaN\n",
    "    combined_df[col] = pd.to_numeric(combined_df[col], errors='coerce')\n",
    "\n",
    "# Fill any NaN values in the DataFrame with the preceding (forward-fill 'ffill') value in each column\n",
    "filled_df = combined_df.fillna(method='ffill')\n",
    "\n",
    "# Display the updated DataFrame\n",
    "display(filled_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48051fc1",
   "metadata": {},
   "source": [
    "Some of the features go back much further than return data is available for both Value and Momentum. Therefore, we need to remove these periods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef8086a",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Set pandas option to display all columns of the DataFrame when it is displayed\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "# Convert the start and end date strings to datetime objects\n",
    "start_date = pd.to_datetime(\"1926-07-01\")\n",
    "end_date = pd.to_datetime(\"2023-02-28\")\n",
    "\n",
    "# Filter the 'filled_df' DataFrame to only include rows where the index (date) is between the start_date and end_date\n",
    "filtered_df = filled_df[(filled_df.index >= start_date) & (filled_df.index <= end_date)]\n",
    "\n",
    "# Display the filtered DataFrame\n",
    "display(filtered_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fde0da2c",
   "metadata": {},
   "source": [
    "Let us look at the **Beta** variable to see if everything looks reasonable so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57809cbb",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(filtered_df[\"Beta\"], color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"return\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Beta\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f66f66a",
   "metadata": {},
   "source": [
    "### Merging the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eab8d7c9",
   "metadata": {},
   "source": [
    "Oversampling is now present due to features that are available on a weekly basis.\n",
    "\n",
    "Let und remind ourselves of our goal: At the end of a month, we want to predict whether Value or Momentum will perform better in the upcoming month.\n",
    "\n",
    "Therefore, we will sample our data down to a monthly time scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c6cfd23",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Resample the 'filtered_df' DataFrame to a monthly frequency, using the last observation of each month\n",
    "df_monthly = filtered_df.resample('M').last()\n",
    "\n",
    "# Display the resampled DataFrame\n",
    "display(df_monthly)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f33d011e",
   "metadata": {},
   "source": [
    "To quickly check for resonability, let us plot our target variable, the Regime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a39ae59",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data with black color\n",
    "plt.plot(df_monthly.iloc[:, -18], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Regime\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12868553",
   "metadata": {},
   "source": [
    "In a last step, we need to shift the regime variable back in time by one month, since we face a predicition problem.\n",
    "However, we will use the current regime as feature to cover possible persistence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98bb0f2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a copy of the original DataFrame\n",
    "df_modified = df_monthly.copy()\n",
    "\n",
    "df_modified[\"Current Regime\"] = df_modified['Regime']\n",
    "\n",
    "# Shift the \"Regime\" column up by one row\n",
    "df_modified['Regime'] = df_modified['Regime'].shift(-1)\n",
    "\n",
    "# Remove the last row\n",
    "df_modified = df_modified.drop(df_modified.tail(1).index)\n",
    "\n",
    "# Display the updated DataFrame\n",
    "display(df_modified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15f2ab11",
   "metadata": {},
   "source": [
    "### Unbalanced Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "196d8680",
   "metadata": {},
   "source": [
    "Now we come to the most critical part of this project: Dealing with the fact that not all features go back equally far into the past, which is visualized by the following bar plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea2e8e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(df_modified, sort='descending', color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de12f28b",
   "metadata": {},
   "source": [
    "We obviously face a trade-off between more potentially informative features and more historical data, which also provides very valuable information.\n",
    "\n",
    "How do we solve this?\n",
    "\n",
    "In statistics, there are two basic objectives in modeling: structural analysis and forecasting. The former does not allow variables to be discarded simply because of statistical redundancy, which is why, for example, L1 regularization (lasso) is not useful when estimating regression models that are used for structural analysis. In the bias-variance dilemma, structural analysis is more concerned about bias, i.e., underfitting.\n",
    "\n",
    "Forecasting, on the other hand, has different priorities and is more concerned about variance, i.e., overfitting. Why a model makes good predictions and where it gets the relevant information from is of secondary importance. For this reason, L1 regularization, with its inherent feature selection that avoids overfitting, is very reasonable.\n",
    "\n",
    "What does that mean for our problem? If shorter variables are very similar to other longer variables, then we can safely discard the shorter one to get more historical data.\n",
    "\n",
    "The coefficient of dtermination is a simple way to assess statsitical similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55fb7f49",
   "metadata": {},
   "source": [
    "In the next steps, we will successively go through our data, starting with the shortest available time series, and see if we can find similarities to longer time series."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eaad194",
   "metadata": {},
   "source": [
    "#### 5/1Y Mortage Rates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "770cc52c",
   "metadata": {},
   "source": [
    "This is our most limiting, i.e. shortest time series, but economically/theoretically, the 5/1Y Mortage Rates are expected to carry much of the same information as the 15Y Mortage Rates and the 30Y Mortage Rates.\n",
    "\n",
    "Let us back this up with the data:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad3a6035",
   "metadata": {},
   "source": [
    "First we need to extract the shortest period, where all features are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "748b4d60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete = df_modified.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d59dbac",
   "metadata": {},
   "source": [
    "Let us take a visual look at the data, where the similarity to especially 15Y Mortage Rates ist particularly striking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62843e0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = ['30Y Mortage Rates', '15Y Mortage Rates', '5/1Y Mortage Rates']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete[columns_to_plot])\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('interest Rate')\n",
    "plt.title('Mortgage Rates')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5463956f",
   "metadata": {},
   "source": [
    "Now, let us compute the pairwise coefficients of determination to all other variable.\n",
    "\n",
    "We define a function for this, since we will need to do this multiple times in the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e96d3d65",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_r_squared(dataframe, column_name):\n",
    "    # Calculate the correlation matrix\n",
    "    corr_matrix = dataframe.corr()\n",
    "\n",
    "    # Select the correlations related to the specified column\n",
    "    correlations = corr_matrix[column_name]\n",
    "\n",
    "    # Compute R^2 from correlations\n",
    "    r_squared = correlations.apply(lambda x: x ** 2)\n",
    "\n",
    "    # Sort the R^2 values\n",
    "    sorted_r_squared = r_squared.sort_values(ascending=True)\n",
    "\n",
    "    # Plot the R^2 values\n",
    "    plt.figure(figsize=(11, 11), dpi = 100)\n",
    "    sorted_r_squared.drop(column_name).plot(kind='barh', color='black')  # Exclude the specified column as its R^2 with itself is 1\n",
    "    plt.xlabel('R^2')\n",
    "    plt.title(f'Pairwise R^2 with {column_name}')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c72b6b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify data\n",
    "df = df_complete\n",
    "column = '5/1Y Mortage Rates'\n",
    "\n",
    "# Plot\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9476a341",
   "metadata": {},
   "source": [
    "The R^2 to to 15Y Mortage Rates is indeed close to one, indicating that differences are mostly due to noise."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed86c693",
   "metadata": {},
   "source": [
    "To test, how likeli we can assume, that both features provide the same information, plus some noise, we can conduct a  bootstrapping-based hypothesis test:\n",
    "\n",
    "**Null Hypothesis (H0):** The two time series (\"5/1Y Mortgage Rates\" and \"15Y Mortgage Rates\") are perfectly dependent, with their difference only attributable to the observed noise structure (given by the differences between the series in the data).\n",
    "\n",
    "**Alternative Hypothesis (H1):** The two time series are not perfectly dependent, even after accounting for the noise structure. In other words: The observed MIC value is significantly lower than what we would expect under perfect dependency with the given noise structure\n",
    "\n",
    "We choose a significance level α of 5% because this usually balances the probabilities of a Type I and a Type II error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44db9b5c",
   "metadata": {},
   "source": [
    "The bootstrapping test defined below takes two time series, scales them uniformly to ensure scale & correlation direction invariance, extracts the difference time series, and then circularly bootstraps it 100,000 times.\n",
    "This ensures that any autodependencies are preserved in this time series context.\n",
    "The optimal block length is chosen according to [Patton et al. (2009)](https://www.tandfonline.com/doi/abs/10.1080/07474930802459016).\n",
    "\n",
    "Then, these difference time series are added to the first time series and the corresponding coefficient of determination is computed with the first time series ifself.\n",
    "This generates a null distribution under the assumption that the true coefficient of determination is 1 (H0) and the differences are just due to noise.\n",
    "\n",
    "By doing so, we can compute a p-value for H1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fe54626",
   "metadata": {},
   "outputs": [],
   "source": [
    "def standardize_series(s):\n",
    "    scaler = StandardScaler()\n",
    "    s_values_scaled = scaler.fit_transform(s.values.reshape(-1, 1))\n",
    "    return pd.Series(s_values_scaled.flatten(), index=s.index)\n",
    "\n",
    "def r_squared_bootstrap_test(ts1, ts2, n_permutations):\n",
    "    \n",
    "    # Standardize the time series\n",
    "    ts1 = standardize_series(ts1)\n",
    "    ts2 = standardize_series(ts2)\n",
    "    \n",
    "    if np.corrcoef(ts1, ts2)[0, 1] < 0:\n",
    "        ts2 = ts2 * -1\n",
    "\n",
    "    # Calculate observed R^2\n",
    "    observed_r_squared = np.square(np.corrcoef(ts1, ts2)[0, 1])\n",
    "    \n",
    "    # Calculate observed R^2\n",
    "    observed_r_squared = np.square(np.corrcoef(ts1, ts2)[0, 1])\n",
    "\n",
    "    # Calculate differences between ts2 and ts1\n",
    "    differences = ts2 - ts1\n",
    "    differences_values = differences.values\n",
    "\n",
    "    # Calculate optimal block length\n",
    "    opt_block_length_df = optimal_block_length(differences_values)\n",
    "    opt_block_length = int(opt_block_length_df['circular'])  # Use the optimal block length\n",
    "    \n",
    "    # Check if the optimal block length is zero, if so, set it to 1\n",
    "    if opt_block_length == 0:\n",
    "        opt_block_length = 1\n",
    "\n",
    "    # Initialize circular block bootstrap with optimal block length\n",
    "    bs = CircularBlockBootstrap(opt_block_length, differences_values)\n",
    "\n",
    "    # Compute R^2 for bootstrapped series and generate null distribution\n",
    "    null_r_squared = []\n",
    "    for _, bs_diffs in zip(range(n_permutations), bs.bootstrap(n_permutations)):\n",
    "        # Generate new ts2 by adding bootstrapped differences to ts1\n",
    "        # Ensure that ts2_new has the same length as ts1 by discarding extra values or padding with zeros\n",
    "        bs_diffs_trimmed = bs_diffs[0][0][:len(ts1)]\n",
    "        ts2_new = ts1 + pd.Series(bs_diffs_trimmed, index=ts1.index)\n",
    "        r_squared = np.square(np.corrcoef(ts1, ts2_new)[0, 1])\n",
    "        null_r_squared.append(r_squared)  # Store R^2 for each bootstrapped series\n",
    "\n",
    "    # Compute p-value: proportion of null R^2 values greater than or equal to observed R^2\n",
    "    p_value = np.mean(np.array(null_r_squared) <= observed_r_squared)\n",
    "\n",
    "    # Output\n",
    "    print(f\"p-value: {p_value*100:.2f}%\")  # Print the calculated p-value as a percentage with two decimal points\n",
    "    \n",
    "    # Set figure size and dpi\n",
    "    plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "    # Plot null distribution\n",
    "    plt.hist(null_r_squared, bins=100, color='black')  # Plot the histogram of null R^2 values\n",
    "    plt.axvline(observed_r_squared, color='red', linestyle='dashed', linewidth=2, label=f'Observed R^2: {observed_r_squared:.2f}')\n",
    "    plt.legend()\n",
    "    plt.xlabel('R^2')\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.title('Null distribution of R^2')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b54645c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete[\"5/1Y Mortage Rates\"]  # Time series 1\n",
    "ts2 = df_complete[\"15Y Mortage Rates\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9842d402",
   "metadata": {},
   "source": [
    "We can see, that - accounting for the diffrence structure of the two time series - we can't reject H0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11c37ec0",
   "metadata": {},
   "source": [
    "Therefore, we decide to drop 5/1Y Mortage Rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bb021fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the column from the DataFrame df_modified\n",
    "lean_sample_1 = df_modified.drop('5/1Y Mortage Rates', axis=1)\n",
    "\n",
    "# Display the DataFrame\n",
    "display(lean_sample_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52c46147",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_1, sort='descending', color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ae7cdaa",
   "metadata": {},
   "source": [
    "#### Federal Reserve Assets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6afa4633",
   "metadata": {},
   "source": [
    "The above steps are now repeated for the variable Federal Reserve Assets.\n",
    "\n",
    "It should be noted that this sequential approach suffers from the same problems as common greedy feature selection approaches such as stepwise backward feature selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4802bfd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_1 = lean_sample_1.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df1ea5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data with black color\n",
    "plt.plot(df_complete_1[\"Federal Reserve Assets\"], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"USD\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Federal Reserve Assets\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf397924",
   "metadata": {},
   "source": [
    "The absolute values of Federal Reserve Assets, representing the level of assets held by the central bank, have limited meaningfulness, since absolute values on their own lack context.\n",
    "\n",
    "By transforming the Federal Reserve Assets into percentage changes, we shift the focus to the relative change over time. This provides an economically more meaningful and insightful feature for our purposes, since large changes in Assets indicate relevant economic events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "961370f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the percentage changes of the \"Federal Reserve Assets\" variable\n",
    "percentage_changes = df_complete_1[\"Federal Reserve Assets\"].pct_change()\n",
    "\n",
    "# Create a new column for percentage changes in the DataFrame\n",
    "df_complete_1[\"Federal Reserve Assets Percentage Changes\"] = percentage_changes\n",
    "\n",
    "# Drop rows with missing values\n",
    "df_complete_1.dropna(inplace=True)\n",
    "\n",
    "# Drop the column 'Federal Reserve Assets'\n",
    "df_complete_1 = df_complete_1.drop('Federal Reserve Assets', axis=1)\n",
    "\n",
    "# Verify the updated DataFrame\n",
    "df_complete_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "891ccc55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data with black color\n",
    "plt.plot(df_complete_1[\"Federal Reserve Assets Percentage Changes\"], color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"change\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Federal Reserve Assets Percentage Changes\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea872179",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify data\n",
    "df = df_complete_1\n",
    "column = 'Federal Reserve Assets Percentage Changes'\n",
    "\n",
    "# Plot\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5384f8cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"Federal Reserve Assets Percentage Changes\", 'St. Louis Fed Financial Stress']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_1[columns_to_plot])\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('')\n",
    "plt.title('Financial Stress Inficators')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71cec80b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_1[\"Federal Reserve Assets Percentage Changes\"]  # Time series 1\n",
    "ts2 = df_complete_1[\"St. Louis Fed Financial Stress\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6555082",
   "metadata": {},
   "source": [
    "**Federal Reserve Assets Percentage Changes** is currently the shortest time series for which we cannot reject H0.\n",
    "\n",
    "This means that we would have to discard all the data going back to 1930. This would be absurd, but to compare how this trade-off between more data and more features behaves, we use this first short data set to train our models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "921cdfda",
   "metadata": {},
   "source": [
    "## First Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef57248",
   "metadata": {},
   "source": [
    "First, the dataset must be divided into a training dataset and a test dataset. The former is used to tune the hyperparameters, perform feature selection, and of course training while the latter is used to test the model.\n",
    "\n",
    "We choose the usual 70/30 split because the training process requires as much data as possible, but the test dataset should not be too short to obtain economically meaningful test results. It also ensures consistency with the longer data sets we will be working with later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9440fe5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the DataFrame by the time-related column\n",
    "df_sorted = df_complete_1.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_1 = df_sorted[:split_index]\n",
    "df_test_1 = df_sorted[split_index:]\n",
    "\n",
    "#Display\n",
    "df_train_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5680f3f3",
   "metadata": {},
   "source": [
    "### Feature Space"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27165db9",
   "metadata": {},
   "source": [
    "The following diagonalized R^2 matrix shows that many features are very similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b63f1f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_1.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f6feea2",
   "metadata": {},
   "source": [
    "Accordingly, the feature space is too high dimensional and we can reduce it while rotating it so that the resulting new features are orthogonal to each other.\n",
    "\n",
    "For example, when fitting trees, rotating the feature space in a direction that aligns with the axes reduces the number of levels needed by the tree. This speeds up computations and makes overfitting less likely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ec8dddc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96dc4296",
   "metadata": {},
   "source": [
    "We see that we can reduce the feature space from 42 to 20, while preserving basically all the information in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ad48f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4e3bbd",
   "metadata": {},
   "source": [
    "Let us drop them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f62bf89e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a85202",
   "metadata": {},
   "source": [
    "The big disadvantage of this transformation is that interpretability is lost. But as mentioned above, our goal is not structural analysis but prediction, so statistical arguments have to take precedence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87a577fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fd3e4e4",
   "metadata": {},
   "source": [
    "By definition, the variance of each principal component decreases progressively. However, since many models are sensitive to different scaling, we unify them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e7f7856",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca386cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf178f04",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb17dfe",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f4912ed",
   "metadata": {},
   "source": [
    "Logistic regression is one of the most fundamental models, especially for binary classification problems, such as the one in this case. The basic principle is to use a sigmoid function to model the probability of occurrence of a particular class y, given X. The sigmoid function takes real numbers as input and outputs values in the range [0, 1], allowing the output to be interpreted as a probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ce73a23",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e942ed4",
   "metadata": {},
   "source": [
    "This model has some hyperparameters that need to be specified or tuned in an informed way:\n",
    "\n",
    "- \"**penalty**\" defines the type of regularization (L1, L2, elastic net, or none). We set it to elasticnet, since it combines the advantages of both.\n",
    "- \"**dual**\" defines the formulation of the problem (primal or dual). We leave it as False, since dual formulation is only implemented for l2 penalty, which we don't use.\n",
    "- \"**tol**\" is the tolerance for the stopping criterion, which we leave at 1e-4. \n",
    "- \"**C**\" is the inverse regularization parameter. This is one of the most important hyperparameters and needs to be tuned.\n",
    "- \"**Fit_intercept**\" decides whether to include an intercept in the model. We leave it as True.\n",
    "- \"**Intercept_scaling**\" is an artificial parameter to scale the intercept. We leave it at 1, since the classes are fairly balanced.\n",
    "- \"**class_weight**\" is the weight of the classes. We set it to balanced, as this helps to avoid possible misclassification of minority classes.\n",
    "- \"**Solver**\" is the optimization algorithm. We set it to saga, since this is the only one that can handle elastic net based objective functions.\n",
    "- \"**max_iter**\" is the maximum number of iterations for the solver. We set it to 1000 to ensure convergence.\n",
    "- \"**multi_class**\" is the strategy for multiple classes. We leave it at auto.\n",
    "- \"**verbose**\" controls the level of detail in the output. We leave it at False as it can mess the process up.\n",
    "- \"**warm_start**\" decides whether to reuse the previous solution for the next fit. We leave at False.\n",
    "- \"**n_jobs**\" determines the number of CPUs used for training. We set it to -1 so that all are used to speed up computations.\n",
    "- \"**l1_ratio**\" is the ratio of the L1 regularization to the elastic net penalty. This hyperparameter must also be tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82ad066c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt.space import Real\n",
    "\n",
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d948a3d",
   "metadata": {},
   "source": [
    "To tune these hyperparameters, we use nested time series cross-validation and Bayesian optimization.\n",
    "This is useful for several reasons:\n",
    "\n",
    "- First, nested cross-validation combines both feature selection and hyperparameter tuning, which is important because they are interdependent. Furthermore, feature selection must also be cross-validated, otherwise we will overfit to the chosen predictors.\n",
    "- Second, time series cross-validation avoids information leakage from the future because it resambles live prediction situations.\n",
    "- Third, Bayesian optimization searches the hyperparameter space in an informed way compared to exhaustive grid search or randomized search. As a result, it speeds up computation and increases the likelihood of finding a globally optimal solution.\n",
    "\n",
    "For feature selection, we use Recursive Feature Elimination, which is one of the multivariate wrapper methods.\n",
    "Wrapper methods refer to a family of supervised feature selection methods that use a specific model to evaluate different subsets of features to finally select the best one.\n",
    "A major advantage of wrapper methods is the fact that they tend to provide the best performing feature set for the particular type of model chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e98ffa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the time series cross-validator for the external and internal loops\n",
    "tscv_outer = TimeSeriesSplit(n_splits=10)\n",
    "tscv_inner = TimeSeriesSplit(n_splits=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f82a28",
   "metadata": {},
   "source": [
    "But how are the subsets of hyperparameters and features evaluated in the validation fold?\n",
    "\n",
    "For classification problems, accuracy is most often taken, but in finance this is not a particularly good choice. The reason is that accuracy scores a classifier in terms of its proportion of correct predictions. This has the disadvantage that it does not take into account class probabilities, which is a problem since we want to use them as portfolio weights in our tactical asset allocation model.\n",
    "\n",
    "A good alternative is log-loss (also known as cross-entropy loss). It evaluates a classifier in terms of the average log-likelihood of the true labels. This intuitively makes the most sense, since it matters how large our bets are when we are wrong or right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4583fcc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scoring metric\n",
    "scorer = make_scorer(log_loss, greater_is_better=False, needs_proba=True, labels=[0.0, 1.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92626de9",
   "metadata": {},
   "source": [
    "Now, we can tune the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d00dc9d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_1['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b06f2eb",
   "metadata": {},
   "source": [
    "(Your results may vary due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94299d68",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93eaacb",
   "metadata": {},
   "source": [
    "The model uses only L1 regularization (due to l1_ratio = 1), but the regularization strength is relatively low (C is close to 2).\n",
    "In addition, it has discarded all but one feature, which is in line with 100% L1 regularization.\n",
    "\n",
    "Overall, it is clear that the model has been greatly simplified to avoid overfitting. This is to be expected, as the signal/noise ratio in finance is usually very low, due to the fact that strong relationships are arbitraged away."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec40e2ec",
   "metadata": {},
   "source": [
    "Now, we can specify its hyperparamaters and and fit on the whole training data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f4cdf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd30e4d1",
   "metadata": {},
   "source": [
    "Let us look at the feature coefficients and the intercept that have been estimated by the fitting process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0299db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b72f5f15",
   "metadata": {},
   "source": [
    "Interestingly, the coefficient is quite low and the intercept is slightly positive, indicating a slight bias toward Momentum."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d24443",
   "metadata": {},
   "source": [
    "Let us look at how both regimes are distributed in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7459fdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_1['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b17a14a",
   "metadata": {},
   "source": [
    "Now we can use the model to predict the regimes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "915b0448",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dcceedb",
   "metadata": {},
   "source": [
    "Next, we visualize the predicted probabilities that we use as weights for our tactical asset allocation implications.\n",
    "The probabilities are always close to 50%, indicating that the signal picked up by the model was quite low."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44a826cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1018c89",
   "metadata": {},
   "source": [
    "For the sake of completeness, the confusion matrix, accuracy and low logg are shown below.\n",
    "\n",
    "The values are actually pretty bad, wore then chance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3deca157",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e720ab68",
   "metadata": {},
   "source": [
    "Now we implement the tactical allocation model based on the predicted probabilities.\n",
    "\n",
    "This means that each month we invest the probability of Class 1 in Momentum and the rest in Value, assuming monthly rebalancing with no transaction costs.\n",
    "\n",
    "As expected, there is little difference in performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed232b34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_1.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8245c35a",
   "metadata": {},
   "source": [
    "The following somewhat sparse histogram confirms what we see in the cumulative return plot and the corresponding Sharpe ratios printed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96a7309c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41dce2e8",
   "metadata": {},
   "source": [
    "It is not necessary at this point, but later we need to formally test whether the higher Sharpe Ratio of our model is statistically significantly different from that of the simple 50/50 Benchmark. Again, this can be done with a right-sided bootstrapping test with the following pair of hypotheses:\n",
    "\n",
    "**Null Hypothesis (H0)**: The model has a Sharpe Ratio equal to or lower than the benchmark.\n",
    "\n",
    "**Alternative Hypothesis (H1)**: The model has a higher Sharpe Ratio than the benchmark."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd4f1182",
   "metadata": {},
   "source": [
    "In this test, the data points are drawn from the return sample of the benchmark to create a new sample. The Sharpe ratio of this new sample is then computed and stored.\n",
    "This process is repeated a total of 100,000 times to obtain a null distribution of the benchmark's Sharpe ratios, which can then be used to perform a right-sided hypothesis test comparing the model's Sharpe ratio to this null distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99c6cf48",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sharpe_ratio_bootstrap_test(sample1, sample2, n_permutations=1000000):\n",
    "    \n",
    "    # Bootstrap sampling from sample1 and compute Sharpe ratios\n",
    "    sharpe_ratios = []\n",
    "    n = len(sample1)\n",
    "    for _ in range(n_permutations):\n",
    "        bootstrap_sample = np.random.choice(sample1, size=n, replace=True)\n",
    "        mean_return = np.mean(bootstrap_sample)\n",
    "        std_return = np.std(bootstrap_sample)\n",
    "        sharpe_ratio = mean_return / std_return\n",
    "        sharpe_ratios.append(sharpe_ratio)\n",
    "\n",
    "    # Compute the observed Sharpe ratio for sample2\n",
    "    observed_mean_return = np.mean(sample2)\n",
    "    observed_std_return = np.std(sample2)\n",
    "    observed_sharpe_ratio = observed_mean_return / observed_std_return\n",
    "\n",
    "    # Calculate p-value: proportion of null Sharpe ratios more extreme than observed Sharpe ratio\n",
    "    p_value = (np.abs(sharpe_ratios) >= np.abs(observed_sharpe_ratio)).mean()\n",
    "    \n",
    "    # Output\n",
    "    print(f\"p-value: {p_value*100:.2f}%\")  # Print the calculated p-value as a percentage with two decimal points\n",
    "\n",
    "    # Plot the null distribution of Sharpe ratios\n",
    "    plt.figure(figsize=(11, 6), dpi=100)\n",
    "    plt.hist(sharpe_ratios, bins=100, color='black')\n",
    "    plt.axvline(observed_sharpe_ratio, color='red', linestyle='dashed', linewidth=2,\n",
    "                label=f'Model Sharpe Ratio: {observed_sharpe_ratio:.2f}')\n",
    "    plt.legend()\n",
    "    plt.xlabel('Sharpe Ratio')\n",
    "    plt.ylabel('frequency')\n",
    "    plt.title('Null Distribution of Benchmark Sharpe Ratios')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48e1c6e3",
   "metadata": {},
   "source": [
    "To demonstrate the test, we can run it for the logistic regression model.\n",
    "\n",
    "We see that we can be very confident that the small difference in the two Sharpe ratios is due to chance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6522a678",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cbcd09b",
   "metadata": {},
   "source": [
    "#### Support Vector Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ee2c4af",
   "metadata": {},
   "source": [
    "Support Vector Machines (SVMs) are a family of supervised learning algorithms used for classification and regression problems. In the context of binary classification, SVMs attempt to find an optimal hyperplane that separates the two classes.\n",
    "Specifically, we use the **C-Support Vector Classification (C-SVC)** variant because it is particularly good at avoiding overfitting, similar to the hyperparameter 'C' in logistic regression.\n",
    "Technically, it controls the trade-off between achieving the largest possible margin and minimizing classification errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2abd36a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64cf3349",
   "metadata": {},
   "source": [
    "This model has some hyperparameters that need to be specified or tuned in an informed way:\n",
    "\n",
    "- \"**C**\" is the regularization parameter. We need to tune this.\n",
    "- \"**Kernel**\" specifies the type of kernel used (linear, poly, rbf, sigmoid) for the so-called \"kernel trick\". We need to tune this as well\n",
    "- **degree**\" is the degree of the polynomial kernel ('poly'). We need to tune this too, if poly is chosen as kernel. \n",
    "- \"**Gamma**\" is the kernel coefficient for \"rbf\", \"poly\" and \"sigmoid\". This must also be adjusted if these kernels are selected.\n",
    "- \"**coef0**\" is the independent term in the kernel functions of 'poly' and 'sigmoid'. This should also be tuned.\n",
    "- \"**shrinking**\" determines whether heuristic shrinking is used or not. We leave it at True as it can speed up the computation significantly.\n",
    "- \"**Probability**\" controls whether probabilities are calculated. Since we need them for our scoring function and portfolio weights, we set it to True.\n",
    "- \"**tol**\" is the tolerance for the stop criterion. We leave it at 1e-3.\n",
    "- \"**cache_size**\" is the size of the kernel cache (in MB). We set it significantly higher than the default to speed up computations.\n",
    "- \"**class_weight**\" is the weight of classes. We set it to 'default' in case the model has unbalanced class distributions.\n",
    "- \"**verbose**\" controls the level of detail in the output. We leave it at False.\n",
    "- \"**max_iter**\" is the maximum number of iterations. We leave it at -1 to ensure convergence.\n",
    "- \"**decision_function_shape**\" specifies the shape of the decision function ('ovr' or 'ovo'). Since we are dealing with a binary classification problem, the specification does not matter, but we set it to 'ovo'.\n",
    "- **break_ties**\" decides whether to break ties when there are multiple classes. We leave it at False since it doesn't really matter for a binary classifcation problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1ffc7b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30a8bc2",
   "metadata": {},
   "source": [
    "As with logistic regression, we use nested time series cross-validation with log loss as the scoring metric.\n",
    "\n",
    "However, Recursive Feature Elimination (RFE) does not work with SVCs because it uses the kernel trick, which prevents the model from providing feature importance. For this reason, we use Sequential Backward Feature Selection, which is similar to RFE but model agnostic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0945019",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_1['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f7fa3e",
   "metadata": {},
   "source": [
    "(Your results may vary due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e00865f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d31b0c4d",
   "metadata": {},
   "source": [
    "Let us talk about the two most important hyperparameters:\n",
    "\n",
    "A C of 100 is quite large, meaning that the model has very little regularization.\n",
    "A 'sigmoid' is typically used in neural networks, and when used as a kernel, the SVM behaves like a two-layer perceptron.\n",
    "Therefore, the combination of a high C and a sigmoid kernel allows for a high complexity model that can fit the data very flexibly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64ea52bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c987108",
   "metadata": {},
   "source": [
    "Let us look at the predicted regimes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ae10ac4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af047cd",
   "metadata": {},
   "source": [
    "It may come as a surprise that Class 1, i.e. Momentum, is consistently given a higher probability below, although Class 0, i.e.Value, is also quite often predicted above.\n",
    "What is the reason for this?\n",
    "\n",
    "The SVC model in scikit-learn predicts class probabilities using Platt scaling, which involves fitting a logistic regression model to the scores of the SVM. This is done in a cross-validated manner and therefore may result in a different decision boundary than the from raw decision function of the model.\n",
    "So it's possible for these two methods to produce different results, expecially for small data sets.\n",
    "\n",
    "Nevertheless, we can still use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b9514ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e23349",
   "metadata": {},
   "source": [
    "The confusion matrix shows that the SVM is even worse than the logistic regression, even though this is training data. It was not able to fit it well.\n",
    "\n",
    "The reason could be the mismatch between the scoring metric used in hyperparameter tuning and feature selection, i.e. log loss, and the loss function of the model, i.e., the so-called margin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972b15ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb48da79",
   "metadata": {},
   "source": [
    "Now let us look at the trading performance:\n",
    "\n",
    "During the critical period of the global financial crisis, the model relies on the wrong of the two strategies and consequently loses relative performance that it cannot recover."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d61024",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_1.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93fb1ed",
   "metadata": {},
   "source": [
    "The distribution of returns is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b17015e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c8f653d",
   "metadata": {},
   "source": [
    "The model's Sharpe ratio is worse, but not statistically significantly worse, when we reformulate the above pair of hypotheses in a left-sided way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c337a292",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af416e43",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77814152",
   "metadata": {},
   "source": [
    "The Random Forest algorithm are a bagging based ensemble technique, which creates an ensemble of multiple decision tree models and uses the majority decision of these trees for prediction. Each individual tree in the Random Forest is trained on a random subset of the training data (called bootstrap samples) and uses a random selection of features to find the best split at each node of the tree. This randomness leads to increased diversity among individual trees and helps avoid overfitting.\n",
    "\n",
    "But why didn't we use boosting, e.g. XGBoost, which is considered the king, and rather bagging, like Random Forests.\n",
    "The reason is, that Boosting works by also correcting the Bias, which comes with greater risk of overfitting, while bagging tries to correct Variance. In finance, where data hast a very low signal to noise ratio, because every strong relationship is arbitraged away, it's very easy to overfit an AI model to noise & random patterns. So Bagging is more favorable in this field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d393a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c3499cf",
   "metadata": {},
   "source": [
    "This model has some hyperparameters that need to be specified or tuned in an informed way:\n",
    "\n",
    "- \"**n_estimators**\" is the number of trees in the forest. We tune it.\n",
    "- Kernel**\" specifies the type of kernel used (linear, poly, rbf, sigmoid) for the so-called \"kernel trick\". We need to tune this as well\n",
    "- **criterion**\" is the function to measure the quality of a split. Allthough 'gini' and 'entropy' hardly differ, we tune it as well.\n",
    "- \"**max_depth**\" is the maximum depth of trees. This is an important interdependent paramater, which has to be tuned.\n",
    "- \"**min_samples_split**\" is the minimum number of samples required to split an interior node. This has to be tuned as well.\n",
    "- \"**min_samples_leaf**\" is the minimum number of samples required to have a leaf node. This has to be tuned as well.\n",
    "- \"**min_weight_fraction_leaf**\" is the minimum weighted fraction of the total number of input samples required to be present at a leaf node. This paramater should be high enough to ensure that out-of-bag accuracy converges to out-of-sample (k-fold) accuracy.\n",
    "- \"**max_features**\" is the number of features considered for finding the best split. This has also to be tuned.\n",
    "- \"**max_leaf_nodes**\" is the maximum number of leaf nodes. This is related to 'max_depth' which is why we leave it at None.\n",
    "- \"**min_impurity_decrease**\" is a threshold for stopping tree growth early. Since the signal is low in finance, we cannot expect high decrease in impurity, so we leaf it as 0.\n",
    "- \"**bootstrap**\" decides whether to use bootstrap samples when building trees. Since this the whole idea of Bagging, we leave it at True.\n",
    "- \"**oob_score**\" decides whether to compute out-of-bag scores. We set it at at because we don't need this metric.\n",
    "- \"**n_jobs**\" determines the number of CPUs used for training. We set it to -1 such that all are used.\n",
    "- \"**verbose**\" controls the level of detail in the output. We leave it as it is.\n",
    "- \"**warm_start**\" decides whether to reuse the previous solution for the next fit. We leave it at False as it can mess up the process.\n",
    "- \"**class_weight**\" is for the weight of the classes. We set it to 'balanced' to be consistent.\n",
    "- \"**ccp_alpha**\" is the complexity parameter for the minimal cost complexity pruning. We don't use it as it can prevent the trees of being diffferent enough to drive down variance.\n",
    "- \"**max_samples**\" is the number of samples drawn for adapting each base learner. We tune this as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "556f1c0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, n_components_99),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eb7b556",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_1['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358791a9",
   "metadata": {},
   "source": [
    "(Your results may vary due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54e9ab64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaad262f",
   "metadata": {},
   "source": [
    "It is interesting that all the features have been retained this time, as if they were all important, which is highly unlikely.\n",
    "What is the reason for this?\n",
    "\n",
    "The reason for this is related to the nature of bagging. Increased performance in bagging is related to the diversity of the constituent models; ensembling models that are effectively the same does not reduce the variance in model predictions. For this reason, random forests force the trees to contain suboptimal splits of predictors using a random sample of them.\n",
    "\n",
    "This is also why we've left the 'ccp_alpha' for pruning and 'min_impurity_decrease' parameters at 0, so as not to interfere with this.\n",
    "\n",
    "Regarding the hyperparameters, we can say that the combination of a low max_depth, relatively high min_samples_split, and high min_weight_fraction_leaf suggests a highly biased model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a12a8a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340bcd04",
   "metadata": {},
   "source": [
    "We see that the Random Forest also has more variability in its predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c548595",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e329cbb",
   "metadata": {},
   "source": [
    "Unlike the SVM model above, the predicted probabilities are consistent with the class predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73bea0c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d648396",
   "metadata": {},
   "source": [
    "The accuracy is now way better than the ones of the logistic regression and SVM. However, the log loss ist still quite weak, and this is still training data, so we should take this with a grain of salt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e41e74b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "357d65cd",
   "metadata": {},
   "source": [
    "The performance is exceptionally good compared to the previous two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99c577ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_1.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57bb2185",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db21dcbc",
   "metadata": {},
   "source": [
    "If this were the test data set, the result of the Sharpe ratio test would be already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a5c284c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8554eff",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f7850db",
   "metadata": {},
   "source": [
    "Multilayer perceptrons (MLPs) are a type of artificial neural network consisting of at least three layers of neurons: an input layer, one or more \"hidden\" layers, and an output layer. Each layer is fully connected to the next, with each node receiving a weighted sum of inputs from the previous layer to which an activation function is applied.\n",
    "\n",
    "MLPs are so-called universal approximators, meaning that they can represent any function, provided there are enough neurons in the hidden layers. This allows them to model complex, nonlinear relationships between our input features and the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f799cf43",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa365948",
   "metadata": {},
   "source": [
    "MLPs have a lot of hyper paramaters:\n",
    "\n",
    "- \"**hidden_layer_sizes**\" is the number of neurons in the hidden layers. This needs to tuned, but we limit it to two layers, since this is theoretically enough to appriximate any function, given enough neurons per layer, and 20 neurons per layer to not exceed the number of input neurons.\n",
    "- \"**activation**\" is the activation function for the hidden layers. We need to tune it as well.#\n",
    "- \"**solver**\" is the optimization algorithm. We set it to 'lbfgs' since it converges better & faster for smaller data sets.\n",
    "- \"**alpha**\" is the L2 regularization term. We tune it as well.\n",
    "- \"**batch_size**\" is the size of the mini-batches for stochastic optimizers. We leave it at 'auto', but it won't be used with 'lbfgs' optimizing anyway.\n",
    "- \"**learning_rate**\" controls the type of learning rate. We set it to 'adaptive', but isn't used with 'lbfgs' anyway.\n",
    "- \"**learning_rate_init**\" is the initial learning rate. With 'lbfgs' it is not used, so we leave it as it is.\n",
    "- \"**power_t**\" is the exponent for the inverse scaling of the learning rate. We leave it as well since it is not used with 'lbfgs'.\n",
    "- \"**max_iter**\" is the maximum number of iterations. We set it to 100,000 to ensure convergence.\n",
    "- \"**shuffle**\" determines whether the samples are shuffled in each iteration. Since it is not used with 'lbfgs' we leave it as it is.\n",
    "- \"**tol**\" is the tolerance for the optimization. We leave it at 1e-4.\n",
    "- \"**verbose**\" controls the level of detail in the output. We leave it at its default.\n",
    "- \"**warm_start**\" decides whether to reuse the previous solution for the next fit. We leave it at False as it can mess up the process.\n",
    "- \"**momentum**\" is the momentum term for  gradient descent update. But we don't use 'sgd' optimizer, so we leave it at default.\n",
    "- \"**nesterovs_momentum**\" enables Nesterovs momentum. It also only important for 'sgd' optimizer.\n",
    "- \"**early_stopping**\" enables early stopping to avoid overfitting. This definitly something you want to set to True.\n",
    "- \"**validation_fraction**\" is the fraction of the training data that is retained for validation. We leave it as 10%, since this resembles a 10-fold partition.\n",
    "- \"**beta_1**\" is the exponential decay rate for estimates of the first moment vector in 'adam'. Not relevant for us.\n",
    "- \"**beta_2**\" is the exponential decay rate for estimates of the second moment vector in 'adam'. Not relevant for us, too.\n",
    "- \"**epsilon**\" is a constant for numerical stability for the 'adam' optimizer. We don't use it so we forget about it.\n",
    "- \"**n_iter_no_change**\" is the number of iterations without improvement before stopping early. Again, only relevant with 'adam'.\n",
    "- \"**max_fun**\" is the maximum number of function calls. We set it to 100,000 for consistency with 'max_iter'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d073ec65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507335c2",
   "metadata": {},
   "source": [
    "The problem with using the BayesSearchCV function from the scikit-optimize library with the MLPClassifier function from scikit-learn is that the MLPClassifier takes the number of neurons and hidden layers as a tuple (for example, (50, 50) for two hidden layers with 50 neurons each). However, the BayesSearchCV function cannot optimize tuples as parameters.\n",
    "\n",
    "To get around this problem, we need to use a little trick.\n",
    "\n",
    "We implemented a wrapper class for the MLPClassifier that takes the number of neurons and hidden layers as separate decoubled parameters. By soing so, we can use BayesSearchCV, which we definittely want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55432d29",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLPWrapper(MLPClassifier):\n",
    "    def __init__(self, layer_size=1, num_layers=1, alpha=0.0001, activation='relu'):\n",
    "        self.layer_size = layer_size\n",
    "        self.num_layers = num_layers\n",
    "        self.alpha = alpha\n",
    "        self.activation = activation\n",
    "        self.hidden_layer_sizes = tuple([self.layer_size]*self.num_layers)\n",
    "        self.model = MLPClassifier(hidden_layer_sizes=self.hidden_layer_sizes, \n",
    "                                   alpha=self.alpha, \n",
    "                                   activation=self.activation,\n",
    "                                   solver='lbfgs', \n",
    "                                   learning_rate='adaptive', \n",
    "                                   max_iter=100000, \n",
    "                                   early_stopping=True)\n",
    "        \n",
    "    def set_params(self, **parameters):\n",
    "        for parameter, value in parameters.items():\n",
    "            if parameter == \"layer_size\":\n",
    "                self.layer_size = value\n",
    "            elif parameter == \"num_layers\":\n",
    "                self.num_layers = value\n",
    "            elif parameter == \"alpha\":\n",
    "                self.alpha = value\n",
    "            elif parameter == \"activation\":\n",
    "                self.activation = value\n",
    "            else:\n",
    "                raise ValueError('Invalid parameter %s for estimator %s. '\n",
    "                                 'Check the list of available parameters '\n",
    "                                 'with `estimator.get_params().keys()`.' %\n",
    "                                 (parameter, self))\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        self.model.fit(X, y)\n",
    "        \n",
    "    def predict(self, X):\n",
    "        return self.model.predict(X)\n",
    "    \n",
    "    def score(self, X, y):\n",
    "        y_pred_proba = self.model.predict_proba(X)\n",
    "        return -log_loss(y, y_pred_proba, labels=[0.0, 1.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb43976",
   "metadata": {},
   "source": [
    "As with the SVM, recursive feature elimination cannot be combined with MLPs, since they also do not provide model-specific feature importance. For this reason, we again use sequential backward feature selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df5b2137",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_1['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 50)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b814e8d",
   "metadata": {},
   "source": [
    "(Your results may differ due to the stochastic nature of the procedure.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d0dbb76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96340e18",
   "metadata": {},
   "source": [
    "Tuning a multilayer perceptron is tricky because there is a high risk of overfitting. So what can we say about the hyperparameters?\n",
    "\n",
    "- The activation function 'tanh' is a non-linear function that allows the model to learn more complex representations. However, it is more controlled than the 'relu' function.\n",
    "\n",
    "- The alpha parameter is an L2 penalty term. A relatively small value (0.0330897095330755) indicates a lower degree of regularization, meaning that the model is allowed more complexity.\n",
    "\n",
    "- The hidden_layer_sizes setting with a single hidden layer and 8 neurons indicates a relatively simple architecture, which is favorable given that overfitting is a problem in finance.\n",
    "\n",
    "In summary, the MLPClassifier is designed with some protection against overfitting (early stopping, adaptive learning rate), but also allows some complexity (tanh activation, low regularization). The hidden layer architecture is relatively simple, which also helps to prevent overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b1e679",
   "metadata": {},
   "source": [
    "Now we can fit it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84eb10dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "389275de",
   "metadata": {},
   "source": [
    "Next we can use to to predict the Regimes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5843920b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22d43d24",
   "metadata": {},
   "source": [
    "And of course we can now predict the class probabilities. The resulting weights are quite extreme.\n",
    "It remains to be seen how well this generalizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6e92df8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebf4fd7",
   "metadata": {},
   "source": [
    "The metrics are exceptionally good, actually a little too good, indicating overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b08dd35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d280ab74",
   "metadata": {},
   "source": [
    "Performance is excellent, also a little too excellent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b18ae27",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_1.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d7aa374",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eba52ca",
   "metadata": {},
   "source": [
    "The bootstrapping test shows similar results, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24336c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "364e6dcb",
   "metadata": {},
   "source": [
    "## First Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "185af2e8",
   "metadata": {},
   "source": [
    "Now we are ready to test all four models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e223f840",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6104fe",
   "metadata": {},
   "source": [
    "Since we have transformed the training data multiple times (PCA & scaling), we need to apply exactly the same transformations to the test data to ensure consistency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89c20208",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_1.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e6f9064",
   "metadata": {},
   "source": [
    "For verification purposes, let us have a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e2ba91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0bf5f8",
   "metadata": {},
   "source": [
    "This data can now be used to create forecasts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beb3ce43",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a16eb39",
   "metadata": {},
   "source": [
    "And we can also predict probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83f1d563",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f1172f1",
   "metadata": {},
   "source": [
    "Unfortunately, the classification metrics are poor. In particular, the MLP model is particularly poor, confirming the suspicion that it is overfitted to the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d54ce30",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_1[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "507754e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_1[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74bebe67",
   "metadata": {},
   "source": [
    "Despite the poor classification metrics, the models perform solidly, including the MLP model, which actually outperforms on a non risk-adjusted basis. This is remarkable given the extreme positioning. The fact that the first three perform solidly is due to their very balanced positioning around 50/50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4fc42cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_1.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_1.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_1.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_1.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_1.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_test_1['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_test_1['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_test_1['Value Returns'] + \\\n",
    "                                           0.5 * df_test_1['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f13109",
   "metadata": {},
   "source": [
    "Let us take a look at the distribution of returns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3387cdd0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_1.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00511189",
   "metadata": {},
   "source": [
    "Finally, let's look at the Sharpe ratios of all the models.\n",
    "\n",
    "To do this, we need to redefine the corresponding bootstrapping test function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad722819",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multi_sharpe_ratio_bootstrap_test(sample1, model_samples, model_names, n_permutations=1000000):\n",
    "    \n",
    "    # Bootstrap sampling from sample1 and compute Sharpe ratios\n",
    "    sharpe_ratios = []\n",
    "    n = len(sample1)\n",
    "    for _ in range(n_permutations):\n",
    "        bootstrap_sample = np.random.choice(sample1, size=n, replace=True)\n",
    "        mean_return = np.mean(bootstrap_sample)\n",
    "        std_return = np.std(bootstrap_sample)\n",
    "        sharpe_ratio = mean_return / std_return\n",
    "        sharpe_ratios.append(sharpe_ratio)\n",
    "\n",
    "    # Plot the null distribution of Sharpe ratios\n",
    "    plt.figure(figsize=(11, 6), dpi=1000)\n",
    "    plt.hist(sharpe_ratios, bins=100, color='black')\n",
    "\n",
    "    # Assign a color for each model\n",
    "    colors = ['red', 'blue', 'green', 'orange']\n",
    "\n",
    "    # Compute the observed Sharpe ratio for each model sample\n",
    "    for model_sample, model_name, color in zip(model_samples, model_names, colors):\n",
    "        observed_mean_return = np.mean(model_sample)\n",
    "        observed_std_return = np.std(model_sample)\n",
    "        observed_sharpe_ratio = observed_mean_return / observed_std_return\n",
    "\n",
    "        # Calculate p-value: proportion of null Sharpe ratios more extreme than observed Sharpe ratio\n",
    "        p_value = (np.abs(sharpe_ratios) >= np.abs(observed_sharpe_ratio)).mean()\n",
    "\n",
    "        # Output\n",
    "        print(f\"{model_name} p-value: {p_value*100:.2f}%\")  # Print the calculated p-value as a percentage with two decimal points\n",
    "\n",
    "        # Add vertical line for the model's Sharpe ratio\n",
    "        plt.axvline(observed_sharpe_ratio, color=color, linestyle='dashed', linewidth=2,\n",
    "                    label=f'{model_name}: {observed_sharpe_ratio:.2f}')\n",
    "\n",
    "    plt.legend()\n",
    "    plt.xlabel('Sharpe Ratio')\n",
    "    plt.ylabel('frequency')\n",
    "    plt.title('Null Distribution of Benchmark Sharpe Ratio')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de20b8e",
   "metadata": {},
   "source": [
    "None of the Sharpe ratios are statistically significantly worse or better. I.e. it is not really worth using these models; especially not after transaction costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "688930c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    \n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_1.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "\n",
    "\n",
    "benchmark_returns = = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']]\n",
    "    \n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29bffd4d",
   "metadata": {},
   "source": [
    "## First further Feature Discarding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddf8fee3",
   "metadata": {},
   "source": [
    "That was the first period, and we are now discarding **Federal Reserve Assets** to extend the historical period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3389e2e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_2 = lean_sample_1.drop('Federal Reserve Assets', axis=1)\n",
    "\n",
    "display(lean_sample_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7ed6726",
   "metadata": {},
   "source": [
    "The next shortest variable is now **St. Louis Fed Financial Stress** Index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "216c2686",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_2, sort='descending', color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8eb6327",
   "metadata": {},
   "source": [
    "#### St. Louis Fed Financial Stress\t"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c38755",
   "metadata": {},
   "source": [
    "We only consider the period in which the St. Louis Fed financial stress variable is present:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "057cfdd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_2 = lean_sample_2.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e2ec2a",
   "metadata": {},
   "source": [
    "Let us plot it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b13602c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data with black color\n",
    "plt.plot(df_complete_1[\"St. Louis Fed Financial Stress\"], color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"stress level\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"St. Louis Fed Financial Stress\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8edcba",
   "metadata": {},
   "source": [
    "As expected, the highest pairwise coefficient of determination is with the one associated with the **Chicago Fed National Financial Stress** Index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03be69a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_complete_2\n",
    "column = 'St. Louis Fed Financial Stress'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "775952f4",
   "metadata": {},
   "source": [
    "Let us take a look at both to visualize the similarity. We see that the resemblance is really high, because economic represents the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2319183",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"St. Louis Fed Financial Stress\", 'Chicago Fed National Financial Stress']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_2[columns_to_plot])\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('stress level')\n",
    "plt.title('Financial Stress Inficators')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f82aa8",
   "metadata": {},
   "source": [
    "The bootstrapping test confirms the visual impression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cd3b8fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_2[\"St. Louis Fed Financial Stress\"]  # Time series 1\n",
    "ts2 = df_complete_2[\"Chicago Fed National Financial Stress\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b90c9b26",
   "metadata": {},
   "source": [
    "Therefore, we discard the St. Louis Fed's financial stress index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bcef3c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_3 = lean_sample_2.drop('St. Louis Fed Financial Stress', axis=1)\n",
    "\n",
    "display(lean_sample_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02169f6c",
   "metadata": {},
   "source": [
    "The shortest time series is now the **15Y Mortgage Rate**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3edbbdae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_3, sort='descending', color='black')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2de0969d",
   "metadata": {},
   "source": [
    "#### 15Y Mortage Rates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c57f72",
   "metadata": {},
   "source": [
    "Again, we need to shorten the time period to the one where all the features left so far are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7c0dec4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_3 = lean_sample_3.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61fa7ded",
   "metadata": {},
   "source": [
    "To get a visual impression of the variable, we plot it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2807d354",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "# Plot the data with black color\n",
    "plt.plot(df_complete_1[\"15Y Mortage Rates\"], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"15Y Mortage Rates\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3d2ab0",
   "metadata": {},
   "source": [
    "The highest similarity is with the variable 30Y Mortage Rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e9d6ec6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_complete_3\n",
    "column = '15Y Mortage Rates'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0338517b",
   "metadata": {},
   "source": [
    "The strong similarity, which can already be assumed theoretically/economically, is also confirmed visually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0ff851e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"15Y Mortage Rates\", '30Y Mortage Rates']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_3[columns_to_plot])\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Mortage Rates')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "954bbb99",
   "metadata": {},
   "source": [
    "The boostrapping test formally confirms this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3567dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_3[\"15Y Mortage Rates\"]  # Time series 1\n",
    "ts2 = df_complete_3[\"30Y Mortage Rates\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b6c13a",
   "metadata": {},
   "source": [
    "Thus, we discard **15Y Mortage Rates**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "350db817",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_4 = lean_sample_3.drop('15Y Mortage Rates', axis=1)\n",
    "\n",
    "display(lean_sample_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a15d894e",
   "metadata": {},
   "source": [
    "The shortest variable is therefore the 10-3M yield spread."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ede6862e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_4, sort='descending', color='black')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd05d218",
   "metadata": {},
   "source": [
    "#### 10-3M Yield Spreads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e4f25c",
   "metadata": {},
   "source": [
    "Alltghough it is just about 4 months, we see that we have 3M Treasury Yields and 10y Treasury Yields as distinct features and can compute them on our own resulting in more data for this particular fature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab2871c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Append the percentage changes as a new column in the DataFrame\n",
    "lean_sample_4[\"10-3M Yield Spreads\"] = lean_sample_4[\"10Y Treasury Yields\"] - lean_sample_4[\"3M Treasury Yields\"]\n",
    "\n",
    "lean_sample_4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62454904",
   "metadata": {},
   "source": [
    "As before, we restrict the data set to the period, where all features are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cce477c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_4 = lean_sample_4.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72fef033",
   "metadata": {},
   "source": [
    "This variable is interesting to visualize because it is often interpreted as a herald of financial turmoil. This makes sense because it is a proxy for the yield curve, which follows a certain logic that has clear implications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe11aba0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_4['10-3M Yield Spreads'], color='black')\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('10-3M Yield Spreads')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af7c01b",
   "metadata": {},
   "source": [
    "As expected, the most similar variable is the **10-2Y Yield Spreads**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "014cccd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the correlation matrix\n",
    "df = df_complete_4\n",
    "\n",
    "# Select the correlations related to '5/1Y Mortage Rates'\n",
    "column = '10-3M Yield Spreads'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59707e4f",
   "metadata": {},
   "source": [
    "The following visualization confirms that both represent the same economic object, the yield curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05633528",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"10-3M Yield Spreads\", '10-2Y Yield Spreads']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_4[columns_to_plot])\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Yield Spreads')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a8b374",
   "metadata": {},
   "source": [
    "The similarity is supported by the bootstrap test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4913e86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_4[\"10-3M Yield Spreads\"]  # Time series 1\n",
    "ts2 = df_complete_4[\"10-2Y Yield Spreads\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b882727",
   "metadata": {},
   "source": [
    "We therefore discard the **10-3M Yield Spreads**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac89b07e",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_5 = lean_sample_4.drop('10-3M Yield Spreads', axis=1)\n",
    "\n",
    "display(lean_sample_5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e50f10a8",
   "metadata": {},
   "source": [
    "The shortest variable is therefore the 3M Treasury Yields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e8a0eee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_5, sort='descending', color='black')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c89a3b1a",
   "metadata": {},
   "source": [
    "#### 3M Treasury Yields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "365969b7",
   "metadata": {},
   "source": [
    "The period of time is again limited to the complete one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccc26a04",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_5 = lean_sample_5.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14a1e915",
   "metadata": {},
   "source": [
    "Here is a plot again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37087996",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "plt.plot(df_complete_5['3M Treasury Yields'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"3M Treasury Yields\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a05916",
   "metadata": {},
   "source": [
    "The highest similarity is with 2Y Treasury Yields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0620d633",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the correlation matrix\n",
    "df = df_complete_5\n",
    "\n",
    "# Select the correlations related to '5/1Y Mortage Rates'\n",
    "column = '3M Treasury Yields'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d31b1ab",
   "metadata": {},
   "source": [
    "Here is a comparative plot of all Treasury Yields over all maturities available to us. They are all relatively closely correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d50c34fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"3M Treasury Yields\", '2Y Treasury Yields', '5Y Treasury Yields', \"10Y Treasury Yields\"]\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_5[columns_to_plot])\n",
    "\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Treasury Yields')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0701f070",
   "metadata": {},
   "source": [
    "The test confirms this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c66d3e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_5[\"3M Treasury Yields\"]  # Time series 1\n",
    "ts2 = df_complete_5[\"2Y Treasury Yields\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40b3fb0",
   "metadata": {},
   "source": [
    "And again, we can discard the current shortest variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c81cfc22",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_6 = lean_sample_5.drop('3M Treasury Yields', axis=1)\n",
    "\n",
    "display(lean_sample_6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd3ca42",
   "metadata": {},
   "source": [
    "The shortest variable is now **2Y Treasury Yields**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b55c6559",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_6, sort='descending', color='black')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "330d04c6",
   "metadata": {},
   "source": [
    "#### 2Y Treasury Yields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2216d42c",
   "metadata": {},
   "source": [
    "The period is shortened again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e40e9267",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_6 = lean_sample_6.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d4c7a9",
   "metadata": {},
   "source": [
    "Here is again a plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d53fefe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "plt.plot(df_complete_6[\"2Y Treasury Yields\"], color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"2Y Treasury Yields\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c854071",
   "metadata": {},
   "source": [
    "The highest similarity is with 5Y Treasury Yields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fa76d94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the correlation matrix\n",
    "df = df_complete_6\n",
    "\n",
    "# Select the correlations related to '5/1Y Mortage Rates'\n",
    "column = '2Y Treasury Yields'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a919bf",
   "metadata": {},
   "source": [
    "And again, a plot that visually supports that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78b052e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"2Y Treasury Yields\", '5Y Treasury Yields', \"10Y Treasury Yields\"]\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_6[columns_to_plot])\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Treasury Yields')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f18477",
   "metadata": {},
   "source": [
    "The bootstrapping test again confirms the visual impression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "231d676b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_6[\"2Y Treasury Yields\"]  # Time series 1\n",
    "ts2 = df_complete_6[\"5Y Treasury Yields\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "def5ff87",
   "metadata": {},
   "source": [
    "Accordingly, this variable is discarded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b145e530",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_7 = lean_sample_6.drop('2Y Treasury Yields', axis=1)\n",
    "\n",
    "display(lean_sample_6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874fc0f4",
   "metadata": {},
   "source": [
    "The shortest feature time series is now the **10-2Y Yield Spread**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "957bcc3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_7, sort='descending', color='black')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e09a9b8c",
   "metadata": {},
   "source": [
    "(Note: I will no longer comment on every step, otherwise I will go crazy. What is being done is clear by now.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd1d5180",
   "metadata": {},
   "source": [
    "#### 10-2Y Yield Spreads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "653340ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_7 = lean_sample_7.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5b3e1b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "plt.plot(df_complete_7[\"10-2Y Yield Spreads\"], color = 'black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"10-2Y Yield Spreads\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7d0061d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the correlation matrix\n",
    "df = df_complete_7\n",
    "\n",
    "# Select the correlations related to '5/1Y Mortage Rates'\n",
    "column = '10-2Y Yield Spreads'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5841dfc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"10-2Y Yield Spreads\", \"Effective Federal Funds Rate\"]\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_7[columns_to_plot])\n",
    "\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbf30e4",
   "metadata": {},
   "source": [
    "The null hypothesis is rejected for the second time, giving us the second period and set of features to model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f59fce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_7[\"10-2Y Yield Spreads\"]  # Time series 1\n",
    "ts2 = df_complete_7[\"Effective Federal Funds Rate\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ba5c34",
   "metadata": {},
   "source": [
    "## Second Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1605737f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the DataFrame by the time-related column\n",
    "df_sorted = df_complete_7.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_2 = df_sorted[:split_index]\n",
    "df_test_2 = df_sorted[split_index:]\n",
    "\n",
    "#Display\n",
    "df_train_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee8c356",
   "metadata": {},
   "source": [
    "### Feature Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afc2c42b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_2.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3913fa45",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94675235",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97ba0ece",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ba198b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "431e544b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a7d85b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "051ef78b",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ede9891",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "609aa575",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16a135e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5021f2f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_2['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d6af00",
   "metadata": {},
   "source": [
    "(Your results may vary due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82631e0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fbaf9ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e5296df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66591558",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_2['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2df11fa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5ac9e88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3cbf33e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b4dcab2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_2.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00ce8e28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9d80d2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa93693",
   "metadata": {},
   "source": [
    "#### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ef6f2cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "888e13bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f84f4fa6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_2['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94f7ff82",
   "metadata": {},
   "source": [
    "(Your results may very due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23475e08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e47b930c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01bd6eed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "769bf92f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f602d829",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20193584",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_2.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6537b56a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "974c6ae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec21b74",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2559ece9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95cd6bdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, 20),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7245659c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_2['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eb41992",
   "metadata": {},
   "source": [
    "(Your results may very due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9239df39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a96dbd5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "076cacab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a22adc79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12f303ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6407df2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_2.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ccd4df1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d608a8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82120d8b",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d09d5988",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41b57dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8383b434",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_2['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 100)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b32e11",
   "metadata": {},
   "source": [
    "(Your results may very due to the stochastic nature of the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b81cb8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99b3674c",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "#best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0ab9c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f51504b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2a4a10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad9bb00a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_2.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64e240ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67a5f9f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7a90be",
   "metadata": {},
   "source": [
    "## Second Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da601388",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e0a00bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_2.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4958e37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1898b0cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machibne\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "589a7537",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10482e0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_2[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df38a226",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_2[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23795606",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_2.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_2.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_2.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_2.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_2.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cdfba80",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_2.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b452983",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_2.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "    \n",
    "benchmark_returns = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "    \n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a30f94",
   "metadata": {},
   "source": [
    "## Second further Feature Discarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffbef38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_8 = lean_sample_7.drop('10-2Y Yield Spreads', axis=1)\n",
    "\n",
    "display(lean_sample_8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfa09655",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_8, sort='descending', color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3812b1d",
   "metadata": {},
   "source": [
    "#### 30Y Mortage Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7f7f0a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_8 = lean_sample_8.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0de2a0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_8[\"30Y Mortage Rates\"], color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"30Y Mortage Rates\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "458e421c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_8\n",
    "column = '30Y Mortage Rates'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6484e056",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"30Y Mortage Rates\", '10Y Treasury Yields']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_8[columns_to_plot])\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Yield')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cdbedfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_8[\"30Y Mortage Rates\"]  # Time series 1\n",
    "ts2 = df_complete_8[\"10Y Treasury Yields\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3d7faf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_9 = lean_sample_8.drop('30Y Mortage Rates', axis=1)\n",
    "\n",
    "display(lean_sample_9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b8331aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_9, sort='descending', color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb357203",
   "metadata": {},
   "source": [
    "#### Chicago Fed National Financial Stress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49a6f343",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_9 = lean_sample_9.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77d34ec0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi = 100)\n",
    "\n",
    "plt.plot(df_complete_9[\"Chicago Fed National Financial Stress\"], color = 'black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"stress\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Chicago Fed National Financial Stress\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5da98a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_9\n",
    "column = 'Chicago Fed National Financial Stress'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e722400",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"Chicago Fed National Financial Stress\", 'Effective Federal Funds Rate']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_9[columns_to_plot])\n",
    "\n",
    "# Show a legend\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('')\n",
    "plt.title('')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94f108ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_9[\"Chicago Fed National Financial Stress\"]  # Time series 1\n",
    "ts2 = df_complete_9[\"Effective Federal Funds Rate\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ec2b7de",
   "metadata": {},
   "source": [
    "## Third Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3764c1ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the DataFrame by the time-related column\n",
    "df_sorted = df_complete_9.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_3 = df_sorted[:split_index]\n",
    "df_test_3 = df_sorted[split_index:]\n",
    "\n",
    "#Display\n",
    "df_train_3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "851ee193",
   "metadata": {},
   "source": [
    "### Feature Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bc477c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_3.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8de68c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93f4024a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c992d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dab510d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fa5217f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6b9a6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53fd4797",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6f57852",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdc793d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "437d8b31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b57fbe7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_3['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33477d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3d43b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93a1caa2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "438305ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_3['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59fc1757",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ff7ab7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a30c35e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "131057cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_3.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff9bfe9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1410ccc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade3afd5",
   "metadata": {},
   "source": [
    "#### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98a522aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b930d79c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a68cd6d1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_3['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b58cd992",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe470d80",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe327310",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cae365d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "344bca7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53432e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_3.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7479b622",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28a7797f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eaaaee4",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf55f357",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4a21a83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, 20),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0dc5fb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_3['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35bc97c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0461f6bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20eb205c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "383432f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b564e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0783b339",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_3.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "581f0977",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a441cc1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e44d8c8",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69c37d58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08b8eafb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e4d334",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_3['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 100)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a709c97d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeea4367",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "#best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27029ad8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "436fd80a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1988d577",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b3350c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_3.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2e4caa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b21254ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f17057f",
   "metadata": {},
   "source": [
    "## Third Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5d1e6c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85d6e548",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_3.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fab13356",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41d9315e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machibne\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d97757e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bca20a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_3[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52a89fa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_3[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8966313",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_3.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_3.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_3.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_3.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_3.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9317d99",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_3.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns'].shift()\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3eceaed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_3.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "    \n",
    "benchmark_returns = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3780b3a0",
   "metadata": {},
   "source": [
    "## Second further Feature Discarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f11c0977",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_10 = lean_sample_9.drop('Chicago Fed National Financial Stress', axis=1)\n",
    "\n",
    "display(lean_sample_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b634dcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_10, sort='descending', color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bfe522c",
   "metadata": {},
   "source": [
    "#### Sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "085eb9f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_10 = lean_sample_10.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f02390",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_10[\"Sentiment\"], color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Sentiment\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c5e05c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_10\n",
    "column = 'Sentiment'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ade1bc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"Sentiment\", 'CAPE']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_10[columns_to_plot])\n",
    "\n",
    "# Show a legend\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('')\n",
    "plt.title('')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a89674f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_10[\"Sentiment\"]  # Time series 1\n",
    "ts2 = df_complete_10[\"CAPE\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fd217fa",
   "metadata": {},
   "source": [
    "## Fourth Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdedd01e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the DataFrame by the time-related column\n",
    "df_sorted = df_complete_10.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_4 = df_sorted[:split_index]\n",
    "df_test_4 = df_sorted[split_index:]\n",
    "\n",
    "#Display\n",
    "df_train_4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb6c928f",
   "metadata": {},
   "source": [
    "### Feature Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "febdb3f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_4.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53596462",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf1a515c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9a1de69",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8fc6709",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "104a017e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30529f5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d948887e",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440f6a37",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e462034b",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2ef399b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ecb6f36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_4['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2bf837e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3038c28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebbe995",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa082b20",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_4['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e67d6a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49cbdfb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f83acf0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a947d55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_4.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e6f3157",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2877242b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "078c7b11",
   "metadata": {},
   "source": [
    "#### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02683f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68df1bc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30c81d46",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_4['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4311bae1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55baeae8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85d7e8ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "699914ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882eb3dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c50c49c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_4.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa29aae7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4cd415b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99934740",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87e0ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3e48855",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, 20),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c849f0b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_4['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8621a640",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "533e7ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "937362dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c9d8ff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff07190e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "557c174d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_4.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8988036",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8ab7ed9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342e3884",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd1c4869",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7771740",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54fc32e1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_4['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 100)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df78faf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cd4c7ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "#best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81abc75e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca363479",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "425b69e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "763a623a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_4.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "385a71c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "947f93d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ad6a42c",
   "metadata": {},
   "source": [
    "## Fourth Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffcc7476",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9636b6b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_4.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e2a1c7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e580a65a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machibne\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33ab7d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dda4e061",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_4[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1605e793",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_4[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23eef286",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_4.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_4.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_4.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_4.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_4.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_test_4['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_test_4['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_test_4['Value Returns'] + \\\n",
    "                                           0.5 * df_test_4['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0784eb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_4.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87848cc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_4.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "\n",
    "benchmark_returns = 0.5 * df_merged['Value Returns'] + \\\n",
    "                    0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3d1ddda",
   "metadata": {},
   "source": [
    "## Third further Feature Discarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21e6e475",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_11 = lean_sample_10.drop('Sentiment', axis=1)\n",
    "\n",
    "display(lean_sample_11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8f4482",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_11, sort='descending', color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e423622e",
   "metadata": {},
   "source": [
    "#### 5Y Treasury Yield"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b932a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_11 = lean_sample_11.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb5e77b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_11[\"5Y Treasury Yields\"], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"5Y Treasury Yields\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c44e8ca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_11\n",
    "column = '5Y Treasury Yields'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11d6f9c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = ['5Y Treasury Yields', '10Y Treasury Yields']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_11[columns_to_plot])\n",
    "\n",
    "# Show a legend\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('Treasury Yields')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce4f6b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_11[\"5Y Treasury Yields\"]  # Time series 1\n",
    "ts2 = df_complete_11[\"10Y Treasury Yields\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5bca555",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_12 = lean_sample_11.drop('5Y Treasury Yields', axis=1)\n",
    "\n",
    "display(lean_sample_12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2524561",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_12, sort='descending', color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "059dbaea",
   "metadata": {},
   "source": [
    "#### 10Y Treasury Yield"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0882ec0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_12 = lean_sample_12.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31aec20c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_12[\"10Y Treasury Yields\"], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"10Y Treasury Yields\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c7ec3c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_12\n",
    "column = '10Y Treasury Yields'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2859517a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"10Y Treasury Yields\", 'Effective Federal Funds Rate']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_12[columns_to_plot])\n",
    "\n",
    "# Show a legend\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('%')\n",
    "plt.title('')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2772131",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_12[\"10Y Treasury Yields\"]  # Time series 1\n",
    "ts2 = df_complete_12[\"Effective Federal Funds Rate\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e9c0dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_13 = lean_sample_12.drop('10Y Treasury Yields', axis=1)\n",
    "\n",
    "display(lean_sample_13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c4d57a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_13, sort='descending', color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9af3c5dd",
   "metadata": {},
   "source": [
    "#### Effective Federal Funds Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a4b491a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_13 = lean_sample_13.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f02bf73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "plt.plot(df_complete_13[\"Effective Federal Funds Rate\"], color = 'black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"%\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Effective Federal Funds Rate\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44c57568",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df is your DataFrame and column is the name of the column\n",
    "df = df_complete_13\n",
    "column = 'Effective Federal Funds Rate'\n",
    "\n",
    "plot_r_squared(df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9fb4591",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Select the desired columns from the DataFrame\n",
    "columns_to_plot = [\"Effective Federal Funds Rate\", 'CAPE']\n",
    "\n",
    "# Plotting the columns\n",
    "plt.plot(df_complete_13[columns_to_plot])\n",
    "\n",
    "# Show a legend\n",
    "plt.legend(columns_to_plot)\n",
    "\n",
    "# Add labels and title to the plot\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('')\n",
    "plt.title('')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944c2b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define time series\n",
    "ts1 = df_complete_13[\"Effective Federal Funds Rate\"]  # Time series 1\n",
    "ts2 = df_complete_13[\"CAPE\"]  # Time series 2\n",
    "\n",
    "# Call the function\n",
    "r_squared_bootstrap_test(ts1, ts2, n_permutations=1000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b1e383",
   "metadata": {},
   "source": [
    "## Fifth Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3777ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the DataFrame by the time-related column\n",
    "df_sorted = df_complete_13.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_5 = df_sorted[:split_index]\n",
    "df_test_5 = df_sorted[split_index:]\n",
    "\n",
    "#Display\n",
    "df_train_5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ca92e2",
   "metadata": {},
   "source": [
    "### Feature Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52bdb544",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_5.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfaa2a34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98179c1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bb6e11f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea118d9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "978d58ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "975f8690",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd3654e0",
   "metadata": {},
   "source": [
    "### Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79803f2e",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eebe286b",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "652962ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "455cc8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_5['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15fd2271",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ec5ad13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abb4b112",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f47e2b69",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_5['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269476c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73cacb99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c142967",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a61dbb51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_5.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eba05a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8668f90",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "034d6b8e",
   "metadata": {},
   "source": [
    "#### Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accd5ca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fffee44a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17aa633e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_5['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e195ec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e2b72d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79ebfb49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38ae87b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49bb6f5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8de0f666",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_5.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ddbf38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3dacd8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1bb55c7",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c4dbbab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df638bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, 20),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b4717a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_5['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c866dda3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4f3fc1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68ebcce3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6d8c65c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90f3e8f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebe55569",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_5.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d40b98be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa1615d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6de8322",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52868df3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcfbc0b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8026de11",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_5['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 100)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b46a30e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8828949",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "#best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "030cba6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91128629",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "546bb6e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "104797fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_5.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "361f46c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9870cbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b321e4ea",
   "metadata": {},
   "source": [
    "## Fifth Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc79e166",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d160cc1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_5.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26112923",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "180fb071",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machibne\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4c591e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e4bb2d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_5[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57ccc1c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_5[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "076236b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_5.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_5.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_5.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_5.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_5.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_test_5['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_test_5['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_test_5['Value Returns'] + \\\n",
    "                                           0.5 * df_test_5['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63fe87d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_5.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c921ab07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_5.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "benchmark_returns = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "    \n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1ce217a",
   "metadata": {},
   "source": [
    "# Fourth further Feature Discarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ef9ff3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "lean_sample_14 = lean_sample_13.drop('Effective Federal Funds Rate', axis=1)\n",
    "\n",
    "display(lean_sample_14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1199c33f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Use the missingno package to visualize the completeness of the 'df_monthly' DataFrame\n",
    "msno.bar(lean_sample_14, sort='descending', color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"Features\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"Missing Values\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Missing Data by Feature\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "371bcaef",
   "metadata": {},
   "source": [
    "# Final Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1106faef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any rows that have missing values\n",
    "df_complete_14 = lean_sample_14.dropna()\n",
    "\n",
    "# Display the smaller DataFrame\n",
    "display(df_complete_14)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c94517f",
   "metadata": {},
   "source": [
    "# Final Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbed486f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming df_complete_1 is your DataFrame and target_variable is the name of your target variable column\n",
    "\n",
    "# Sort the DataFrame by the time-related column (e.g., date)\n",
    "df_sorted = df_complete_14.sort_values('Date')\n",
    "\n",
    "# Calculate the index to split the data (e.g., last 20%)\n",
    "split_index = int(0.7 * len(df_sorted))\n",
    "\n",
    "# Split the data into training and test sets\n",
    "df_train_6 = df_sorted[:split_index]\n",
    "df_test_6 = df_sorted[split_index:]\n",
    "\n",
    "df_train_6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77e39f7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the target variable\n",
    "X = df_train_6.drop('Regime', axis=1)\n",
    "\n",
    "# Initialize an empty matrix for R^2 values\n",
    "r2_matrix = np.zeros((len(X.columns), len(X.columns)))\n",
    "\n",
    "# Calculate R^2 values for each pair of variables\n",
    "for i, col1 in enumerate(X.columns):\n",
    "    for j, col2 in enumerate(X.columns):\n",
    "        if i != j:\n",
    "            # Fit a linear regression model\n",
    "            lr = LinearRegression()\n",
    "            lr.fit(X[col1].values.reshape(-1, 1), X[col2])\n",
    "\n",
    "            # Calculate R^2 score\n",
    "            r2 = lr.score(X[col1].values.reshape(-1, 1), X[col2])\n",
    "            r2_matrix[i, j] = r2\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "linkage_matrix = hierarchy.linkage(r2_matrix, method='complete')\n",
    "\n",
    "# Obtain the order of rows and columns based on the dendrogram\n",
    "order = hierarchy.dendrogram(linkage_matrix, no_plot=True)['leaves']\n",
    "\n",
    "# Sort the R^2 matrix based on the order\n",
    "sorted_r2_matrix = pd.DataFrame(r2_matrix[order, :][:, order], index=X.columns[order], columns=X.columns[order])\n",
    "\n",
    "# Plot the sorted R^2 matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(sorted_r2_matrix, annot=False, cmap='coolwarm', cbar=True)\n",
    "plt.title('Sorted R^2 Matrix with Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccfa0dd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of StandardScaler and perform scaling\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Create an instance of PCA and perform PCA transformation\n",
    "pca = PCA()\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Create a new DataFrame with the date index and components\n",
    "df_pca = pd.DataFrame(X_pca, index=X.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_pca.columns = [f\"PC {i+1}\" for i in range(X_pca.shape[1])]\n",
    "\n",
    "# Print the new DataFrame\n",
    "df_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bde79d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cumulative explained variance ratio\n",
    "explained_variance_ratio_cumulative = np.cumsum(pca.explained_variance_ratio_)\n",
    "\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "plt.plot(range(1, len(explained_variance_ratio_cumulative) + 1), explained_variance_ratio_cumulative, color = 'black')\n",
    "plt.xlabel('Number of Components')\n",
    "plt.ylabel('Cumulative Explained Variance Ratio')\n",
    "plt.title('Cumulative Explained Variance')\n",
    "plt.grid(True)\n",
    "\n",
    "# Determine the number of components explaining at least 99% of variance\n",
    "n_components_99 = np.argmax(explained_variance_ratio_cumulative >= 0.99) + 1\n",
    "print(\"Number of components explaining at least 99% of variance:\", n_components_99)\n",
    "\n",
    "# Add a vertical line at the number of components where 95% is reached\n",
    "plt.axvline(x=n_components_99, color='red', linestyle='--')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "939fa3f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only the principal components explaining at least 99% of variance\n",
    "df_pca_99 = df_pca.iloc[:, :n_components_99]\n",
    "\n",
    "#Display\n",
    "df_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5851649d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce051e76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StandardScaler\n",
    "pca_scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler to the data and transform the data\n",
    "df_pca_99_scaled = pca_scaler.fit_transform(df_pca_99)\n",
    "\n",
    "# If you want to convert the scaled data back to a DataFrame:\n",
    "df_pca_99_scaled = pd.DataFrame(df_pca_99_scaled, index=df_pca_99.index, columns=df_pca_99.columns)\n",
    "\n",
    "df_pca_99_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "228c3978",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_pca_99_scaled)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f34ce9e",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31954a24",
   "metadata": {},
   "source": [
    "#### Logistic Regression (baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb48e649",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression(penalty = 'elasticnet',\n",
    "                             class_weight = 'balanced',\n",
    "                             solver = 'saga',\n",
    "                             l1_ratio=0.5,\n",
    "                             max_iter =100000,\n",
    "                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f958434",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'l1_ratio': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "746b9022",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_6['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(log_reg, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=log_reg,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_log_reg = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_log_reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7325bd5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_log_reg])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36af857a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "log_reg_final = LogisticRegression(penalty='elasticnet',\n",
    "                                   class_weight='balanced',\n",
    "                                   solver='saga',\n",
    "                                   max_iter=100000,\n",
    "                                   n_jobs=-1,\n",
    "                                   **best_hyperparameters)\n",
    "# Fit the model\n",
    "log_reg_final.fit(X[best_features_log_reg], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5c3cabd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Coeffiencts\n",
    "print(log_reg_final.coef_)\n",
    "\n",
    "# Intercept\n",
    "print(log_reg_final.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f08850c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "proportion = df_train_6['Regime'].value_counts(normalize=True)\n",
    "print(proportion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4efabc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = log_reg_final.predict(X[best_features_log_reg])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Logistic Regression (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75aa0367",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Logistic Regression (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df73aa9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f1fb0f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_6.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Logistic Regression Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Logistic Regression Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a3d80aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Logistic Regression Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Logistic Regression Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a0c3ef2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a3157a",
   "metadata": {},
   "source": [
    "#### Support Vector Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e274846",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Support Vector Classifier\n",
    "svm = SVC(shrinking=True,\n",
    "          probability=True,\n",
    "          cache_size=1000,\n",
    "          class_weight='balanced',\n",
    "          decision_function_shape ='ovo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6276d473",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'C': Real(0.01, 100, prior='log-uniform'),\n",
    "    'kernel': Categorical(['linear', 'poly', 'rbf', 'sigmoid']),\n",
    "    'degree': Integer(1, 10),\n",
    "    'gamma': Categorical(['scale', 'auto']),\n",
    "    'coef0': Real(0, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "045d441e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_6['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(svm,\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=svm,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_svm = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51adc0d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_svm])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76544110",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the final model on all of the training data using the best feature subset and the best hyperparameters\n",
    "svm_final = SVC(shrinking=True,\n",
    "                probability=True,\n",
    "                cache_size=1000,\n",
    "                class_weight='balanced',\n",
    "                decision_function_shape ='ovo',\n",
    "                **best_hyperparameters)\n",
    "# Fit the model\n",
    "svm_final.fit(X[best_features_svm], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc7b5dbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = svm_final.predict(X[best_features_svm])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Support Vector Machine (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef25ef3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = svm_final.predict_proba(X[best_features_svm])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Support Vector Machine (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49b12b3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25f39a49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_6.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Support Vector Machine Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Support Vector Machine Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba917b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Support Vector Machine Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Support Vector Machine Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "879cf384",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c98f1b",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8cd1e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf = RandomForestClassifier(n_jobs=-1,\n",
    "                            class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "970c9dce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid you want to search over\n",
    "param_dist = {\n",
    "    'n_estimators': Integer(100, 1000),  # Number of trees in the forest\n",
    "    'criterion': Categorical(['gini', 'entropy']),\n",
    "    'max_depth': Integer(1, n_components_99),  # Maximum number of levels in each decision tree\n",
    "    'min_samples_split': Integer(2, 10),  # Minimum number of data points placed in a node before the node is split\n",
    "    'min_samples_leaf': Integer(1, 10),  # Minimum number of data points allowed in a leaf node\n",
    "    'min_weight_fraction_leaf': Real(0.05, 0.2),  # Minimum weighted fraction of the total population required to be at a leaf node\n",
    "    'max_features': Categorical(['sqrt', 'log2']),  # Number of features to consider at every split\n",
    "    'max_samples': Real(0.01, 1.0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34e24549",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_6['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = RFECV(rf, step=1, cv=tscv_outer, scoring=scorer, n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=rf,\n",
    "                                 search_spaces=param_dist,\n",
    "                                 cv=tscv_inner,\n",
    "                                 scoring=scorer,\n",
    "                                 n_iter=100,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_rf = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_rf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cae4705",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_rf])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a5bf52f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Random Forest Classifier\n",
    "rf_final = RandomForestClassifier(n_jobs=-1,\n",
    "                                  class_weight='balanced',\n",
    "                                  **best_hyperparameters)\n",
    "# Fit the model\n",
    "rf_final.fit(X[best_features_rf], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16b47cc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = rf_final.predict(X[best_features_rf])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Random Forest (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2629e1fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = rf_final.predict_proba(X[best_features_rf])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Random Forest (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ad8b5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b9e563",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_6.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Random Forest Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Random Forest Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c2f405e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Random Forest Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Random Forest Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f87e3a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79bba772",
   "metadata": {},
   "source": [
    "#### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d1a257",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Multi-layer Perceptron Classifier\n",
    "mlp = MLPClassifier(solver = 'lbfgs',\n",
    "                    learning_rate = 'adaptive',\n",
    "                    max_iter = 100000,\n",
    "                    early_stopping = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "478644c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter distributions\n",
    "param_dist = {\n",
    "    'clf__layer_size': Integer(1, int(n_components_99/2)),\n",
    "    'clf__num_layers': Integer(1, 2),\n",
    "    'clf__alpha': Real(1e-6, 1e-1, prior='log-uniform'),\n",
    "    'clf__activation': Categorical(['identity', 'logistic', 'tanh', 'relu'])\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32d31ed2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Measure the time needed to execute this cell\n",
    "start_time = time.time()\n",
    "\n",
    "# Wrapper Pipeline\n",
    "pipe = Pipeline([\n",
    "    ('clf', MLPWrapper())\n",
    "])\n",
    "\n",
    "# Define data\n",
    "X = df_pca_99_scaled\n",
    "y = df_train_6['Regime']\n",
    "\n",
    "# Lists to store results\n",
    "best_params = []\n",
    "best_scores = []\n",
    "selected_features_list = []\n",
    "\n",
    "# Outer loop for feature selection\n",
    "for train_index_outer, test_index_outer in tscv_outer.split(X):\n",
    "    X_train_outer, _ = X.iloc[train_index_outer], X.iloc[test_index_outer]\n",
    "    y_train_outer, _ = y.iloc[train_index_outer], y.iloc[test_index_outer]\n",
    "    \n",
    "    # Recursive feature elimination\n",
    "    selector = SequentialFeatureSelector(MLPWrapper(),\n",
    "                                         n_features_to_select='auto',\n",
    "                                         direction='backward',\n",
    "                                         tol=None,\n",
    "                                         cv=tscv_outer,\n",
    "                                         #scoring=scorer,\n",
    "                                         n_jobs=-1)\n",
    "    selector = selector.fit(X_train_outer, y_train_outer)\n",
    "    selected_features = X_train_outer.columns[selector.support_]\n",
    "    selected_features_list.append(selected_features)\n",
    "    \n",
    "    X_train_outer_selected = X_train_outer[selected_features]\n",
    "    \n",
    "    # Initialize the Bayes Search CV object for the internal loop\n",
    "    bayes_search = BayesSearchCV(estimator=pipe,\n",
    "                                 search_spaces=[(param_dist, 100)],\n",
    "                                 cv=tscv_inner,\n",
    "                                 #scoring=scorer,\n",
    "                                 n_jobs=-1)\n",
    "\n",
    "    # Inner loop for hyperparameter tuning\n",
    "    bayes_search.fit(X_train_outer_selected, y_train_outer)\n",
    "    \n",
    "    # Store the best parameters and the best score\n",
    "    best_params.append(bayes_search.best_params_)\n",
    "    best_scores.append(bayes_search.best_score_)\n",
    "    \n",
    "# Find the index of the best score\n",
    "best_index = np.argmax(best_scores)\n",
    "\n",
    "# Find the best hyperparameters\n",
    "best_hyperparameters = best_params[best_index]\n",
    "\n",
    "# Find the best feature subset\n",
    "best_features_mlp = selected_features_list[best_index]\n",
    "\n",
    "# End Time\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(\"Execution time:\", execution_time / 60, \"min\")\n",
    "\n",
    "print(\"Best hyperparameters: \", best_hyperparameters)\n",
    "print(\"Best feature subset: \", best_features_mlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eeeed97",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(X[best_features_mlp])\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Selected Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2144cae7",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_dict = dict(best_hyperparameters)\n",
    "\n",
    "activation = hyperparameters_dict['clf__activation']\n",
    "alpha = hyperparameters_dict['clf__alpha']\n",
    "num_layers = hyperparameters_dict['clf__num_layers']\n",
    "layer_size = hyperparameters_dict['clf__layer_size']\n",
    "\n",
    "hidden_layer_sizes = tuple(layer_size for _ in range(num_layers))\n",
    "\n",
    "# Convert the Index to a list\n",
    "#best_features = best_features.tolist()\n",
    "\n",
    "# Initialize the Random Forest Classifier\n",
    "mlp_final = MLPClassifier(solver = 'lbfgs',\n",
    "                          learning_rate = 'adaptive',\n",
    "                          max_iter = 100000,\n",
    "                          early_stopping = True,\n",
    "                          activation=activation,\n",
    "                          alpha=alpha,\n",
    "                          hidden_layer_sizes=hidden_layer_sizes)\n",
    "# Fit the model\n",
    "mlp_final.fit(X[best_features_mlp], y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f2d9517",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict classes\n",
    "y_pred = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Convert predictions to a pandas dataframe with time index\n",
    "df_predictions = pd.DataFrame(y_pred, columns=['Predictions'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_predictions.index, df_predictions['Predictions'], color='black')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel(\"regime\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Predicted Regime by Multi-layer Perceptron (Training)\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9622972a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict probabilities\n",
    "y_proba = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Convert probabilities to a pandas DataFrame with time index\n",
    "df_proba = pd.DataFrame(y_proba, columns=['Probability_0', 'Probability_1'], index=df_pca.index)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the stacked graph of predicted probabilities\n",
    "plt.stackplot(df_proba.index, df_proba['Probability_0'], df_proba['Probability_1'],\n",
    "              labels=['Value', 'Momentum'],\n",
    "              colors=['blue', 'black'])\n",
    "\n",
    "# Add a horizontal line\n",
    "plt.axhline(y=0.5, color='red', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel('time')\n",
    "\n",
    "# Set y-axis label\n",
    "plt.ylabel('probability')\n",
    "\n",
    "# Set plot title\n",
    "plt.title('Predicted Probabilities by Multi-layer Perceptron (Training)')\n",
    "\n",
    "# Show a legend\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "220551f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate confusion matrix\n",
    "cm = confusion_matrix(y, y_pred, normalize='all')\n",
    "accuracy = accuracy_score(y, y_pred)\n",
    "logloss = log_loss(y, y_proba)\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Log Loss: {logloss}\")\n",
    "\n",
    "# Create a heat map from the confusion matrix\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "sns.heatmap(cm, annot=True, cmap='coolwarm')\n",
    "\n",
    "# Add labels to the plot\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "282af10a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the DataFrames based on their indexes\n",
    "df_merged = df_train_6.merge(df_proba, left_index=True, right_index=True)\n",
    "\n",
    "# Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "# Calculate the weighted returns using the shifted probabilities\n",
    "df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + 0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "# Compute monthly returns\n",
    "model_monthly_returns = df_merged['Weighted Returns']\n",
    "benchmark_monthly_returns = df_merged['Simple Weighted Returns']\n",
    "\n",
    "# Drop rows with NaN values\n",
    "df_merged = df_merged.dropna()\n",
    "\n",
    "# Compute Sharpe ratio based on monthly returns\n",
    "model_sharpe_ratio = model_monthly_returns.mean() / model_monthly_returns.std()\n",
    "benchmark_sharpe_ratio = benchmark_monthly_returns.mean() / benchmark_monthly_returns.std()\n",
    "\n",
    "# Calculate the cumulative returns\n",
    "df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "# Sharpe Ratio\n",
    "print(\"Model Sharpe Ratio:\", model_sharpe_ratio)\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)\n",
    "\n",
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the cumulative returns on a logarithmic scale\n",
    "plt.plot(df_merged['Cumulative Weighted Returns'], label='Multi-layer Perceptron Model', color=\"red\")\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Multi-layer Perceptron Model (Training)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dfc612d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the densities of monthly returns\n",
    "plt.hist(model_monthly_returns, bins=100, alpha=0.5, label='Multi-layer Perceptron Model', color='red')\n",
    "plt.hist(benchmark_monthly_returns, bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "plt.axvline(0, color='red', linestyle='--')\n",
    "plt.legend()\n",
    "plt.xlabel('monthly returns')\n",
    "plt.ylabel('frequency')\n",
    "plt.title('Return Histogram of Multi-layer Perceptron Model (Training)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0333b80",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the sample to test\n",
    "sample1= benchmark_monthly_returns # Benchmark\n",
    "sample2= model_monthly_returns # Model\n",
    "\n",
    "# Bootstrapping test\n",
    "sharpe_ratio_bootstrap_test(sample1, sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b546ad17",
   "metadata": {},
   "source": [
    "## Final Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f79373f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test_6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "293c8efd",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test_6.drop(\"Regime\", axis=1)\n",
    "\n",
    "# Perform scaling on the test set using the same scaler\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform PCA transformation on the scaled test set using the same PCA instance\n",
    "X_test_pca = pca.transform(X_test_scaled)\n",
    "\n",
    "# Cap the data\n",
    "X_test_pca = X_test_pca[:, :n_components_99]\n",
    "\n",
    "# Fit the scaler to the PCA data and transform the data\n",
    "df_test_pca = pca_scaler.transform(X_test_pca)\n",
    "\n",
    "# Create a new DataFrame with the transformed test set\n",
    "df_test_pca_99 = pd.DataFrame(df_test_pca, index=X_test.index)\n",
    "\n",
    "# Rename the columns to indicate the component number\n",
    "df_test_pca_99.columns = [f\"PC {i+1}\" for i in range(X_test_pca.shape[1])]\n",
    "\n",
    "# Print the transformed test set DataFrame\n",
    "df_test_pca_99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe02b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "# Plot the data\n",
    "plt.plot(df_test_pca_99)\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(0, color='black', linestyle='--')\n",
    "\n",
    "# Set x-axis label\n",
    "plt.xlabel(\"time\")\n",
    "\n",
    "# Set plot title\n",
    "plt.title(\"Scaled PCA Transformed Test Features\")\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3c3363e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_test_pca_99\n",
    "\n",
    "# Predict classes\n",
    "y_log_reg = log_reg_final.predict(X[best_features_log_reg])\n",
    "y_svm = svm_final.predict(X[best_features_svm])\n",
    "y_rf = rf_final.predict(X[best_features_rf])\n",
    "y_mlp = mlp_final.predict(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['black', 'red', 'green', 'blue']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].plot(df_test_pca_99.index, y_log_reg, color=colors[0], label='Logistic Regression')\n",
    "axs[1].plot(df_test_pca_99.index, y_svm, color=colors[1], label='SVM')\n",
    "axs[2].plot(df_test_pca_99.index, y_rf, color=colors[2], label='Random Forest')\n",
    "axs[3].plot(df_test_pca_99.index, y_mlp, color=colors[3], label='MLP')\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machibne\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Regime by Models (Test)\")\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d96419",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predicted probabilities for each model\n",
    "y_log_reg_prob = log_reg_final.predict_proba(X[best_features_log_reg])\n",
    "y_svm_prob = svm_final.predict_proba(X[best_features_svm])\n",
    "y_rf_prob = rf_final.predict_proba(X[best_features_rf])\n",
    "y_mlp_prob = mlp_final.predict_proba(X[best_features_mlp])\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(4, 1, figsize=(11, 11), dpi=100, sharex=True)\n",
    "\n",
    "# Define colors for each model\n",
    "colors = ['blue', 'black', 'green', 'red']\n",
    "\n",
    "# Plot the data on each subplot with distinct colors\n",
    "axs[0].stackplot(df_test_pca_99.index, y_log_reg_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[1].stackplot(df_test_pca_99.index, y_svm_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[2].stackplot(df_test_pca_99.index, y_rf_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "axs[3].stackplot(df_test_pca_99.index, y_mlp_prob.T, colors=colors, labels=['Value', 'Momentum'])\n",
    "\n",
    "# Set x-axis label for the bottom subplot\n",
    "axs[3].set_xlabel(\"time\")\n",
    "\n",
    "# Set y-axis label for each subplot\n",
    "axs[0].set_ylabel(\"Logistic Regression\")\n",
    "axs[1].set_ylabel(\"Support Vector Machine\")\n",
    "axs[2].set_ylabel(\"Random Forest\")\n",
    "axs[3].set_ylabel(\"Multi-Layer Perceptron\")\n",
    "\n",
    "# Set plot title\n",
    "plt.suptitle(\"Predicted Probabilities by Models (Test)\")\n",
    "\n",
    "# Add legend to the plot\n",
    "handles, labels = axs[0].get_legend_handles_labels()\n",
    "fig.legend(handles, labels)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12515dc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_test_6[\"Regime\"]\n",
    "\n",
    "# Calculate confusion matrices for each model\n",
    "cm1 = confusion_matrix(y, y_log_reg, normalize='all')\n",
    "cm2 = confusion_matrix(y, y_svm, normalize='all')\n",
    "cm3 = confusion_matrix(y, y_rf, normalize='all')\n",
    "cm4 = confusion_matrix(y, y_mlp, normalize='all')\n",
    "\n",
    "# Set figure size and dpi\n",
    "fig, axs = plt.subplots(2, 2, figsize=(11, 11), dpi=100)\n",
    "\n",
    "# Generate confusion matrix and metrics for each model\n",
    "cms = [cm1, cm2, cm3, cm4]\n",
    "models = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Percetron']\n",
    "metrics = []\n",
    "\n",
    "for i, cm in enumerate(cms):\n",
    "    # Calculate metrics\n",
    "    accuracy = np.diag(cm).sum() / cm.sum()\n",
    "    logloss = -np.log(np.diag(cm) / np.sum(cm, axis=1))\n",
    "    \n",
    "    # Store metrics\n",
    "    metrics.append((accuracy, logloss))\n",
    "    \n",
    "    # Plot confusion matrix\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    sns.heatmap(cm, annot=True, cmap='coolwarm', ax=ax)\n",
    "    ax.set_title(f'{models[i]}')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "401dc7da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# assuming y_true is your ground truth\n",
    "y_true = df_test_6[\"Regime\"]\n",
    "\n",
    "# Calculate accuracy scores\n",
    "log_reg_acc = accuracy_score(y_true, y_log_reg)\n",
    "svm_acc = accuracy_score(y_true, y_svm)\n",
    "rf_acc = accuracy_score(y_true, y_rf)\n",
    "mlp_acc = accuracy_score(y_true, y_mlp)\n",
    "\n",
    "# Calculate log loss scores\n",
    "log_reg_loss = log_loss(y_true, y_log_reg_prob)\n",
    "svm_loss = log_loss(y_true, y_svm_prob)\n",
    "rf_loss = log_loss(y_true, y_rf_prob)\n",
    "mlp_loss = log_loss(y_true, y_mlp_prob)\n",
    "\n",
    "# Create a dataframe to display\n",
    "df = pd.DataFrame({\n",
    "    'Model': ['Logistic Regression', 'Suport Vector Machine', 'Random Forest', 'Multi-layer Perceptron'],\n",
    "    'Accuracy': [log_reg_acc, svm_acc, rf_acc, mlp_acc],\n",
    "    'Log Loss': [log_reg_loss, svm_loss, rf_loss, mlp_loss]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76558093",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set figure size and dpi\n",
    "plt.figure(figsize=(11, 6), dpi=100)\n",
    "\n",
    "df_log_reg_prob = pd.DataFrame(y_log_reg_prob, index=df_test_6.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_svm_prob = pd.DataFrame(y_svm_prob, index=df_test_6.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_rf_prob = pd.DataFrame(y_rf_prob, index=df_test_6.index, columns=['Probability_0', 'Probability_1'])\n",
    "df_mlp_prob = pd.DataFrame(y_mlp_prob, index=df_test_6.index, columns=['Probability_0', 'Probability_1'])\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                               ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_6.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_test_6['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_test_6['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_test_6['Value Returns'] + \\\n",
    "                                           0.5 * df_test_6['Momentum Returns']\n",
    "    \n",
    "    # Compute Sharpe ratios\n",
    "    model_sharpe_ratio = df_merged['Weighted Returns'].mean() / df_merged['Weighted Returns'].std()\n",
    "\n",
    "    # Print Sharpe ratios\n",
    "    print(f\"{model_name} Sharpe Ratio:\", model_sharpe_ratio)\n",
    "\n",
    "    # Calculate the cumulative returns\n",
    "    df_merged['Cumulative Weighted Returns'] = (1 + df_merged['Weighted Returns']).cumprod()\n",
    "    df_merged['Cumulative Simple Weighted Returns'] = (1 + df_merged['Simple Weighted Returns']).cumprod()\n",
    "\n",
    "    # Plot the cumulative returns\n",
    "    plt.plot(df_merged['Cumulative Weighted Returns'], label=model_name)\n",
    "\n",
    "# Plot the 50/50 benchmark\n",
    "plt.plot(df_merged['Cumulative Simple Weighted Returns'], label='50/50 Benchmark', color='black')\n",
    "\n",
    "# Add a horizontal line at zero\n",
    "plt.axhline(1, color='black', linestyle='--')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('returns')\n",
    "plt.title('Model Comparison (Test)')\n",
    "plt.yscale('log')  # Set y-axis scale to logarithmic\n",
    "plt.show()\n",
    "\n",
    "benchmark_sharpe_ratio = df_merged['Simple Weighted Returns'].mean() / df_merged['Simple Weighted Returns'].std()\n",
    "print(\"Benchmark Sharpe Ratio:\", benchmark_sharpe_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32c2cbb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(11, 12), dpi=100)\n",
    "\n",
    "# Flatten the axes array for easy iteration\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Repeat the process for each model\n",
    "for df_prob, model_name, ax in zip([df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob], \n",
    "                                   ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron'], \n",
    "                                   axes):\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_6.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "\n",
    "    # Calculate the simple 50/50 weighted return stream\n",
    "    df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                           0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "    # Shift the simple weighted returns by one row to match with the models\n",
    "    df_merged['Simple Weighted Returns'] = df_merged['Simple Weighted Returns']\n",
    "\n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Plot the densities of monthly returns for the model and the benchmark\n",
    "    ax.hist(df_merged['Weighted Returns'], bins=100, alpha=0.5, label=model_name, color='red')\n",
    "    ax.hist(df_merged['Simple Weighted Returns'], bins=100, alpha=0.5, label='50/50 Benchmark', color='black')\n",
    "\n",
    "    # Add a vertical line at zero\n",
    "    ax.axvline(0, color='red', linestyle='--')\n",
    "    \n",
    "    ax.legend()\n",
    "    ax.set_xlabel('monthly returns')\n",
    "    ax.set_ylabel('frequency')\n",
    "    ax.set_title(f'Return Histogram of {model_name} (Test)')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "938dfd69",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare lists to store model returns and model names\n",
    "model_returns = []\n",
    "model_names = ['Logistic Regression', 'Support Vector Machine', 'Random Forest', 'Multi-layer Perceptron']\n",
    "\n",
    "# Get returns for each model\n",
    "for df_prob in [df_log_reg_prob, df_svm_prob, df_rf_prob, df_mlp_prob]:\n",
    "    # Merge the DataFrames based on their indexes\n",
    "    df_merged = df_test_6.merge(df_prob, left_index=True, right_index=True)\n",
    "\n",
    "    # Shift the probabilities by one row to use the probabilities from the previous time step\n",
    "    shifted_probabilities = df_merged[['Probability_0', 'Probability_1']].shift()\n",
    "\n",
    "    # Calculate the weighted returns using the shifted probabilities\n",
    "    df_merged['Weighted Returns'] = shifted_probabilities['Probability_0'] * df_merged['Value Returns'] + \\\n",
    "                                    shifted_probabilities['Probability_1'] * df_merged['Momentum Returns']\n",
    "    \n",
    "    # Drop rows with NaN values\n",
    "    df_merged = df_merged.dropna()\n",
    "\n",
    "    # Append model returns to the list\n",
    "    model_returns.append(df_merged['Weighted Returns'])\n",
    "\n",
    "\n",
    "# Calculate the simple 50/50 weighted return stream\n",
    "df_merged['Simple Weighted Returns'] = 0.5 * df_merged['Value Returns'] + \\\n",
    "                                       0.5 * df_merged['Momentum Returns']\n",
    "\n",
    "benchmark_returns = df_merged['Simple Weighted Returns']\n",
    "    \n",
    "# Run bootstrap test\n",
    "multi_sharpe_ratio_bootstrap_test(benchmark_returns, model_returns, model_names)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}