PreProcessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will try to find good techniques to apply to our original data to improve the precision of our algorithms and to reduce the size of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "from src.regressions import *\n",
    "from sklearn.decomposition import PCA\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FOLDER = \"data/DA/\"\n",
    "X = np.load(DATA_FOLDER + \"feature_mat_radial_compression.npy\")\n",
    "y = np.load(DATA_FOLDER + \"CSD500-r_train-H_total.npy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X: (30049, 15961)\n",
      "y: (30049,)\n"
     ]
    }
   ],
   "source": [
    "print(\"X: \" + str(X.shape))\n",
    "print(\"y: \" + str(y.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_df = pd.DataFrame(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X: (30049, 15961)\n"
     ]
    }
   ],
   "source": [
    "x_df = x_df.drop_duplicates()\n",
    "print(\"X: \" + str(x_df.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The split function is used to split the data set into train and test sets with a 75/25 ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split(x_df, y, perc):\n",
    "    train_set_size = int(x_df.shape[0] * 0.75)\n",
    "    x_tr = x_df.head(train_set_size)\n",
    "    x_te = x_df.tail(int(x_df.shape[0] - train_set_size))\n",
    "    y_tr = y[: train_set_size]\n",
    "    y_te = y[train_set_size :]\n",
    "    return x_tr, y_tr, x_te, y_te"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that a polynomial expansion really improves the results. This expansion is equivalent to add a column of 1s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_cte_col(df):\n",
    "    df_tmp = df.copy()\n",
    "    df_tmp[df_tmp.shape[1]] = pd.Series(np.ones(df_tmp.shape[0]), index=df_tmp.index)\n",
    "    return df_tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can create our test function that splits the data, adds a column of constant term and performs ridge regression for a few lambdas. Only the lowest error is kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_quality(x_df, y):\n",
    "    x_df = add_cte_col(x_df)\n",
    "    test_perc = 0.75\n",
    "    x_train, y_train, x_test, y_test = split(x_df, y, test_perc)\n",
    "    best = 100\n",
    "    for lambda_ in [1e-2, 1e-3, 1e-4, 1e-5, 1e-6]:\n",
    "        err = rmse(y_test, x_test, ridge_regression(y_train, x_train, lambda_))\n",
    "        best = err if err < best else best\n",
    "    return best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7373143070214498"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_quality(x_df, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that a simple ridge regression with polynomial expansion gives us an RMSE of 0.73. This will be the value that we consider as the reference for the upcoming tests on the techniques we try."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We encountered in our readings that deep learning can get only better with more data. So we try to artificially generate some samples based on what we have. We use jittering whhich consists in adding a little bit of noise to an existing, small enough to consider that our evaluation y will stay the same. Basically, for a given number of x_i, we create new samples by replacing x_ij by x_ij + 1% of the mean of the j th feature of the samples. We made it such that we can choose how many samples should we add and how many features should we modify per sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#to have more samples\n",
    "def add_jitter(df, y, perc_sample, perc_col):\n",
    "    means = [df[j].mean() for j in df]\n",
    "    df_tmp = df.copy()\n",
    "    y_tmp = y.copy()\n",
    "    ids = random.sample(range(df_tmp.shape[0]), int(df_tmp.shape[0]*perc_sample))\n",
    "    for id_ in ids:\n",
    "        new_sample = df_tmp.iloc[[id_]].copy()\n",
    "        col = random.sample(range(df_tmp.shape[1]), int(df_tmp.shape[1]*perc_col))\n",
    "        for j in col:\n",
    "            new_sample[j] = new_sample[j] + 0.01*means[j]\n",
    "        df_tmp = df_tmp.append(new_sample, ignore_index=True)\n",
    "        y_tmp = np.append(y_tmp, y_tmp[id_])\n",
    "    return df_tmp, y_tmp        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_df_aug, y_aug = add_jitter(x_df, y, 0.01, 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7239879009\n"
     ]
    }
   ],
   "source": [
    "test_quality(x_df_aug, y_aug)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that adding 1% of samples and modifying 1% of the features lowers the RMSE to 0.72. This is some improvement !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>15950</th>\n",
       "      <th>15951</th>\n",
       "      <th>15952</th>\n",
       "      <th>15953</th>\n",
       "      <th>15954</th>\n",
       "      <th>15955</th>\n",
       "      <th>15956</th>\n",
       "      <th>15957</th>\n",
       "      <th>15958</th>\n",
       "      <th>15959</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.117537</td>\n",
       "      <td>0.130346</td>\n",
       "      <td>0.013696</td>\n",
       "      <td>-0.103711</td>\n",
       "      <td>0.037901</td>\n",
       "      <td>0.269922</td>\n",
       "      <td>0.399752</td>\n",
       "      <td>0.415404</td>\n",
       "      <td>0.358259</td>\n",
       "      <td>0.303114</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.393410</td>\n",
       "      <td>-0.301521</td>\n",
       "      <td>-0.367537</td>\n",
       "      <td>-0.394536</td>\n",
       "      <td>-0.370814</td>\n",
       "      <td>-0.407677</td>\n",
       "      <td>-0.416760</td>\n",
       "      <td>-0.401391</td>\n",
       "      <td>-0.409848</td>\n",
       "      <td>-0.426182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.141154</td>\n",
       "      <td>0.360481</td>\n",
       "      <td>0.266127</td>\n",
       "      <td>0.079885</td>\n",
       "      <td>0.149372</td>\n",
       "      <td>0.394671</td>\n",
       "      <td>0.586697</td>\n",
       "      <td>0.697098</td>\n",
       "      <td>0.761406</td>\n",
       "      <td>0.658471</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.439485</td>\n",
       "      <td>-0.473423</td>\n",
       "      <td>-0.539600</td>\n",
       "      <td>-0.589093</td>\n",
       "      <td>-0.570107</td>\n",
       "      <td>-0.601153</td>\n",
       "      <td>-0.609641</td>\n",
       "      <td>-0.615344</td>\n",
       "      <td>-0.617178</td>\n",
       "      <td>-0.622111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.177216</td>\n",
       "      <td>-0.267615</td>\n",
       "      <td>-0.349863</td>\n",
       "      <td>-0.416966</td>\n",
       "      <td>-0.393240</td>\n",
       "      <td>-0.265429</td>\n",
       "      <td>-0.145420</td>\n",
       "      <td>-0.109582</td>\n",
       "      <td>-0.151950</td>\n",
       "      <td>-0.193078</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.430591</td>\n",
       "      <td>-0.422903</td>\n",
       "      <td>-0.480570</td>\n",
       "      <td>-0.536433</td>\n",
       "      <td>-0.519772</td>\n",
       "      <td>-0.540244</td>\n",
       "      <td>-0.550040</td>\n",
       "      <td>-0.558432</td>\n",
       "      <td>-0.558878</td>\n",
       "      <td>-0.562048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.125873</td>\n",
       "      <td>0.708498</td>\n",
       "      <td>0.511915</td>\n",
       "      <td>0.345341</td>\n",
       "      <td>0.678018</td>\n",
       "      <td>0.960304</td>\n",
       "      <td>0.994918</td>\n",
       "      <td>0.879390</td>\n",
       "      <td>0.731289</td>\n",
       "      <td>0.780614</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.458453</td>\n",
       "      <td>-0.547680</td>\n",
       "      <td>-0.616144</td>\n",
       "      <td>-0.673036</td>\n",
       "      <td>-0.654511</td>\n",
       "      <td>-0.685598</td>\n",
       "      <td>-0.695242</td>\n",
       "      <td>-0.707149</td>\n",
       "      <td>-0.706268</td>\n",
       "      <td>-0.709202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.301681</td>\n",
       "      <td>0.035031</td>\n",
       "      <td>-0.075285</td>\n",
       "      <td>-0.190359</td>\n",
       "      <td>-0.102609</td>\n",
       "      <td>0.108780</td>\n",
       "      <td>0.216456</td>\n",
       "      <td>0.411433</td>\n",
       "      <td>0.163525</td>\n",
       "      <td>0.113291</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.494304</td>\n",
       "      <td>-0.691341</td>\n",
       "      <td>-0.765498</td>\n",
       "      <td>-0.832934</td>\n",
       "      <td>-0.815301</td>\n",
       "      <td>-0.849757</td>\n",
       "      <td>-0.864321</td>\n",
       "      <td>-0.889805</td>\n",
       "      <td>-0.880755</td>\n",
       "      <td>-0.883626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-1.289225</td>\n",
       "      <td>0.618264</td>\n",
       "      <td>0.440396</td>\n",
       "      <td>0.333169</td>\n",
       "      <td>0.531440</td>\n",
       "      <td>0.795734</td>\n",
       "      <td>0.815010</td>\n",
       "      <td>0.934647</td>\n",
       "      <td>0.583534</td>\n",
       "      <td>0.673116</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.493536</td>\n",
       "      <td>-0.691717</td>\n",
       "      <td>-0.767147</td>\n",
       "      <td>-0.832168</td>\n",
       "      <td>-0.814264</td>\n",
       "      <td>-0.850489</td>\n",
       "      <td>-0.864632</td>\n",
       "      <td>-0.889108</td>\n",
       "      <td>-0.880674</td>\n",
       "      <td>-0.884081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-1.280896</td>\n",
       "      <td>-0.263859</td>\n",
       "      <td>-0.355730</td>\n",
       "      <td>-0.418794</td>\n",
       "      <td>-0.397849</td>\n",
       "      <td>-0.229321</td>\n",
       "      <td>-0.186046</td>\n",
       "      <td>-0.037326</td>\n",
       "      <td>-0.213264</td>\n",
       "      <td>-0.233464</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.490703</td>\n",
       "      <td>-0.694423</td>\n",
       "      <td>-0.769091</td>\n",
       "      <td>-0.828314</td>\n",
       "      <td>-0.812143</td>\n",
       "      <td>-0.851598</td>\n",
       "      <td>-0.864812</td>\n",
       "      <td>-0.884870</td>\n",
       "      <td>-0.880759</td>\n",
       "      <td>-0.885678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-1.244046</td>\n",
       "      <td>0.480159</td>\n",
       "      <td>0.363812</td>\n",
       "      <td>0.202020</td>\n",
       "      <td>0.349861</td>\n",
       "      <td>0.564196</td>\n",
       "      <td>0.670118</td>\n",
       "      <td>0.849204</td>\n",
       "      <td>0.574173</td>\n",
       "      <td>0.503616</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.484862</td>\n",
       "      <td>-0.684433</td>\n",
       "      <td>-0.766651</td>\n",
       "      <td>-0.814271</td>\n",
       "      <td>-0.794596</td>\n",
       "      <td>-0.843836</td>\n",
       "      <td>-0.857055</td>\n",
       "      <td>-0.869302</td>\n",
       "      <td>-0.864939</td>\n",
       "      <td>-0.877402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.121709</td>\n",
       "      <td>0.117122</td>\n",
       "      <td>0.008087</td>\n",
       "      <td>-0.111065</td>\n",
       "      <td>0.043313</td>\n",
       "      <td>0.242077</td>\n",
       "      <td>0.372759</td>\n",
       "      <td>0.408476</td>\n",
       "      <td>0.373977</td>\n",
       "      <td>0.232905</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.388380</td>\n",
       "      <td>-0.487897</td>\n",
       "      <td>-0.678482</td>\n",
       "      <td>-0.671614</td>\n",
       "      <td>-0.536393</td>\n",
       "      <td>-0.580206</td>\n",
       "      <td>-0.699196</td>\n",
       "      <td>-0.727652</td>\n",
       "      <td>-0.622976</td>\n",
       "      <td>-0.591967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-1.120282</td>\n",
       "      <td>0.673164</td>\n",
       "      <td>0.520629</td>\n",
       "      <td>0.353339</td>\n",
       "      <td>0.604866</td>\n",
       "      <td>0.836481</td>\n",
       "      <td>0.911321</td>\n",
       "      <td>0.919284</td>\n",
       "      <td>0.833911</td>\n",
       "      <td>0.680132</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.430117</td>\n",
       "      <td>-0.472262</td>\n",
       "      <td>-0.542216</td>\n",
       "      <td>-0.584586</td>\n",
       "      <td>-0.566216</td>\n",
       "      <td>-0.587505</td>\n",
       "      <td>-0.606463</td>\n",
       "      <td>-0.627249</td>\n",
       "      <td>-0.609860</td>\n",
       "      <td>-0.606361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-1.161999</td>\n",
       "      <td>-0.170808</td>\n",
       "      <td>-0.280091</td>\n",
       "      <td>-0.345389</td>\n",
       "      <td>-0.215225</td>\n",
       "      <td>-0.066473</td>\n",
       "      <td>0.016016</td>\n",
       "      <td>-0.005436</td>\n",
       "      <td>-0.030528</td>\n",
       "      <td>-0.125808</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.381401</td>\n",
       "      <td>-0.405521</td>\n",
       "      <td>-0.553988</td>\n",
       "      <td>-0.555117</td>\n",
       "      <td>-0.461654</td>\n",
       "      <td>-0.492020</td>\n",
       "      <td>-0.577277</td>\n",
       "      <td>-0.603402</td>\n",
       "      <td>-0.526465</td>\n",
       "      <td>-0.502749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-1.109925</td>\n",
       "      <td>0.348278</td>\n",
       "      <td>0.253625</td>\n",
       "      <td>0.084104</td>\n",
       "      <td>0.181172</td>\n",
       "      <td>0.375732</td>\n",
       "      <td>0.543186</td>\n",
       "      <td>0.703394</td>\n",
       "      <td>0.773510</td>\n",
       "      <td>0.455338</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.409108</td>\n",
       "      <td>-0.389607</td>\n",
       "      <td>-0.450645</td>\n",
       "      <td>-0.486197</td>\n",
       "      <td>-0.474785</td>\n",
       "      <td>-0.491295</td>\n",
       "      <td>-0.505612</td>\n",
       "      <td>-0.522966</td>\n",
       "      <td>-0.508114</td>\n",
       "      <td>-0.506073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.583999</td>\n",
       "      <td>-0.101838</td>\n",
       "      <td>-0.187720</td>\n",
       "      <td>-0.265902</td>\n",
       "      <td>-0.204427</td>\n",
       "      <td>-0.061798</td>\n",
       "      <td>0.052346</td>\n",
       "      <td>0.112209</td>\n",
       "      <td>0.046005</td>\n",
       "      <td>-0.054308</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.490387</td>\n",
       "      <td>-0.693437</td>\n",
       "      <td>-0.770339</td>\n",
       "      <td>-0.829602</td>\n",
       "      <td>-0.813115</td>\n",
       "      <td>-0.852158</td>\n",
       "      <td>-0.862470</td>\n",
       "      <td>-0.881090</td>\n",
       "      <td>-0.876895</td>\n",
       "      <td>-0.881600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>-1.532105</td>\n",
       "      <td>1.795420</td>\n",
       "      <td>1.494828</td>\n",
       "      <td>1.117432</td>\n",
       "      <td>1.433854</td>\n",
       "      <td>1.637722</td>\n",
       "      <td>1.598290</td>\n",
       "      <td>1.470541</td>\n",
       "      <td>1.201826</td>\n",
       "      <td>1.022441</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.458935</td>\n",
       "      <td>-0.596976</td>\n",
       "      <td>-0.675757</td>\n",
       "      <td>-0.711822</td>\n",
       "      <td>-0.694355</td>\n",
       "      <td>-0.740111</td>\n",
       "      <td>-0.751211</td>\n",
       "      <td>-0.754784</td>\n",
       "      <td>-0.750471</td>\n",
       "      <td>-0.766673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-1.556035</td>\n",
       "      <td>0.854810</td>\n",
       "      <td>0.650248</td>\n",
       "      <td>0.425941</td>\n",
       "      <td>0.679988</td>\n",
       "      <td>0.941011</td>\n",
       "      <td>1.006641</td>\n",
       "      <td>0.975626</td>\n",
       "      <td>0.819083</td>\n",
       "      <td>0.644963</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.478982</td>\n",
       "      <td>-0.658896</td>\n",
       "      <td>-0.736396</td>\n",
       "      <td>-0.786870</td>\n",
       "      <td>-0.770309</td>\n",
       "      <td>-0.812058</td>\n",
       "      <td>-0.822155</td>\n",
       "      <td>-0.835113</td>\n",
       "      <td>-0.831429</td>\n",
       "      <td>-0.840272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-1.509894</td>\n",
       "      <td>0.789489</td>\n",
       "      <td>0.734525</td>\n",
       "      <td>0.508362</td>\n",
       "      <td>0.519640</td>\n",
       "      <td>0.669836</td>\n",
       "      <td>0.828015</td>\n",
       "      <td>0.975663</td>\n",
       "      <td>0.960331</td>\n",
       "      <td>0.713935</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.453918</td>\n",
       "      <td>-0.576785</td>\n",
       "      <td>-0.653024</td>\n",
       "      <td>-0.687564</td>\n",
       "      <td>-0.671862</td>\n",
       "      <td>-0.716031</td>\n",
       "      <td>-0.725677</td>\n",
       "      <td>-0.729007</td>\n",
       "      <td>-0.725913</td>\n",
       "      <td>-0.741710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>-1.468620</td>\n",
       "      <td>-0.150050</td>\n",
       "      <td>-0.231988</td>\n",
       "      <td>-0.305689</td>\n",
       "      <td>-0.293819</td>\n",
       "      <td>-0.157557</td>\n",
       "      <td>-0.017568</td>\n",
       "      <td>0.111731</td>\n",
       "      <td>0.032982</td>\n",
       "      <td>-0.195766</td>\n",
       "      <td>...</td>\n",
       "      <td>0.107181</td>\n",
       "      <td>0.317355</td>\n",
       "      <td>0.280894</td>\n",
       "      <td>-0.044644</td>\n",
       "      <td>-0.254345</td>\n",
       "      <td>-0.306427</td>\n",
       "      <td>-0.400406</td>\n",
       "      <td>0.201104</td>\n",
       "      <td>0.281676</td>\n",
       "      <td>0.180174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-1.348397</td>\n",
       "      <td>1.315641</td>\n",
       "      <td>1.142931</td>\n",
       "      <td>0.767735</td>\n",
       "      <td>0.831600</td>\n",
       "      <td>1.091384</td>\n",
       "      <td>1.288360</td>\n",
       "      <td>1.449918</td>\n",
       "      <td>1.262782</td>\n",
       "      <td>1.106336</td>\n",
       "      <td>...</td>\n",
       "      <td>0.226749</td>\n",
       "      <td>0.966048</td>\n",
       "      <td>0.364869</td>\n",
       "      <td>-0.167786</td>\n",
       "      <td>-0.200911</td>\n",
       "      <td>-0.319185</td>\n",
       "      <td>-0.324164</td>\n",
       "      <td>0.068363</td>\n",
       "      <td>0.586419</td>\n",
       "      <td>0.695491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.424677</td>\n",
       "      <td>0.814512</td>\n",
       "      <td>0.635795</td>\n",
       "      <td>0.410424</td>\n",
       "      <td>0.590902</td>\n",
       "      <td>0.841481</td>\n",
       "      <td>0.965004</td>\n",
       "      <td>1.077745</td>\n",
       "      <td>0.814894</td>\n",
       "      <td>0.579525</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.000878</td>\n",
       "      <td>0.100409</td>\n",
       "      <td>0.058536</td>\n",
       "      <td>-0.368178</td>\n",
       "      <td>-0.425352</td>\n",
       "      <td>-0.474284</td>\n",
       "      <td>-0.557892</td>\n",
       "      <td>-0.182215</td>\n",
       "      <td>0.025371</td>\n",
       "      <td>-0.024643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.448649</td>\n",
       "      <td>0.953664</td>\n",
       "      <td>0.895262</td>\n",
       "      <td>0.685877</td>\n",
       "      <td>0.683935</td>\n",
       "      <td>0.826307</td>\n",
       "      <td>0.964385</td>\n",
       "      <td>1.128858</td>\n",
       "      <td>0.946639</td>\n",
       "      <td>0.595656</td>\n",
       "      <td>...</td>\n",
       "      <td>0.110359</td>\n",
       "      <td>0.549694</td>\n",
       "      <td>0.188414</td>\n",
       "      <td>-0.263452</td>\n",
       "      <td>-0.286069</td>\n",
       "      <td>-0.386958</td>\n",
       "      <td>-0.403285</td>\n",
       "      <td>0.008902</td>\n",
       "      <td>0.312088</td>\n",
       "      <td>0.348862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>-1.278326</td>\n",
       "      <td>0.070219</td>\n",
       "      <td>-0.079248</td>\n",
       "      <td>-0.149880</td>\n",
       "      <td>0.035801</td>\n",
       "      <td>0.252874</td>\n",
       "      <td>0.299863</td>\n",
       "      <td>0.363914</td>\n",
       "      <td>0.041698</td>\n",
       "      <td>0.194562</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.481081</td>\n",
       "      <td>-0.384489</td>\n",
       "      <td>-0.305489</td>\n",
       "      <td>-0.486858</td>\n",
       "      <td>-0.565668</td>\n",
       "      <td>-0.485949</td>\n",
       "      <td>-0.434295</td>\n",
       "      <td>-0.501755</td>\n",
       "      <td>-0.554358</td>\n",
       "      <td>-0.533938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>-1.252883</td>\n",
       "      <td>1.984057</td>\n",
       "      <td>1.699722</td>\n",
       "      <td>1.269265</td>\n",
       "      <td>1.680939</td>\n",
       "      <td>1.881720</td>\n",
       "      <td>1.838494</td>\n",
       "      <td>1.685328</td>\n",
       "      <td>1.238330</td>\n",
       "      <td>1.162364</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.481664</td>\n",
       "      <td>-0.278626</td>\n",
       "      <td>-0.206689</td>\n",
       "      <td>-0.456319</td>\n",
       "      <td>-0.476558</td>\n",
       "      <td>-0.353213</td>\n",
       "      <td>-0.384215</td>\n",
       "      <td>-0.448523</td>\n",
       "      <td>-0.418716</td>\n",
       "      <td>-0.439497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>-1.262766</td>\n",
       "      <td>0.841785</td>\n",
       "      <td>0.652353</td>\n",
       "      <td>0.419199</td>\n",
       "      <td>0.595580</td>\n",
       "      <td>0.874963</td>\n",
       "      <td>0.988802</td>\n",
       "      <td>1.150711</td>\n",
       "      <td>0.816094</td>\n",
       "      <td>0.826144</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.490537</td>\n",
       "      <td>-0.272377</td>\n",
       "      <td>-0.196730</td>\n",
       "      <td>-0.474109</td>\n",
       "      <td>-0.486586</td>\n",
       "      <td>-0.343549</td>\n",
       "      <td>-0.385533</td>\n",
       "      <td>-0.464477</td>\n",
       "      <td>-0.425519</td>\n",
       "      <td>-0.435991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>-1.218103</td>\n",
       "      <td>1.756039</td>\n",
       "      <td>1.288217</td>\n",
       "      <td>1.090831</td>\n",
       "      <td>1.765646</td>\n",
       "      <td>2.002212</td>\n",
       "      <td>1.701695</td>\n",
       "      <td>1.420702</td>\n",
       "      <td>0.913363</td>\n",
       "      <td>1.152832</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.504447</td>\n",
       "      <td>-0.481353</td>\n",
       "      <td>-0.451276</td>\n",
       "      <td>-0.645964</td>\n",
       "      <td>-0.666999</td>\n",
       "      <td>-0.599047</td>\n",
       "      <td>-0.605200</td>\n",
       "      <td>-0.662121</td>\n",
       "      <td>-0.662725</td>\n",
       "      <td>-0.666251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>-1.466737</td>\n",
       "      <td>0.364266</td>\n",
       "      <td>0.242847</td>\n",
       "      <td>0.073562</td>\n",
       "      <td>0.230399</td>\n",
       "      <td>0.449496</td>\n",
       "      <td>0.555549</td>\n",
       "      <td>0.666431</td>\n",
       "      <td>0.578242</td>\n",
       "      <td>0.337261</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.507080</td>\n",
       "      <td>-0.664343</td>\n",
       "      <td>-0.723486</td>\n",
       "      <td>-0.839001</td>\n",
       "      <td>-0.819098</td>\n",
       "      <td>-0.836303</td>\n",
       "      <td>-0.846125</td>\n",
       "      <td>-0.863365</td>\n",
       "      <td>-0.877635</td>\n",
       "      <td>-0.875626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>-1.352696</td>\n",
       "      <td>1.183222</td>\n",
       "      <td>1.100144</td>\n",
       "      <td>1.028277</td>\n",
       "      <td>1.242592</td>\n",
       "      <td>1.268293</td>\n",
       "      <td>1.135938</td>\n",
       "      <td>1.037624</td>\n",
       "      <td>0.812252</td>\n",
       "      <td>0.582882</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.505223</td>\n",
       "      <td>-0.605229</td>\n",
       "      <td>-0.602400</td>\n",
       "      <td>-0.741086</td>\n",
       "      <td>-0.779414</td>\n",
       "      <td>-0.765720</td>\n",
       "      <td>-0.728375</td>\n",
       "      <td>-0.762693</td>\n",
       "      <td>-0.814653</td>\n",
       "      <td>-0.799628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>-1.468931</td>\n",
       "      <td>1.565876</td>\n",
       "      <td>1.444560</td>\n",
       "      <td>1.036704</td>\n",
       "      <td>1.104169</td>\n",
       "      <td>1.244886</td>\n",
       "      <td>1.364998</td>\n",
       "      <td>1.461079</td>\n",
       "      <td>1.428367</td>\n",
       "      <td>0.861658</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.506524</td>\n",
       "      <td>-0.666557</td>\n",
       "      <td>-0.697989</td>\n",
       "      <td>-0.811486</td>\n",
       "      <td>-0.818870</td>\n",
       "      <td>-0.820644</td>\n",
       "      <td>-0.813846</td>\n",
       "      <td>-0.853022</td>\n",
       "      <td>-0.876866</td>\n",
       "      <td>-0.854819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>-1.458226</td>\n",
       "      <td>0.916002</td>\n",
       "      <td>0.717254</td>\n",
       "      <td>0.499095</td>\n",
       "      <td>0.826848</td>\n",
       "      <td>1.040904</td>\n",
       "      <td>1.066190</td>\n",
       "      <td>1.015816</td>\n",
       "      <td>0.818418</td>\n",
       "      <td>0.581877</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.507031</td>\n",
       "      <td>-0.663400</td>\n",
       "      <td>-0.696159</td>\n",
       "      <td>-0.808843</td>\n",
       "      <td>-0.817185</td>\n",
       "      <td>-0.820607</td>\n",
       "      <td>-0.811046</td>\n",
       "      <td>-0.848498</td>\n",
       "      <td>-0.872878</td>\n",
       "      <td>-0.856313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>-1.244967</td>\n",
       "      <td>0.528076</td>\n",
       "      <td>0.344597</td>\n",
       "      <td>0.199056</td>\n",
       "      <td>0.392641</td>\n",
       "      <td>0.695089</td>\n",
       "      <td>0.841445</td>\n",
       "      <td>0.788050</td>\n",
       "      <td>0.620654</td>\n",
       "      <td>0.633079</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.500227</td>\n",
       "      <td>-0.435424</td>\n",
       "      <td>-0.764507</td>\n",
       "      <td>-0.574717</td>\n",
       "      <td>-0.790852</td>\n",
       "      <td>-0.635907</td>\n",
       "      <td>-0.775147</td>\n",
       "      <td>-0.717072</td>\n",
       "      <td>-0.744430</td>\n",
       "      <td>-0.754885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>-1.293424</td>\n",
       "      <td>0.144455</td>\n",
       "      <td>0.085388</td>\n",
       "      <td>-0.061860</td>\n",
       "      <td>-0.129136</td>\n",
       "      <td>0.019201</td>\n",
       "      <td>0.244424</td>\n",
       "      <td>0.500393</td>\n",
       "      <td>0.708577</td>\n",
       "      <td>0.530286</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.500603</td>\n",
       "      <td>-0.392792</td>\n",
       "      <td>-0.770519</td>\n",
       "      <td>-0.548080</td>\n",
       "      <td>-0.806400</td>\n",
       "      <td>-0.604118</td>\n",
       "      <td>-0.797249</td>\n",
       "      <td>-0.664572</td>\n",
       "      <td>-0.764990</td>\n",
       "      <td>-0.706043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30019</th>\n",
       "      <td>-0.370322</td>\n",
       "      <td>-0.226526</td>\n",
       "      <td>-0.080004</td>\n",
       "      <td>0.137031</td>\n",
       "      <td>0.086162</td>\n",
       "      <td>-0.119531</td>\n",
       "      <td>-0.156182</td>\n",
       "      <td>-0.074760</td>\n",
       "      <td>0.064308</td>\n",
       "      <td>0.348836</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.485064</td>\n",
       "      <td>-0.706802</td>\n",
       "      <td>-0.775669</td>\n",
       "      <td>-0.843690</td>\n",
       "      <td>-0.842398</td>\n",
       "      <td>-0.833092</td>\n",
       "      <td>-0.876333</td>\n",
       "      <td>-0.904293</td>\n",
       "      <td>-0.890757</td>\n",
       "      <td>-0.892930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30020</th>\n",
       "      <td>-0.450635</td>\n",
       "      <td>-0.273962</td>\n",
       "      <td>-0.139269</td>\n",
       "      <td>0.032197</td>\n",
       "      <td>-0.022757</td>\n",
       "      <td>-0.183572</td>\n",
       "      <td>-0.200784</td>\n",
       "      <td>-0.168052</td>\n",
       "      <td>-0.012166</td>\n",
       "      <td>0.224280</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.500063</td>\n",
       "      <td>-0.705547</td>\n",
       "      <td>-0.792829</td>\n",
       "      <td>-0.869371</td>\n",
       "      <td>-0.855394</td>\n",
       "      <td>-0.865562</td>\n",
       "      <td>-0.892436</td>\n",
       "      <td>-0.922987</td>\n",
       "      <td>-0.913269</td>\n",
       "      <td>-0.915168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30021</th>\n",
       "      <td>-0.187353</td>\n",
       "      <td>-0.270201</td>\n",
       "      <td>0.036723</td>\n",
       "      <td>0.143085</td>\n",
       "      <td>0.077921</td>\n",
       "      <td>-0.010722</td>\n",
       "      <td>-0.187752</td>\n",
       "      <td>0.038560</td>\n",
       "      <td>0.140613</td>\n",
       "      <td>0.255097</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.426354</td>\n",
       "      <td>-0.136415</td>\n",
       "      <td>-0.323139</td>\n",
       "      <td>-0.566377</td>\n",
       "      <td>-0.335159</td>\n",
       "      <td>-0.632194</td>\n",
       "      <td>-0.419565</td>\n",
       "      <td>-0.375516</td>\n",
       "      <td>-0.523034</td>\n",
       "      <td>-0.301023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30022</th>\n",
       "      <td>-0.131709</td>\n",
       "      <td>-0.348489</td>\n",
       "      <td>0.022155</td>\n",
       "      <td>0.146214</td>\n",
       "      <td>-0.060469</td>\n",
       "      <td>0.040867</td>\n",
       "      <td>-0.220123</td>\n",
       "      <td>0.010850</td>\n",
       "      <td>0.237173</td>\n",
       "      <td>0.125278</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.347662</td>\n",
       "      <td>0.414243</td>\n",
       "      <td>-0.150452</td>\n",
       "      <td>-0.455553</td>\n",
       "      <td>-0.210063</td>\n",
       "      <td>-0.671269</td>\n",
       "      <td>-0.196376</td>\n",
       "      <td>-0.079386</td>\n",
       "      <td>-0.263368</td>\n",
       "      <td>0.075856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30023</th>\n",
       "      <td>-0.257295</td>\n",
       "      <td>-0.020362</td>\n",
       "      <td>0.223802</td>\n",
       "      <td>0.497554</td>\n",
       "      <td>0.422577</td>\n",
       "      <td>0.160234</td>\n",
       "      <td>0.069173</td>\n",
       "      <td>0.154584</td>\n",
       "      <td>0.333823</td>\n",
       "      <td>0.488834</td>\n",
       "      <td>...</td>\n",
       "      <td>0.870766</td>\n",
       "      <td>1.392696</td>\n",
       "      <td>0.523158</td>\n",
       "      <td>2.352454</td>\n",
       "      <td>0.231209</td>\n",
       "      <td>0.724496</td>\n",
       "      <td>1.456699</td>\n",
       "      <td>0.416698</td>\n",
       "      <td>1.021780</td>\n",
       "      <td>1.426201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30024</th>\n",
       "      <td>-0.235252</td>\n",
       "      <td>-0.348016</td>\n",
       "      <td>-0.273893</td>\n",
       "      <td>-0.148297</td>\n",
       "      <td>-0.189325</td>\n",
       "      <td>-0.300012</td>\n",
       "      <td>-0.311168</td>\n",
       "      <td>-0.208134</td>\n",
       "      <td>-0.122402</td>\n",
       "      <td>-0.099617</td>\n",
       "      <td>...</td>\n",
       "      <td>1.505774</td>\n",
       "      <td>2.067066</td>\n",
       "      <td>1.219951</td>\n",
       "      <td>3.248626</td>\n",
       "      <td>0.528642</td>\n",
       "      <td>1.468564</td>\n",
       "      <td>2.180165</td>\n",
       "      <td>0.794355</td>\n",
       "      <td>1.481247</td>\n",
       "      <td>2.325696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30025</th>\n",
       "      <td>-0.640975</td>\n",
       "      <td>-0.069737</td>\n",
       "      <td>0.169179</td>\n",
       "      <td>0.413493</td>\n",
       "      <td>0.326494</td>\n",
       "      <td>0.088665</td>\n",
       "      <td>0.006096</td>\n",
       "      <td>0.033707</td>\n",
       "      <td>0.180667</td>\n",
       "      <td>0.612531</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.347120</td>\n",
       "      <td>-0.463630</td>\n",
       "      <td>-0.278552</td>\n",
       "      <td>-0.568210</td>\n",
       "      <td>-0.260883</td>\n",
       "      <td>-0.317195</td>\n",
       "      <td>-0.248857</td>\n",
       "      <td>-0.135519</td>\n",
       "      <td>-0.345025</td>\n",
       "      <td>-0.143403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30026</th>\n",
       "      <td>-0.663981</td>\n",
       "      <td>-0.372109</td>\n",
       "      <td>-0.311566</td>\n",
       "      <td>-0.198239</td>\n",
       "      <td>-0.240748</td>\n",
       "      <td>-0.326981</td>\n",
       "      <td>-0.340134</td>\n",
       "      <td>-0.322195</td>\n",
       "      <td>-0.273446</td>\n",
       "      <td>0.002196</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.402475</td>\n",
       "      <td>-0.618906</td>\n",
       "      <td>-0.528591</td>\n",
       "      <td>-0.747857</td>\n",
       "      <td>-0.519908</td>\n",
       "      <td>-0.555486</td>\n",
       "      <td>-0.473275</td>\n",
       "      <td>-0.472852</td>\n",
       "      <td>-0.569673</td>\n",
       "      <td>-0.447912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30027</th>\n",
       "      <td>-0.230168</td>\n",
       "      <td>-0.593888</td>\n",
       "      <td>-0.690791</td>\n",
       "      <td>-0.726956</td>\n",
       "      <td>-0.790438</td>\n",
       "      <td>-0.786404</td>\n",
       "      <td>-0.779407</td>\n",
       "      <td>-0.828310</td>\n",
       "      <td>-0.906732</td>\n",
       "      <td>-0.953445</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.086721</td>\n",
       "      <td>0.082109</td>\n",
       "      <td>0.166492</td>\n",
       "      <td>0.858999</td>\n",
       "      <td>0.291664</td>\n",
       "      <td>-0.241802</td>\n",
       "      <td>0.136455</td>\n",
       "      <td>0.268587</td>\n",
       "      <td>-0.090329</td>\n",
       "      <td>-0.208947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30028</th>\n",
       "      <td>-0.152333</td>\n",
       "      <td>-0.593858</td>\n",
       "      <td>-0.690671</td>\n",
       "      <td>-0.726842</td>\n",
       "      <td>-0.789973</td>\n",
       "      <td>-0.786032</td>\n",
       "      <td>-0.779394</td>\n",
       "      <td>-0.828454</td>\n",
       "      <td>-0.909023</td>\n",
       "      <td>-0.966400</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.035467</td>\n",
       "      <td>-0.128871</td>\n",
       "      <td>-0.052516</td>\n",
       "      <td>0.921492</td>\n",
       "      <td>0.443399</td>\n",
       "      <td>-0.091067</td>\n",
       "      <td>0.284263</td>\n",
       "      <td>0.286236</td>\n",
       "      <td>-0.104119</td>\n",
       "      <td>0.025490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30029</th>\n",
       "      <td>-0.177146</td>\n",
       "      <td>-0.593535</td>\n",
       "      <td>-0.690716</td>\n",
       "      <td>-0.726345</td>\n",
       "      <td>-0.790382</td>\n",
       "      <td>-0.786526</td>\n",
       "      <td>-0.779568</td>\n",
       "      <td>-0.830750</td>\n",
       "      <td>-0.915029</td>\n",
       "      <td>-0.967179</td>\n",
       "      <td>...</td>\n",
       "      <td>0.029496</td>\n",
       "      <td>0.066922</td>\n",
       "      <td>-0.645774</td>\n",
       "      <td>0.464397</td>\n",
       "      <td>0.530886</td>\n",
       "      <td>0.429157</td>\n",
       "      <td>0.934412</td>\n",
       "      <td>0.625979</td>\n",
       "      <td>0.881137</td>\n",
       "      <td>0.397740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30030</th>\n",
       "      <td>-0.103032</td>\n",
       "      <td>-0.591436</td>\n",
       "      <td>-0.688319</td>\n",
       "      <td>-0.720435</td>\n",
       "      <td>-0.786312</td>\n",
       "      <td>-0.781232</td>\n",
       "      <td>-0.776616</td>\n",
       "      <td>-0.824739</td>\n",
       "      <td>-0.911843</td>\n",
       "      <td>-0.972773</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.061042</td>\n",
       "      <td>-0.047908</td>\n",
       "      <td>-0.443287</td>\n",
       "      <td>0.265483</td>\n",
       "      <td>0.316084</td>\n",
       "      <td>0.449696</td>\n",
       "      <td>0.861230</td>\n",
       "      <td>0.568396</td>\n",
       "      <td>0.553170</td>\n",
       "      <td>0.311128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30031</th>\n",
       "      <td>-0.628451</td>\n",
       "      <td>1.478194</td>\n",
       "      <td>1.291831</td>\n",
       "      <td>0.971006</td>\n",
       "      <td>1.226689</td>\n",
       "      <td>1.495998</td>\n",
       "      <td>1.564524</td>\n",
       "      <td>1.637986</td>\n",
       "      <td>1.511991</td>\n",
       "      <td>1.233386</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.446681</td>\n",
       "      <td>0.032039</td>\n",
       "      <td>-0.407846</td>\n",
       "      <td>0.603686</td>\n",
       "      <td>0.335089</td>\n",
       "      <td>-0.486294</td>\n",
       "      <td>0.209921</td>\n",
       "      <td>-0.125476</td>\n",
       "      <td>-0.050803</td>\n",
       "      <td>0.137369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30032</th>\n",
       "      <td>-0.491152</td>\n",
       "      <td>4.312278</td>\n",
       "      <td>3.026329</td>\n",
       "      <td>3.150740</td>\n",
       "      <td>4.943784</td>\n",
       "      <td>4.504436</td>\n",
       "      <td>3.031359</td>\n",
       "      <td>2.001803</td>\n",
       "      <td>1.712816</td>\n",
       "      <td>2.783904</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.480531</td>\n",
       "      <td>-0.423175</td>\n",
       "      <td>-0.539200</td>\n",
       "      <td>0.170848</td>\n",
       "      <td>-0.032509</td>\n",
       "      <td>-0.668649</td>\n",
       "      <td>-0.224764</td>\n",
       "      <td>-0.268442</td>\n",
       "      <td>-0.317023</td>\n",
       "      <td>-0.264707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30033</th>\n",
       "      <td>-0.463041</td>\n",
       "      <td>1.042719</td>\n",
       "      <td>1.027699</td>\n",
       "      <td>1.067549</td>\n",
       "      <td>0.942277</td>\n",
       "      <td>1.278240</td>\n",
       "      <td>1.298286</td>\n",
       "      <td>1.825039</td>\n",
       "      <td>1.912332</td>\n",
       "      <td>0.895738</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125908</td>\n",
       "      <td>-0.487457</td>\n",
       "      <td>0.213026</td>\n",
       "      <td>-0.251535</td>\n",
       "      <td>-0.356534</td>\n",
       "      <td>0.075975</td>\n",
       "      <td>-0.365294</td>\n",
       "      <td>0.040902</td>\n",
       "      <td>-0.308515</td>\n",
       "      <td>-0.184766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30034</th>\n",
       "      <td>-0.339190</td>\n",
       "      <td>1.840738</td>\n",
       "      <td>2.743920</td>\n",
       "      <td>2.352763</td>\n",
       "      <td>1.766222</td>\n",
       "      <td>1.875938</td>\n",
       "      <td>1.432170</td>\n",
       "      <td>2.252015</td>\n",
       "      <td>2.148173</td>\n",
       "      <td>1.370799</td>\n",
       "      <td>...</td>\n",
       "      <td>0.737523</td>\n",
       "      <td>0.678625</td>\n",
       "      <td>0.144455</td>\n",
       "      <td>-0.201774</td>\n",
       "      <td>0.256249</td>\n",
       "      <td>1.104631</td>\n",
       "      <td>0.688357</td>\n",
       "      <td>0.337480</td>\n",
       "      <td>-0.141274</td>\n",
       "      <td>0.988306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30035</th>\n",
       "      <td>-0.664271</td>\n",
       "      <td>1.717509</td>\n",
       "      <td>1.396907</td>\n",
       "      <td>1.150691</td>\n",
       "      <td>1.599951</td>\n",
       "      <td>1.855306</td>\n",
       "      <td>1.841164</td>\n",
       "      <td>1.660395</td>\n",
       "      <td>1.481867</td>\n",
       "      <td>1.361102</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.178550</td>\n",
       "      <td>-0.197096</td>\n",
       "      <td>-0.045155</td>\n",
       "      <td>0.751134</td>\n",
       "      <td>0.414169</td>\n",
       "      <td>-0.052535</td>\n",
       "      <td>0.934385</td>\n",
       "      <td>0.271538</td>\n",
       "      <td>-0.430419</td>\n",
       "      <td>0.403294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30036</th>\n",
       "      <td>-0.661311</td>\n",
       "      <td>2.016270</td>\n",
       "      <td>2.271368</td>\n",
       "      <td>2.057335</td>\n",
       "      <td>1.715165</td>\n",
       "      <td>1.402226</td>\n",
       "      <td>1.495051</td>\n",
       "      <td>1.728025</td>\n",
       "      <td>1.814081</td>\n",
       "      <td>1.498268</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.238017</td>\n",
       "      <td>-0.355145</td>\n",
       "      <td>-0.121101</td>\n",
       "      <td>0.325191</td>\n",
       "      <td>0.007153</td>\n",
       "      <td>-0.266456</td>\n",
       "      <td>0.242683</td>\n",
       "      <td>0.138633</td>\n",
       "      <td>-0.357971</td>\n",
       "      <td>-0.080311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30037</th>\n",
       "      <td>-0.314646</td>\n",
       "      <td>1.329707</td>\n",
       "      <td>1.024431</td>\n",
       "      <td>0.905956</td>\n",
       "      <td>1.607941</td>\n",
       "      <td>1.844308</td>\n",
       "      <td>1.620211</td>\n",
       "      <td>1.295304</td>\n",
       "      <td>1.073473</td>\n",
       "      <td>1.660289</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.183786</td>\n",
       "      <td>0.291758</td>\n",
       "      <td>-0.228583</td>\n",
       "      <td>0.408812</td>\n",
       "      <td>-0.150886</td>\n",
       "      <td>-0.373962</td>\n",
       "      <td>0.320034</td>\n",
       "      <td>0.338477</td>\n",
       "      <td>-0.283166</td>\n",
       "      <td>-0.307333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30038</th>\n",
       "      <td>-0.350195</td>\n",
       "      <td>0.644452</td>\n",
       "      <td>0.515732</td>\n",
       "      <td>0.374304</td>\n",
       "      <td>0.642930</td>\n",
       "      <td>0.964873</td>\n",
       "      <td>0.991537</td>\n",
       "      <td>1.077226</td>\n",
       "      <td>0.980576</td>\n",
       "      <td>0.973787</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.361140</td>\n",
       "      <td>0.092962</td>\n",
       "      <td>-0.076956</td>\n",
       "      <td>0.227946</td>\n",
       "      <td>-0.217697</td>\n",
       "      <td>-0.249879</td>\n",
       "      <td>0.340848</td>\n",
       "      <td>0.263588</td>\n",
       "      <td>-0.309642</td>\n",
       "      <td>-0.246748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30039</th>\n",
       "      <td>-0.744814</td>\n",
       "      <td>0.386672</td>\n",
       "      <td>0.350511</td>\n",
       "      <td>0.200695</td>\n",
       "      <td>0.202944</td>\n",
       "      <td>0.351672</td>\n",
       "      <td>0.575217</td>\n",
       "      <td>0.747850</td>\n",
       "      <td>0.849147</td>\n",
       "      <td>0.906206</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.117338</td>\n",
       "      <td>0.167958</td>\n",
       "      <td>-0.482008</td>\n",
       "      <td>-0.273804</td>\n",
       "      <td>0.403851</td>\n",
       "      <td>0.304514</td>\n",
       "      <td>-0.280903</td>\n",
       "      <td>-0.343468</td>\n",
       "      <td>0.021163</td>\n",
       "      <td>0.251853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30040</th>\n",
       "      <td>-0.738952</td>\n",
       "      <td>0.475599</td>\n",
       "      <td>0.345284</td>\n",
       "      <td>0.173255</td>\n",
       "      <td>0.332724</td>\n",
       "      <td>0.602177</td>\n",
       "      <td>0.783089</td>\n",
       "      <td>0.834250</td>\n",
       "      <td>0.784363</td>\n",
       "      <td>0.764884</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.025538</td>\n",
       "      <td>0.314229</td>\n",
       "      <td>-0.387365</td>\n",
       "      <td>-0.280155</td>\n",
       "      <td>0.223797</td>\n",
       "      <td>0.203964</td>\n",
       "      <td>-0.260722</td>\n",
       "      <td>-0.344310</td>\n",
       "      <td>0.079906</td>\n",
       "      <td>0.316942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30041</th>\n",
       "      <td>-0.677697</td>\n",
       "      <td>0.708271</td>\n",
       "      <td>0.632062</td>\n",
       "      <td>0.560702</td>\n",
       "      <td>0.755174</td>\n",
       "      <td>1.008145</td>\n",
       "      <td>0.813930</td>\n",
       "      <td>0.901619</td>\n",
       "      <td>1.233414</td>\n",
       "      <td>0.937846</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.172257</td>\n",
       "      <td>0.030701</td>\n",
       "      <td>-0.062837</td>\n",
       "      <td>-0.389247</td>\n",
       "      <td>0.082119</td>\n",
       "      <td>0.356923</td>\n",
       "      <td>-0.310838</td>\n",
       "      <td>-0.124292</td>\n",
       "      <td>-0.053573</td>\n",
       "      <td>-0.305619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30042</th>\n",
       "      <td>-0.635979</td>\n",
       "      <td>0.442597</td>\n",
       "      <td>0.324517</td>\n",
       "      <td>0.233800</td>\n",
       "      <td>0.394775</td>\n",
       "      <td>0.977192</td>\n",
       "      <td>0.779703</td>\n",
       "      <td>0.735811</td>\n",
       "      <td>1.207538</td>\n",
       "      <td>0.847067</td>\n",
       "      <td>...</td>\n",
       "      <td>0.052458</td>\n",
       "      <td>-0.005545</td>\n",
       "      <td>-0.315993</td>\n",
       "      <td>0.291077</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>-0.338434</td>\n",
       "      <td>-0.161119</td>\n",
       "      <td>-0.341574</td>\n",
       "      <td>-0.463772</td>\n",
       "      <td>-0.179059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30043</th>\n",
       "      <td>-0.652526</td>\n",
       "      <td>1.292866</td>\n",
       "      <td>1.023303</td>\n",
       "      <td>0.830107</td>\n",
       "      <td>1.276824</td>\n",
       "      <td>1.539570</td>\n",
       "      <td>1.502081</td>\n",
       "      <td>1.294785</td>\n",
       "      <td>1.249680</td>\n",
       "      <td>1.224159</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.503992</td>\n",
       "      <td>-0.624366</td>\n",
       "      <td>-0.789669</td>\n",
       "      <td>-0.782865</td>\n",
       "      <td>-0.832993</td>\n",
       "      <td>-0.812527</td>\n",
       "      <td>-0.847734</td>\n",
       "      <td>-0.877216</td>\n",
       "      <td>-0.855595</td>\n",
       "      <td>-0.884564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30044</th>\n",
       "      <td>-0.671671</td>\n",
       "      <td>0.063439</td>\n",
       "      <td>-0.090574</td>\n",
       "      <td>-0.139220</td>\n",
       "      <td>0.043530</td>\n",
       "      <td>0.299586</td>\n",
       "      <td>0.409878</td>\n",
       "      <td>0.267186</td>\n",
       "      <td>0.237834</td>\n",
       "      <td>0.428187</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.502257</td>\n",
       "      <td>-0.531408</td>\n",
       "      <td>-0.768533</td>\n",
       "      <td>-0.700554</td>\n",
       "      <td>-0.808599</td>\n",
       "      <td>-0.764569</td>\n",
       "      <td>-0.814271</td>\n",
       "      <td>-0.853243</td>\n",
       "      <td>-0.811112</td>\n",
       "      <td>-0.867129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30045</th>\n",
       "      <td>-0.132189</td>\n",
       "      <td>0.758994</td>\n",
       "      <td>0.652591</td>\n",
       "      <td>0.451061</td>\n",
       "      <td>0.483651</td>\n",
       "      <td>0.801676</td>\n",
       "      <td>1.056698</td>\n",
       "      <td>1.207219</td>\n",
       "      <td>1.456055</td>\n",
       "      <td>1.155180</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.389129</td>\n",
       "      <td>-0.233668</td>\n",
       "      <td>-0.307036</td>\n",
       "      <td>-0.318123</td>\n",
       "      <td>-0.359112</td>\n",
       "      <td>-0.315541</td>\n",
       "      <td>-0.349536</td>\n",
       "      <td>-0.342887</td>\n",
       "      <td>-0.360608</td>\n",
       "      <td>-0.338035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30046</th>\n",
       "      <td>-0.191880</td>\n",
       "      <td>-0.046368</td>\n",
       "      <td>-0.154252</td>\n",
       "      <td>-0.203884</td>\n",
       "      <td>-0.174703</td>\n",
       "      <td>0.100610</td>\n",
       "      <td>0.263635</td>\n",
       "      <td>0.205355</td>\n",
       "      <td>0.292900</td>\n",
       "      <td>0.228203</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.246144</td>\n",
       "      <td>-0.063090</td>\n",
       "      <td>-0.383923</td>\n",
       "      <td>-0.252833</td>\n",
       "      <td>-0.064862</td>\n",
       "      <td>-0.066433</td>\n",
       "      <td>-0.311726</td>\n",
       "      <td>-0.299746</td>\n",
       "      <td>-0.120635</td>\n",
       "      <td>-0.067132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30047</th>\n",
       "      <td>-0.529276</td>\n",
       "      <td>1.138306</td>\n",
       "      <td>0.988815</td>\n",
       "      <td>0.784250</td>\n",
       "      <td>1.036595</td>\n",
       "      <td>1.264824</td>\n",
       "      <td>1.365760</td>\n",
       "      <td>1.301662</td>\n",
       "      <td>1.295606</td>\n",
       "      <td>1.025296</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.335600</td>\n",
       "      <td>-0.374958</td>\n",
       "      <td>-0.439427</td>\n",
       "      <td>-0.000496</td>\n",
       "      <td>-0.292897</td>\n",
       "      <td>-0.263174</td>\n",
       "      <td>0.047646</td>\n",
       "      <td>0.085782</td>\n",
       "      <td>-0.197808</td>\n",
       "      <td>0.000465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30048</th>\n",
       "      <td>-0.575663</td>\n",
       "      <td>-0.115743</td>\n",
       "      <td>-0.194081</td>\n",
       "      <td>-0.265141</td>\n",
       "      <td>-0.212708</td>\n",
       "      <td>-0.072489</td>\n",
       "      <td>0.078034</td>\n",
       "      <td>0.133352</td>\n",
       "      <td>0.188843</td>\n",
       "      <td>0.118130</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.311010</td>\n",
       "      <td>-0.202092</td>\n",
       "      <td>-0.235552</td>\n",
       "      <td>0.241658</td>\n",
       "      <td>-0.243268</td>\n",
       "      <td>-0.117610</td>\n",
       "      <td>0.228114</td>\n",
       "      <td>0.176249</td>\n",
       "      <td>-0.164444</td>\n",
       "      <td>0.112222</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30049 rows × 15960 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5         6      \\\n",
       "0     -1.117537  0.130346  0.013696 -0.103711  0.037901  0.269922  0.399752   \n",
       "1     -1.141154  0.360481  0.266127  0.079885  0.149372  0.394671  0.586697   \n",
       "2     -1.177216 -0.267615 -0.349863 -0.416966 -0.393240 -0.265429 -0.145420   \n",
       "3     -1.125873  0.708498  0.511915  0.345341  0.678018  0.960304  0.994918   \n",
       "4     -1.301681  0.035031 -0.075285 -0.190359 -0.102609  0.108780  0.216456   \n",
       "5     -1.289225  0.618264  0.440396  0.333169  0.531440  0.795734  0.815010   \n",
       "6     -1.280896 -0.263859 -0.355730 -0.418794 -0.397849 -0.229321 -0.186046   \n",
       "7     -1.244046  0.480159  0.363812  0.202020  0.349861  0.564196  0.670118   \n",
       "8     -1.121709  0.117122  0.008087 -0.111065  0.043313  0.242077  0.372759   \n",
       "9     -1.120282  0.673164  0.520629  0.353339  0.604866  0.836481  0.911321   \n",
       "10    -1.161999 -0.170808 -0.280091 -0.345389 -0.215225 -0.066473  0.016016   \n",
       "11    -1.109925  0.348278  0.253625  0.084104  0.181172  0.375732  0.543186   \n",
       "12    -1.583999 -0.101838 -0.187720 -0.265902 -0.204427 -0.061798  0.052346   \n",
       "13    -1.532105  1.795420  1.494828  1.117432  1.433854  1.637722  1.598290   \n",
       "14    -1.556035  0.854810  0.650248  0.425941  0.679988  0.941011  1.006641   \n",
       "15    -1.509894  0.789489  0.734525  0.508362  0.519640  0.669836  0.828015   \n",
       "16    -1.468620 -0.150050 -0.231988 -0.305689 -0.293819 -0.157557 -0.017568   \n",
       "17    -1.348397  1.315641  1.142931  0.767735  0.831600  1.091384  1.288360   \n",
       "18    -1.424677  0.814512  0.635795  0.410424  0.590902  0.841481  0.965004   \n",
       "19    -1.448649  0.953664  0.895262  0.685877  0.683935  0.826307  0.964385   \n",
       "20    -1.278326  0.070219 -0.079248 -0.149880  0.035801  0.252874  0.299863   \n",
       "21    -1.252883  1.984057  1.699722  1.269265  1.680939  1.881720  1.838494   \n",
       "22    -1.262766  0.841785  0.652353  0.419199  0.595580  0.874963  0.988802   \n",
       "23    -1.218103  1.756039  1.288217  1.090831  1.765646  2.002212  1.701695   \n",
       "24    -1.466737  0.364266  0.242847  0.073562  0.230399  0.449496  0.555549   \n",
       "25    -1.352696  1.183222  1.100144  1.028277  1.242592  1.268293  1.135938   \n",
       "26    -1.468931  1.565876  1.444560  1.036704  1.104169  1.244886  1.364998   \n",
       "27    -1.458226  0.916002  0.717254  0.499095  0.826848  1.040904  1.066190   \n",
       "28    -1.244967  0.528076  0.344597  0.199056  0.392641  0.695089  0.841445   \n",
       "29    -1.293424  0.144455  0.085388 -0.061860 -0.129136  0.019201  0.244424   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "30019 -0.370322 -0.226526 -0.080004  0.137031  0.086162 -0.119531 -0.156182   \n",
       "30020 -0.450635 -0.273962 -0.139269  0.032197 -0.022757 -0.183572 -0.200784   \n",
       "30021 -0.187353 -0.270201  0.036723  0.143085  0.077921 -0.010722 -0.187752   \n",
       "30022 -0.131709 -0.348489  0.022155  0.146214 -0.060469  0.040867 -0.220123   \n",
       "30023 -0.257295 -0.020362  0.223802  0.497554  0.422577  0.160234  0.069173   \n",
       "30024 -0.235252 -0.348016 -0.273893 -0.148297 -0.189325 -0.300012 -0.311168   \n",
       "30025 -0.640975 -0.069737  0.169179  0.413493  0.326494  0.088665  0.006096   \n",
       "30026 -0.663981 -0.372109 -0.311566 -0.198239 -0.240748 -0.326981 -0.340134   \n",
       "30027 -0.230168 -0.593888 -0.690791 -0.726956 -0.790438 -0.786404 -0.779407   \n",
       "30028 -0.152333 -0.593858 -0.690671 -0.726842 -0.789973 -0.786032 -0.779394   \n",
       "30029 -0.177146 -0.593535 -0.690716 -0.726345 -0.790382 -0.786526 -0.779568   \n",
       "30030 -0.103032 -0.591436 -0.688319 -0.720435 -0.786312 -0.781232 -0.776616   \n",
       "30031 -0.628451  1.478194  1.291831  0.971006  1.226689  1.495998  1.564524   \n",
       "30032 -0.491152  4.312278  3.026329  3.150740  4.943784  4.504436  3.031359   \n",
       "30033 -0.463041  1.042719  1.027699  1.067549  0.942277  1.278240  1.298286   \n",
       "30034 -0.339190  1.840738  2.743920  2.352763  1.766222  1.875938  1.432170   \n",
       "30035 -0.664271  1.717509  1.396907  1.150691  1.599951  1.855306  1.841164   \n",
       "30036 -0.661311  2.016270  2.271368  2.057335  1.715165  1.402226  1.495051   \n",
       "30037 -0.314646  1.329707  1.024431  0.905956  1.607941  1.844308  1.620211   \n",
       "30038 -0.350195  0.644452  0.515732  0.374304  0.642930  0.964873  0.991537   \n",
       "30039 -0.744814  0.386672  0.350511  0.200695  0.202944  0.351672  0.575217   \n",
       "30040 -0.738952  0.475599  0.345284  0.173255  0.332724  0.602177  0.783089   \n",
       "30041 -0.677697  0.708271  0.632062  0.560702  0.755174  1.008145  0.813930   \n",
       "30042 -0.635979  0.442597  0.324517  0.233800  0.394775  0.977192  0.779703   \n",
       "30043 -0.652526  1.292866  1.023303  0.830107  1.276824  1.539570  1.502081   \n",
       "30044 -0.671671  0.063439 -0.090574 -0.139220  0.043530  0.299586  0.409878   \n",
       "30045 -0.132189  0.758994  0.652591  0.451061  0.483651  0.801676  1.056698   \n",
       "30046 -0.191880 -0.046368 -0.154252 -0.203884 -0.174703  0.100610  0.263635   \n",
       "30047 -0.529276  1.138306  0.988815  0.784250  1.036595  1.264824  1.365760   \n",
       "30048 -0.575663 -0.115743 -0.194081 -0.265141 -0.212708 -0.072489  0.078034   \n",
       "\n",
       "          7         8         9        ...        15950     15951     15952  \\\n",
       "0      0.415404  0.358259  0.303114    ...    -0.393410 -0.301521 -0.367537   \n",
       "1      0.697098  0.761406  0.658471    ...    -0.439485 -0.473423 -0.539600   \n",
       "2     -0.109582 -0.151950 -0.193078    ...    -0.430591 -0.422903 -0.480570   \n",
       "3      0.879390  0.731289  0.780614    ...    -0.458453 -0.547680 -0.616144   \n",
       "4      0.411433  0.163525  0.113291    ...    -0.494304 -0.691341 -0.765498   \n",
       "5      0.934647  0.583534  0.673116    ...    -0.493536 -0.691717 -0.767147   \n",
       "6     -0.037326 -0.213264 -0.233464    ...    -0.490703 -0.694423 -0.769091   \n",
       "7      0.849204  0.574173  0.503616    ...    -0.484862 -0.684433 -0.766651   \n",
       "8      0.408476  0.373977  0.232905    ...    -0.388380 -0.487897 -0.678482   \n",
       "9      0.919284  0.833911  0.680132    ...    -0.430117 -0.472262 -0.542216   \n",
       "10    -0.005436 -0.030528 -0.125808    ...    -0.381401 -0.405521 -0.553988   \n",
       "11     0.703394  0.773510  0.455338    ...    -0.409108 -0.389607 -0.450645   \n",
       "12     0.112209  0.046005 -0.054308    ...    -0.490387 -0.693437 -0.770339   \n",
       "13     1.470541  1.201826  1.022441    ...    -0.458935 -0.596976 -0.675757   \n",
       "14     0.975626  0.819083  0.644963    ...    -0.478982 -0.658896 -0.736396   \n",
       "15     0.975663  0.960331  0.713935    ...    -0.453918 -0.576785 -0.653024   \n",
       "16     0.111731  0.032982 -0.195766    ...     0.107181  0.317355  0.280894   \n",
       "17     1.449918  1.262782  1.106336    ...     0.226749  0.966048  0.364869   \n",
       "18     1.077745  0.814894  0.579525    ...    -0.000878  0.100409  0.058536   \n",
       "19     1.128858  0.946639  0.595656    ...     0.110359  0.549694  0.188414   \n",
       "20     0.363914  0.041698  0.194562    ...    -0.481081 -0.384489 -0.305489   \n",
       "21     1.685328  1.238330  1.162364    ...    -0.481664 -0.278626 -0.206689   \n",
       "22     1.150711  0.816094  0.826144    ...    -0.490537 -0.272377 -0.196730   \n",
       "23     1.420702  0.913363  1.152832    ...    -0.504447 -0.481353 -0.451276   \n",
       "24     0.666431  0.578242  0.337261    ...    -0.507080 -0.664343 -0.723486   \n",
       "25     1.037624  0.812252  0.582882    ...    -0.505223 -0.605229 -0.602400   \n",
       "26     1.461079  1.428367  0.861658    ...    -0.506524 -0.666557 -0.697989   \n",
       "27     1.015816  0.818418  0.581877    ...    -0.507031 -0.663400 -0.696159   \n",
       "28     0.788050  0.620654  0.633079    ...    -0.500227 -0.435424 -0.764507   \n",
       "29     0.500393  0.708577  0.530286    ...    -0.500603 -0.392792 -0.770519   \n",
       "...         ...       ...       ...    ...          ...       ...       ...   \n",
       "30019 -0.074760  0.064308  0.348836    ...    -0.485064 -0.706802 -0.775669   \n",
       "30020 -0.168052 -0.012166  0.224280    ...    -0.500063 -0.705547 -0.792829   \n",
       "30021  0.038560  0.140613  0.255097    ...    -0.426354 -0.136415 -0.323139   \n",
       "30022  0.010850  0.237173  0.125278    ...    -0.347662  0.414243 -0.150452   \n",
       "30023  0.154584  0.333823  0.488834    ...     0.870766  1.392696  0.523158   \n",
       "30024 -0.208134 -0.122402 -0.099617    ...     1.505774  2.067066  1.219951   \n",
       "30025  0.033707  0.180667  0.612531    ...    -0.347120 -0.463630 -0.278552   \n",
       "30026 -0.322195 -0.273446  0.002196    ...    -0.402475 -0.618906 -0.528591   \n",
       "30027 -0.828310 -0.906732 -0.953445    ...    -0.086721  0.082109  0.166492   \n",
       "30028 -0.828454 -0.909023 -0.966400    ...    -0.035467 -0.128871 -0.052516   \n",
       "30029 -0.830750 -0.915029 -0.967179    ...     0.029496  0.066922 -0.645774   \n",
       "30030 -0.824739 -0.911843 -0.972773    ...    -0.061042 -0.047908 -0.443287   \n",
       "30031  1.637986  1.511991  1.233386    ...    -0.446681  0.032039 -0.407846   \n",
       "30032  2.001803  1.712816  2.783904    ...    -0.480531 -0.423175 -0.539200   \n",
       "30033  1.825039  1.912332  0.895738    ...    -0.125908 -0.487457  0.213026   \n",
       "30034  2.252015  2.148173  1.370799    ...     0.737523  0.678625  0.144455   \n",
       "30035  1.660395  1.481867  1.361102    ...    -0.178550 -0.197096 -0.045155   \n",
       "30036  1.728025  1.814081  1.498268    ...    -0.238017 -0.355145 -0.121101   \n",
       "30037  1.295304  1.073473  1.660289    ...    -0.183786  0.291758 -0.228583   \n",
       "30038  1.077226  0.980576  0.973787    ...    -0.361140  0.092962 -0.076956   \n",
       "30039  0.747850  0.849147  0.906206    ...    -0.117338  0.167958 -0.482008   \n",
       "30040  0.834250  0.784363  0.764884    ...    -0.025538  0.314229 -0.387365   \n",
       "30041  0.901619  1.233414  0.937846    ...    -0.172257  0.030701 -0.062837   \n",
       "30042  0.735811  1.207538  0.847067    ...     0.052458 -0.005545 -0.315993   \n",
       "30043  1.294785  1.249680  1.224159    ...    -0.503992 -0.624366 -0.789669   \n",
       "30044  0.267186  0.237834  0.428187    ...    -0.502257 -0.531408 -0.768533   \n",
       "30045  1.207219  1.456055  1.155180    ...    -0.389129 -0.233668 -0.307036   \n",
       "30046  0.205355  0.292900  0.228203    ...    -0.246144 -0.063090 -0.383923   \n",
       "30047  1.301662  1.295606  1.025296    ...    -0.335600 -0.374958 -0.439427   \n",
       "30048  0.133352  0.188843  0.118130    ...    -0.311010 -0.202092 -0.235552   \n",
       "\n",
       "          15953     15954     15955     15956     15957     15958     15959  \n",
       "0     -0.394536 -0.370814 -0.407677 -0.416760 -0.401391 -0.409848 -0.426182  \n",
       "1     -0.589093 -0.570107 -0.601153 -0.609641 -0.615344 -0.617178 -0.622111  \n",
       "2     -0.536433 -0.519772 -0.540244 -0.550040 -0.558432 -0.558878 -0.562048  \n",
       "3     -0.673036 -0.654511 -0.685598 -0.695242 -0.707149 -0.706268 -0.709202  \n",
       "4     -0.832934 -0.815301 -0.849757 -0.864321 -0.889805 -0.880755 -0.883626  \n",
       "5     -0.832168 -0.814264 -0.850489 -0.864632 -0.889108 -0.880674 -0.884081  \n",
       "6     -0.828314 -0.812143 -0.851598 -0.864812 -0.884870 -0.880759 -0.885678  \n",
       "7     -0.814271 -0.794596 -0.843836 -0.857055 -0.869302 -0.864939 -0.877402  \n",
       "8     -0.671614 -0.536393 -0.580206 -0.699196 -0.727652 -0.622976 -0.591967  \n",
       "9     -0.584586 -0.566216 -0.587505 -0.606463 -0.627249 -0.609860 -0.606361  \n",
       "10    -0.555117 -0.461654 -0.492020 -0.577277 -0.603402 -0.526465 -0.502749  \n",
       "11    -0.486197 -0.474785 -0.491295 -0.505612 -0.522966 -0.508114 -0.506073  \n",
       "12    -0.829602 -0.813115 -0.852158 -0.862470 -0.881090 -0.876895 -0.881600  \n",
       "13    -0.711822 -0.694355 -0.740111 -0.751211 -0.754784 -0.750471 -0.766673  \n",
       "14    -0.786870 -0.770309 -0.812058 -0.822155 -0.835113 -0.831429 -0.840272  \n",
       "15    -0.687564 -0.671862 -0.716031 -0.725677 -0.729007 -0.725913 -0.741710  \n",
       "16    -0.044644 -0.254345 -0.306427 -0.400406  0.201104  0.281676  0.180174  \n",
       "17    -0.167786 -0.200911 -0.319185 -0.324164  0.068363  0.586419  0.695491  \n",
       "18    -0.368178 -0.425352 -0.474284 -0.557892 -0.182215  0.025371 -0.024643  \n",
       "19    -0.263452 -0.286069 -0.386958 -0.403285  0.008902  0.312088  0.348862  \n",
       "20    -0.486858 -0.565668 -0.485949 -0.434295 -0.501755 -0.554358 -0.533938  \n",
       "21    -0.456319 -0.476558 -0.353213 -0.384215 -0.448523 -0.418716 -0.439497  \n",
       "22    -0.474109 -0.486586 -0.343549 -0.385533 -0.464477 -0.425519 -0.435991  \n",
       "23    -0.645964 -0.666999 -0.599047 -0.605200 -0.662121 -0.662725 -0.666251  \n",
       "24    -0.839001 -0.819098 -0.836303 -0.846125 -0.863365 -0.877635 -0.875626  \n",
       "25    -0.741086 -0.779414 -0.765720 -0.728375 -0.762693 -0.814653 -0.799628  \n",
       "26    -0.811486 -0.818870 -0.820644 -0.813846 -0.853022 -0.876866 -0.854819  \n",
       "27    -0.808843 -0.817185 -0.820607 -0.811046 -0.848498 -0.872878 -0.856313  \n",
       "28    -0.574717 -0.790852 -0.635907 -0.775147 -0.717072 -0.744430 -0.754885  \n",
       "29    -0.548080 -0.806400 -0.604118 -0.797249 -0.664572 -0.764990 -0.706043  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "30019 -0.843690 -0.842398 -0.833092 -0.876333 -0.904293 -0.890757 -0.892930  \n",
       "30020 -0.869371 -0.855394 -0.865562 -0.892436 -0.922987 -0.913269 -0.915168  \n",
       "30021 -0.566377 -0.335159 -0.632194 -0.419565 -0.375516 -0.523034 -0.301023  \n",
       "30022 -0.455553 -0.210063 -0.671269 -0.196376 -0.079386 -0.263368  0.075856  \n",
       "30023  2.352454  0.231209  0.724496  1.456699  0.416698  1.021780  1.426201  \n",
       "30024  3.248626  0.528642  1.468564  2.180165  0.794355  1.481247  2.325696  \n",
       "30025 -0.568210 -0.260883 -0.317195 -0.248857 -0.135519 -0.345025 -0.143403  \n",
       "30026 -0.747857 -0.519908 -0.555486 -0.473275 -0.472852 -0.569673 -0.447912  \n",
       "30027  0.858999  0.291664 -0.241802  0.136455  0.268587 -0.090329 -0.208947  \n",
       "30028  0.921492  0.443399 -0.091067  0.284263  0.286236 -0.104119  0.025490  \n",
       "30029  0.464397  0.530886  0.429157  0.934412  0.625979  0.881137  0.397740  \n",
       "30030  0.265483  0.316084  0.449696  0.861230  0.568396  0.553170  0.311128  \n",
       "30031  0.603686  0.335089 -0.486294  0.209921 -0.125476 -0.050803  0.137369  \n",
       "30032  0.170848 -0.032509 -0.668649 -0.224764 -0.268442 -0.317023 -0.264707  \n",
       "30033 -0.251535 -0.356534  0.075975 -0.365294  0.040902 -0.308515 -0.184766  \n",
       "30034 -0.201774  0.256249  1.104631  0.688357  0.337480 -0.141274  0.988306  \n",
       "30035  0.751134  0.414169 -0.052535  0.934385  0.271538 -0.430419  0.403294  \n",
       "30036  0.325191  0.007153 -0.266456  0.242683  0.138633 -0.357971 -0.080311  \n",
       "30037  0.408812 -0.150886 -0.373962  0.320034  0.338477 -0.283166 -0.307333  \n",
       "30038  0.227946 -0.217697 -0.249879  0.340848  0.263588 -0.309642 -0.246748  \n",
       "30039 -0.273804  0.403851  0.304514 -0.280903 -0.343468  0.021163  0.251853  \n",
       "30040 -0.280155  0.223797  0.203964 -0.260722 -0.344310  0.079906  0.316942  \n",
       "30041 -0.389247  0.082119  0.356923 -0.310838 -0.124292 -0.053573 -0.305619  \n",
       "30042  0.291077  0.024980 -0.338434 -0.161119 -0.341574 -0.463772 -0.179059  \n",
       "30043 -0.782865 -0.832993 -0.812527 -0.847734 -0.877216 -0.855595 -0.884564  \n",
       "30044 -0.700554 -0.808599 -0.764569 -0.814271 -0.853243 -0.811112 -0.867129  \n",
       "30045 -0.318123 -0.359112 -0.315541 -0.349536 -0.342887 -0.360608 -0.338035  \n",
       "30046 -0.252833 -0.064862 -0.066433 -0.311726 -0.299746 -0.120635 -0.067132  \n",
       "30047 -0.000496 -0.292897 -0.263174  0.047646  0.085782 -0.197808  0.000465  \n",
       "30048  0.241658 -0.243268 -0.117610  0.228114  0.176249 -0.164444  0.112222  \n",
       "\n",
       "[30049 rows x 15960 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_df=(x_df-x_df.mean())/x_df.std()\n",
    "x_df = x_df.drop(15960, axis=1)\n",
    "x_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalization is a linear transformation and thus should have any influence on the predictions. We choose to normalize not for results but for smoothness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_rm = []\n",
    "for i in [i for i in x_df if i < 50]:\n",
    "    for j in [j for j in x_df if j > i]:\n",
    "        if x_df[i].corr(x_df[j]) > 0.95:\n",
    "            to_rm.append(j)\n",
    "x_df_uncorr = x_df.drop(to_rm, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(30049, 15731)\n",
      "0.833248890938342\n"
     ]
    }
   ],
   "source": [
    "print(x_df_uncorr.shape)\n",
    "test_quality(x_df_uncorr, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those results are surprising as we were expecting the error to go down by removing some noise, but we can see that correlation actually make the results worse. We won't use it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500 1.24682949543426\n",
      "1000 1.0759735036260647\n",
      "1500 1.0285683540379094\n",
      "2000 0.9957650465725961\n",
      "2500 0.9758586786783904\n",
      "3000 0.9779433826750218\n",
      "3500 1.0375654778387124\n",
      "4000 0.974439824755514\n",
      "4500 0.954479099241363\n"
     ]
    }
   ],
   "source": [
    "rmses = []\n",
    "ys = range(500, 5501, 500)\n",
    "for i in ys:\n",
    "    pca = PCA(n_components=i, whiten=True)\n",
    "    principalComponents = pca.fit_transform(x_df, x_df.shape)\n",
    "    principalDf = pd.DataFrame(data = principalComponents\n",
    "                 , columns = range(principalComponents.shape[1]))\n",
    "    err = test_quality(principalDf, y)\n",
    "    rmses.append(err)\n",
    "    print(i, err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.xlabel(\"Number of components\")\n",
    "plt.ylabel(\"RMSE\")\n",
    "plt.plot(ys, rmses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error plotted is relatively high when reducing the number of components below 6000. We are not sure of the conlusion to give here, as we were expecting results less good then the full matrix but not to that extend. We can argue that our `test_quality` function is absolutely not optimized, but after a look at the scientific side, 1000 features should be enough to get good predictions. We then have to choose by default the best one of the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!! __Note for the reader__: the cells above are the ones that generated the plot Fig. 1 you can find on the report. Unfortunately, we add to re run the cell and could not do it to completion due to time issues (takes several hours to run on our machines). If you want to check the correctness of our plot and regenerate it, feel free to run the cells. !!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.771265777776529\n"
     ]
    }
   ],
   "source": [
    "i_star = 3003\n",
    "pca = PCA(n_components=i_star)\n",
    "principalComponents = pca.fit_transform(x_df, x_df.shape)\n",
    "principalDf_star = pd.DataFrame(data = principalComponents\n",
    "             , columns = range(principalComponents.shape[1]))\n",
    "test_quality(principalDf_star, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.841288980072011\n"
     ]
    }
   ],
   "source": [
    "i_star = 3003\n",
    "pca = PCA(n_components=i_star, whiten=True)\n",
    "principalComponents_no_w = pca.fit_transform(x_df, x_df.shape)\n",
    "principalDf_star_no_w = pd.DataFrame(data = principalComponents_no_w\n",
    "             , columns = range(principalComponents_no_w.shape[1]))\n",
    "test_quality(principalDf_star_no_w, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whitening does not improve anything, even worsen the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ValueError: math domain error is known bug : https://github.com/scikit-learn/scikit-learn/issues/10217: Cannot use MLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The disapointement is real not to be able to use Maximum Likelyhood Estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the data for Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FOLDER = \"data/\"\n",
    "X = np.load(DATA_FOLDER + \"feature_mat_radial_compression.npy\")\n",
    "y = np.load(DATA_FOLDER + \"CSD500-r_train-H_total.npy\")\n",
    "x_df = pd.DataFrame(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Apply PCA\n",
    "i_star = 4500\n",
    "pca = PCA(n_components=i_star)\n",
    "principalComponents = pca.fit_transform(x_df, x_df.shape)\n",
    "x_pca_df = pd.DataFrame(data = principalComponents\n",
    "             , columns = range(principalComponents.shape[1]))\n",
    "x_pca_df.columns = range(x_pca_df.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Add jitter\n",
    "x_with_jitter_df, y_with_jitter = add_jitter(x_pca_df, y, 0.01, 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Normalize\n",
    "x_with_jitter_df=(x_with_jitter_df-x_with_jitter_df.mean())/x_with_jitter_df.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(\"data/ML/x_train.npy\", x_with_jitter_df)\n",
    "np.save(\"data/ML/y.npy\", y_with_jitter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the data for Deep Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FOLDER = \"data/\"\n",
    "X = np.load(DATA_FOLDER + \"feature_mat_radial_compression.npy\")\n",
    "y = np.load(DATA_FOLDER + \"CSD500-r_train-H_total.npy\")\n",
    "x_df = pd.DataFrame(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Apply PCA\n",
    "i_star = 3004\n",
    "pca = PCA(n_components=i_star)\n",
    "principalComponents = pca.fit_transform(x_df, x_df.shape)\n",
    "x_pca_df = pd.DataFrame(data = principalComponents\n",
    "             , columns = range(principalComponents.shape[1]))\n",
    "x_pca_df.columns = range(x_pca_df.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Normalize\n",
    "x_pca_df=(x_pca_df-x_pca_df.mean())/x_pca_df.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(\"data/feature_mat_radial_compression_normalized_red.npy\", x_pca_df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}