Downsample-time-series-using-local-median.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8fada4de",
   "metadata": {},
   "source": [
    "# Downsample-timeseries-using-local-median\n",
    "\n",
    "Suppose you have data sampled at a higher frequency, e.g. every second and you only need samples every minute.  If you simply use the [Pandas resample function](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html) your resampled signal will still be sensitive to any noise or other high frequency variations in the data that you are not interested in.\n",
    "\n",
    "```python\n",
    "samples_1min = x.resample(rule='1min').agg('first')\n",
    "```\n",
    "\n",
    "Alternatively, you could take an average of the values in each sample period.\n",
    "\n",
    "```python\n",
    "samples_1min = x.resample(rule='1min').mean()\n",
    "```\n",
    "\n",
    "But this does a lot of smoothing of the data and actually introduces a delay into any dynamics you are trying to identify.\n",
    "\n",
    "Obviously, the best thing to do is to use an appropriate filter that takes into account the expected time-response of the system generating the data, but this requires a model of the dynamics.  When you are doing system identification, you don't have this model at the outset.  You could try fitting a model to all the data but that might be computationally expensive and unnecessary if the sampling frequency is very much faster than the dynamics you are trying to identify.\n",
    "\n",
    "The technique I propose here is to take a short sample of values immediately before and after each sample time and take a median of these values to represent the estimated true value at the centre of each sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f7037e0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "30559b44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:17</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:18</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:19</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:20</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:21</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:53</th>\n",
       "      <td>156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:54</th>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:55</th>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:56</th>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:57</th>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>161 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       x\n",
       "t                       \n",
       "2021-12-01 06:00:17    0\n",
       "2021-12-01 06:00:18    1\n",
       "2021-12-01 06:00:19    2\n",
       "2021-12-01 06:00:20    3\n",
       "2021-12-01 06:00:21    4\n",
       "...                  ...\n",
       "2021-12-01 06:02:53  156\n",
       "2021-12-01 06:02:54  157\n",
       "2021-12-01 06:02:55  158\n",
       "2021-12-01 06:02:56  159\n",
       "2021-12-01 06:02:57  160\n",
       "\n",
       "[161 rows x 1 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare an artificial time series with a sample period of\n",
    "# 1 second to represent the signal data\n",
    "index = pd.date_range('2021-12-01 06:00:17', '2021-12-01 06:02:57', \n",
    "                      freq='1s', name='t')\n",
    "data = {'x': range(len(index))}\n",
    "x = pd.DataFrame(data, index=index)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f467a53",
   "metadata": {},
   "source": [
    "In this example the values sampled every minute should be the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9b178397",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       x\n",
      "t                       \n",
      "2021-12-01 06:01:00   43\n",
      "2021-12-01 06:02:00  103\n"
     ]
    }
   ],
   "source": [
    "print(x.loc[['2021-12-01 06:01:00', '2021-12-01 06:02:00']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9cac3fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:10</th>\n",
       "      <th>x</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:20</th>\n",
       "      <th>x</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:30</th>\n",
       "      <th>x</th>\n",
       "      <td>13.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:40</th>\n",
       "      <th>x</th>\n",
       "      <td>23.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:50</th>\n",
       "      <th>x</th>\n",
       "      <td>33.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>42.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          0     1     2     3     4     5     6     7     8  \\\n",
       "2021-12-01 06:00:10 x   0.0   1.0   2.0   NaN   NaN   NaN   NaN   NaN   NaN   \n",
       "2021-12-01 06:00:20 x   3.0   4.0   5.0   6.0   7.0   8.0   9.0  10.0  11.0   \n",
       "2021-12-01 06:00:30 x  13.0  14.0  15.0  16.0  17.0  18.0  19.0  20.0  21.0   \n",
       "2021-12-01 06:00:40 x  23.0  24.0  25.0  26.0  27.0  28.0  29.0  30.0  31.0   \n",
       "2021-12-01 06:00:50 x  33.0  34.0  35.0  36.0  37.0  38.0  39.0  40.0  41.0   \n",
       "\n",
       "                          9  \n",
       "2021-12-01 06:00:10 x   NaN  \n",
       "2021-12-01 06:00:20 x  12.0  \n",
       "2021-12-01 06:00:30 x  22.0  \n",
       "2021-12-01 06:00:40 x  32.0  \n",
       "2021-12-01 06:00:50 x  42.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First, lets try sampling the signal at 10 second intervals \n",
    "# and organize the samples into a dataframe so we can see them\n",
    "resample_10sec = x.resample(rule='10s', offset=0)\n",
    "keys = [k for k, v in resample_10sec]\n",
    "samples = [v for k, v in resample_10sec]\n",
    "summary = pd.concat([s.reset_index(drop=True) for s in samples], axis=1, keys=keys).T\n",
    "summary.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "519d0ead",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:10</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:20</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:30</th>\n",
       "      <td>13.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:40</th>\n",
       "      <td>23.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:50</th>\n",
       "      <td>33.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>42.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        0     1     2     3     4     5     6     7     8  \\\n",
       "2021-12-01 06:00:10   0.0   1.0   2.0   NaN   NaN   NaN   NaN   NaN   NaN   \n",
       "2021-12-01 06:00:20   3.0   4.0   5.0   6.0   7.0   8.0   9.0  10.0  11.0   \n",
       "2021-12-01 06:00:30  13.0  14.0  15.0  16.0  17.0  18.0  19.0  20.0  21.0   \n",
       "2021-12-01 06:00:40  23.0  24.0  25.0  26.0  27.0  28.0  29.0  30.0  31.0   \n",
       "2021-12-01 06:00:50  33.0  34.0  35.0  36.0  37.0  38.0  39.0  40.0  41.0   \n",
       "\n",
       "                        9  \n",
       "2021-12-01 06:00:10   NaN  \n",
       "2021-12-01 06:00:20  12.0  \n",
       "2021-12-01 06:00:30  22.0  \n",
       "2021-12-01 06:00:40  32.0  \n",
       "2021-12-01 06:00:50  42.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the unnecessary 'x' values in the time index\n",
    "summary = summary.droplevel(1, axis=0)\n",
    "summary.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "80fefa2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that each sample of 10 values starts at one sample time\n",
    "# and ends just before the next. This is not what we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "545b11fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:15</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:25</th>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:35</th>\n",
       "      <td>18.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:45</th>\n",
       "      <td>28.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:55</th>\n",
       "      <td>38.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>47.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        0     1     2     3     4     5     6     7     8  \\\n",
       "2021-12-01 06:00:15   0.0   1.0   2.0   3.0   4.0   5.0   6.0   7.0   NaN   \n",
       "2021-12-01 06:00:25   8.0   9.0  10.0  11.0  12.0  13.0  14.0  15.0  16.0   \n",
       "2021-12-01 06:00:35  18.0  19.0  20.0  21.0  22.0  23.0  24.0  25.0  26.0   \n",
       "2021-12-01 06:00:45  28.0  29.0  30.0  31.0  32.0  33.0  34.0  35.0  36.0   \n",
       "2021-12-01 06:00:55  38.0  39.0  40.0  41.0  42.0  43.0  44.0  45.0  46.0   \n",
       "\n",
       "                        9  \n",
       "2021-12-01 06:00:15   NaN  \n",
       "2021-12-01 06:00:25  17.0  \n",
       "2021-12-01 06:00:35  27.0  \n",
       "2021-12-01 06:00:45  37.0  \n",
       "2021-12-01 06:00:55  47.0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To sample values that are centered around the desired\n",
    "# sample time, we can use the offset argument\n",
    "resample_10sec = x.resample(rule='10s', offset='5s')\n",
    "keys = [k for k, v in resample_10sec]\n",
    "samples = [v for k, v in resample_10sec]\n",
    "summary = pd.concat([s.reset_index(drop=True) for s in samples], axis=1, keys=keys)\n",
    "summary = summary.droplevel(1, axis=1).T\n",
    "summary.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a4165f5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, note that each sample starts 5 seconds before each \n",
    "# desired sample time (e.g. 06:00:25) and ends 4 seconds\n",
    "# after (e.g. 06:00:34)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "94dcea1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:15</th>\n",
       "      <td>118.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>122.0</td>\n",
       "      <td>123.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>127.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:25</th>\n",
       "      <td>128.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>131.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>133.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>137.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:35</th>\n",
       "      <td>138.0</td>\n",
       "      <td>139.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>142.0</td>\n",
       "      <td>143.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>146.0</td>\n",
       "      <td>147.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:45</th>\n",
       "      <td>148.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>152.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>156.0</td>\n",
       "      <td>157.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:55</th>\n",
       "      <td>158.0</td>\n",
       "      <td>159.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         0      1      2      3      4      5      6      7  \\\n",
       "2021-12-01 06:02:15  118.0  119.0  120.0  121.0  122.0  123.0  124.0  125.0   \n",
       "2021-12-01 06:02:25  128.0  129.0  130.0  131.0  132.0  133.0  134.0  135.0   \n",
       "2021-12-01 06:02:35  138.0  139.0  140.0  141.0  142.0  143.0  144.0  145.0   \n",
       "2021-12-01 06:02:45  148.0  149.0  150.0  151.0  152.0  153.0  154.0  155.0   \n",
       "2021-12-01 06:02:55  158.0  159.0  160.0    NaN    NaN    NaN    NaN    NaN   \n",
       "\n",
       "                         8      9  \n",
       "2021-12-01 06:02:15  126.0  127.0  \n",
       "2021-12-01 06:02:25  136.0  137.0  \n",
       "2021-12-01 06:02:35  146.0  147.0  \n",
       "2021-12-01 06:02:45  156.0  157.0  \n",
       "2021-12-01 06:02:55    NaN    NaN  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f6f77dd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:25</th>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:35</th>\n",
       "      <td>18.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:45</th>\n",
       "      <td>28.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:55</th>\n",
       "      <td>38.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>47.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:05</th>\n",
       "      <td>48.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>57.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:15</th>\n",
       "      <td>58.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:25</th>\n",
       "      <td>68.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>77.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:35</th>\n",
       "      <td>78.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>87.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:45</th>\n",
       "      <td>88.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>91.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>97.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:55</th>\n",
       "      <td>98.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>105.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>107.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:05</th>\n",
       "      <td>108.0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>111.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>116.0</td>\n",
       "      <td>117.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:15</th>\n",
       "      <td>118.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>122.0</td>\n",
       "      <td>123.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>127.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:25</th>\n",
       "      <td>128.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>131.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>133.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>137.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:35</th>\n",
       "      <td>138.0</td>\n",
       "      <td>139.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>142.0</td>\n",
       "      <td>143.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>146.0</td>\n",
       "      <td>147.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:45</th>\n",
       "      <td>148.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>152.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>156.0</td>\n",
       "      <td>157.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         0      1      2      3      4      5      6      7  \\\n",
       "2021-12-01 06:00:25    8.0    9.0   10.0   11.0   12.0   13.0   14.0   15.0   \n",
       "2021-12-01 06:00:35   18.0   19.0   20.0   21.0   22.0   23.0   24.0   25.0   \n",
       "2021-12-01 06:00:45   28.0   29.0   30.0   31.0   32.0   33.0   34.0   35.0   \n",
       "2021-12-01 06:00:55   38.0   39.0   40.0   41.0   42.0   43.0   44.0   45.0   \n",
       "2021-12-01 06:01:05   48.0   49.0   50.0   51.0   52.0   53.0   54.0   55.0   \n",
       "2021-12-01 06:01:15   58.0   59.0   60.0   61.0   62.0   63.0   64.0   65.0   \n",
       "2021-12-01 06:01:25   68.0   69.0   70.0   71.0   72.0   73.0   74.0   75.0   \n",
       "2021-12-01 06:01:35   78.0   79.0   80.0   81.0   82.0   83.0   84.0   85.0   \n",
       "2021-12-01 06:01:45   88.0   89.0   90.0   91.0   92.0   93.0   94.0   95.0   \n",
       "2021-12-01 06:01:55   98.0   99.0  100.0  101.0  102.0  103.0  104.0  105.0   \n",
       "2021-12-01 06:02:05  108.0  109.0  110.0  111.0  112.0  113.0  114.0  115.0   \n",
       "2021-12-01 06:02:15  118.0  119.0  120.0  121.0  122.0  123.0  124.0  125.0   \n",
       "2021-12-01 06:02:25  128.0  129.0  130.0  131.0  132.0  133.0  134.0  135.0   \n",
       "2021-12-01 06:02:35  138.0  139.0  140.0  141.0  142.0  143.0  144.0  145.0   \n",
       "2021-12-01 06:02:45  148.0  149.0  150.0  151.0  152.0  153.0  154.0  155.0   \n",
       "\n",
       "                         8      9  \n",
       "2021-12-01 06:00:25   16.0   17.0  \n",
       "2021-12-01 06:00:35   26.0   27.0  \n",
       "2021-12-01 06:00:45   36.0   37.0  \n",
       "2021-12-01 06:00:55   46.0   47.0  \n",
       "2021-12-01 06:01:05   56.0   57.0  \n",
       "2021-12-01 06:01:15   66.0   67.0  \n",
       "2021-12-01 06:01:25   76.0   77.0  \n",
       "2021-12-01 06:01:35   86.0   87.0  \n",
       "2021-12-01 06:01:45   96.0   97.0  \n",
       "2021-12-01 06:01:55  106.0  107.0  \n",
       "2021-12-01 06:02:05  116.0  117.0  \n",
       "2021-12-01 06:02:15  126.0  127.0  \n",
       "2021-12-01 06:02:25  136.0  137.0  \n",
       "2021-12-01 06:02:35  146.0  147.0  \n",
       "2021-12-01 06:02:45  156.0  157.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop any incomplete samples (those with NaNs at the beginning\n",
    "# and end of the signal data)\n",
    "summary = summary.dropna()\n",
    "summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "10e8e75f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally adjust the index values so that they\n",
    "# represent the desired sample times\n",
    "summary = summary.set_index(summary.index + pd.Timedelta('5s'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b1bff223",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-12-01 06:00:30     12.5\n",
       "2021-12-01 06:00:40     22.5\n",
       "2021-12-01 06:00:50     32.5\n",
       "2021-12-01 06:01:00     42.5\n",
       "2021-12-01 06:01:10     52.5\n",
       "2021-12-01 06:01:20     62.5\n",
       "2021-12-01 06:01:30     72.5\n",
       "2021-12-01 06:01:40     82.5\n",
       "2021-12-01 06:01:50     92.5\n",
       "2021-12-01 06:02:00    102.5\n",
       "2021-12-01 06:02:10    112.5\n",
       "2021-12-01 06:02:20    122.5\n",
       "2021-12-01 06:02:30    132.5\n",
       "2021-12-01 06:02:40    142.5\n",
       "2021-12-01 06:02:50    152.5\n",
       "dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now calculate the median values\n",
    "x_sampled_10sec = summary.median(axis=1)\n",
    "x_sampled_10sec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f96f4f51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2021-12-01 06:01:00', '2021-12-01 06:02:00'], dtype='datetime64[ns]', freq='T')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_index_values = pd.date_range(\n",
    "    x_sampled_10sec.first_valid_index().ceil('min'),\n",
    "    x_sampled_10sec.last_valid_index().floor('min'),\n",
    "    freq='1min'\n",
    ")\n",
    "new_index_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4aca882",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-12-01 06:01:00     42.5\n",
       "2021-12-01 06:02:00    102.5\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we can resample this sequence at the lower frequency\n",
    "# we are interested in\n",
    "x_sampled_1min = x_sampled_10sec.loc[new_index_values]\n",
    "x_sampled_1min"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "505285e0",
   "metadata": {},
   "source": [
    "***Note that the median values calculated are not quite the\n",
    "same as the original sampled values at these times. This is\n",
    "because this method uses a window that is an even-number of\n",
    "samples in length, and the original value at exactly the \n",
    "desired time is considered to be a member of the second half \n",
    "of window, not the first.***\n",
    "\n",
    "However, if you choose a window size that is an odd number\n",
    "of samples (but still a factor of the desired sampling \n",
    "frequency) then this problem does not occur.\n",
    "\n",
    "For example, if down sampling to a period of 1 minute, the\n",
    "following half-window sizes will work:\n",
    "\n",
    "```python\n",
    "half_window_size=1.5  # 60 / 1.5 = 40.0, window_size = 3\n",
    "half_window_size=2.5  # 60 / 2.5 = 24.0, window_size = 5\n",
    "half_window_size=7.5  # 60 / 7.5 = 8.0, window_size = 15\n",
    "```\n",
    "\n",
    "In most cases though, (unless your data values are all unique\n",
    "as in this case), the median will be equal to the most common \n",
    "value, or close to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "92fa9a4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:00:00</th>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:00</th>\n",
       "      <td>42.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:00</th>\n",
       "      <td>102.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:03:00</th>\n",
       "      <td>159.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         x\n",
       "t                         \n",
       "2021-12-01 06:00:00    3.5\n",
       "2021-12-01 06:01:00   42.5\n",
       "2021-12-01 06:02:00  102.5\n",
       "2021-12-01 06:03:00  159.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This can be done in just a few lines if you're not concerned\n",
    "# about dropping the first and last sample which might not be\n",
    "# a complete set of values\n",
    "x_10sec_medians = x.resample(rule='10s', offset='5s').median()\n",
    "x_10sec_medians = x_10sec_medians.set_index(x_10sec_medians.index + pd.Timedelta('5s'))\n",
    "x_sampled_1min = x_10sec_medians.resample(rule='1min').agg('first')\n",
    "x_sampled_1min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7a4ecb9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is a function to do the whole operation\n",
    "\n",
    "def downsample_using_local_median(x, freq, new_freq, \n",
    "                                  half_window_size=2.5):\n",
    "    \"\"\"Samples from a dataframe x of time series at a lower\n",
    "    frequency by calculating the median value over a window\n",
    "    of points around each desired sample time.\n",
    "    \n",
    "    For this method, the desired sample period must be \n",
    "    divisible by the half the window size. For example, if \n",
    "    the original data is sampled every second, and the \n",
    "    desired sampling period is every minute, then the half-\n",
    "    window size could be 1.5, 2.5, 5, 7.5, or 10, for \n",
    "    example. Note that if the whole window size is an odd \n",
    "    number, then the window is more precicely centred\n",
    "    around the desired sample-periods.\n",
    "    \n",
    "    Arguments\n",
    "    ---------\n",
    "    x : dataframe\n",
    "        Pandas dataframe with a date-time index that has a\n",
    "        constant sample period.\n",
    "    freq : str\n",
    "        Sample period of the signals in x expressed as a \n",
    "        Pandas frequency alias (e.g. 'min' or '15s').\n",
    "    new_freq : str\n",
    "        Desired sampling period after resampling, expressed\n",
    "        as a Pandas frequency alias.\n",
    "    half_window_size : float\n",
    "        Number of samples in the calculation window \n",
    "        divided by two. E.g. for a window size of 5,\n",
    "        half_window_size = 2.5. However, half_window_size\n",
    "        must be a factor of the desired sampling period.\n",
    "        E.g. 45 seconds is a factor of 6 minutes but not\n",
    "        a factor of 5 minutes.\n",
    "    \"\"\"\n",
    "    x = pd.DataFrame(x)\n",
    "    # Check that half-window size is a factor of the\n",
    "    # desired sampling period.\n",
    "    freq = pd.Timedelta(freq)\n",
    "    new_freq = pd.Timedelta(new_freq)\n",
    "    offset = freq * half_window_size\n",
    "    assert((new_freq / offset).is_integer())\n",
    "    x_medians = x.resample(rule=offset*2, offset=offset).median()\n",
    "    x_medians = x_medians.set_index(x_medians.index + offset)\n",
    "    new_index = pd.date_range(\n",
    "        x_medians.first_valid_index().ceil(new_freq),\n",
    "        x_medians.last_valid_index().floor(new_freq),\n",
    "        freq=new_freq\n",
    "    )\n",
    "    return x_medians.loc[new_index]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "241447d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:00</th>\n",
       "      <td>42.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:00</th>\n",
       "      <td>102.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:03:00</th>\n",
       "      <td>159.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         x\n",
       "t                         \n",
       "2021-12-01 06:01:00   42.5\n",
       "2021-12-01 06:02:00  102.5\n",
       "2021-12-01 06:03:00  159.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "downsample_using_local_median(x, '1sec', '1min', half_window_size=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "94819706",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:01:00</th>\n",
       "      <td>43.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-12-01 06:02:00</th>\n",
       "      <td>103.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         x\n",
       "t                         \n",
       "2021-12-01 06:01:00   43.0\n",
       "2021-12-01 06:02:00  103.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sample using a window with an odd number of samples\n",
    "downsample_using_local_median(x, '1sec', '1min', half_window_size=2.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decc886b",
   "metadata": {},
   "source": [
    "## Test function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3f4f3014",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-01-01 00:00:56    100\n",
       "2021-01-01 00:00:57    101\n",
       "2021-01-01 00:00:58    102\n",
       "2021-01-01 00:00:59    103\n",
       "2021-01-01 00:01:00    104\n",
       "2021-01-01 00:01:01    105\n",
       "2021-01-01 00:01:02    106\n",
       "2021-01-01 00:01:03    107\n",
       "2021-01-01 00:01:04    108\n",
       "2021-01-01 00:01:05    109\n",
       "2021-01-01 00:01:06    110\n",
       "Freq: S, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test data 1\n",
    "y = pd.Series(\n",
    "    [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110],\n",
    "    index=pd.date_range(\"2021-01-01 00:00:56\", \"2021-01-01 00:01:06\", freq='S')\n",
    ")\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "96fa256a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(104, 103.5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Result should be one of these values\n",
    "y_target = y.loc[\"2021-01-01 00:01:00\"]\n",
    "y_target_off = y_target - 0.5\n",
    "y_target, y_target_off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fdfffa87",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=1.5)\n",
    "assert(y_sampled.values.item() == y_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8ea641f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=2.5)\n",
    "assert(y_sampled.values.item() == y_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "943ce3a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=3)\n",
    "assert(y_sampled.values.item() == y_target_off)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "110319e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# When window is too big returns median of all values\n",
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=7.5)\n",
    "assert(y_sampled.values.item() == y.median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b230304b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2021-01-01 00:00:56    100.0\n",
       "2021-01-01 00:00:57    101.0\n",
       "2021-01-01 00:00:58    102.0\n",
       "2021-01-01 00:00:59    103.0\n",
       "2021-01-01 00:01:00    104.0\n",
       "2021-01-01 00:01:01    105.0\n",
       "2021-01-01 00:01:02      NaN\n",
       "2021-01-01 00:01:03    107.0\n",
       "2021-01-01 00:01:04    108.0\n",
       "2021-01-01 00:01:05    109.0\n",
       "2021-01-01 00:01:06    110.0\n",
       "Freq: S, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if y has nan values\n",
    "y.loc['2021-01-01 00:01:02'] = np.nan\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "004a88b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Window of size 3 samples should not be affected\n",
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=1.5)\n",
    "assert(y_sampled.values.item() == y_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "40debe7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Window of size 5 samples\n",
    "y_sampled = downsample_using_local_median(y, '1sec', '1min', half_window_size=2.5)\n",
    "assert(y_sampled.values.item() == np.nanmedian([102, 103, 104, 105, np.nan]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47cd5ab8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261b1d1c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1451a7da",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:torch]",
   "language": "python",
   "name": "conda-env-torch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}