Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A python 3 api for ODB2 data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ODB2 is a format for encoding tabular observation data. It is formed of a sequence of Frames. Each Frame consists of a header followed by a compressed binary format for encoded data. An ODB2 file may contain any number of frames which may or may not be *compatible* with each other (that is, having the same columns and data types).\n",
    "\n",
    "The ODB-API package was the supported package for encoding and decoding ODB2 data at ECMWF. This package will shortly be replaced by the newly developed ODC package, and will soon be deprecated.\n",
    "\n",
    "The ODB-API package contained two different python APIs. These did slightly different things:\n",
    "\n",
    "  * One thinly wrapped the C/Fortran API in ODB-API to present a row-based view of ODB2 data.\n",
    "  * One presented an sqlite-like interface to interrogate the contents of ODB2 files\n",
    "\n",
    "Neither of these two APIs was especially satisfactory. They were slow, buggy, and inconsistent and they had very poor compatibility with existing python tooling for handling data sets.\n",
    "\n",
    "Both of these APIs have been deprecated, and in the ODC package they have been removed. We present here a first attempt at a new, much simplified, python API for handling ODB2 data. This has a number of properties\n",
    "\n",
    "  * It is a pure python implementation - the `pyodc` module\n",
    "  * In the same manner as `pickle`/`cpickle`, there is a `codc` module. This module comes with *much* better performance.\n",
    "  * It is designed to interact with the `pandas` package. Data is encoded to, and decoded from pandas `DataFrame` objects.\n",
    "  * It is a thin decoder and encoder. We have no functionality for SQL-like queries.\n",
    "  * There is an API for exploring the structure and contents of an ODB2 file without decoding it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preconfiguration of notebook\n",
    "\n",
    "To use this notebook you need only to have a cloned copy of the repo it contains and to run the following cell.\n",
    "\n",
    "If you wish to use the `codc` module, you need to start `ipython notebook` with `libodccore.so` in the `LD_PRELOAD` path, and swap the import statements over in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime as datetime\n",
    "from itertools import islice, cycle\n",
    "import sys\n",
    "import os\n",
    "sys.path.insert(0, os.path.abspath(''))\n",
    "import pyodc as odc\n",
    "#import codc as odc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring ODB2 data\n",
    "\n",
    "We anticipate that almost all of the use of the pyodc module will be for encoding and decoding ODB2 data. The module also contains a number of classes that facilitate the exploration of the ODB2 data. In particular the `Reader` class provides the access point to further details.\n",
    "\n",
    "We are not going to elaborate on the use of that class in this document, unless there turns out to be sufficient demand for it. However, we make use of elements of it to demonstrate the functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple encoding and decoding ODB2 data\n",
    "\n",
    "The encoding API is designed to be as straightforward as possible. Given a pandas `DataFrame` the data should simply be encoded. We provide here a very simple, cut-down, example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "3   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "4   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "5   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "6   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "7   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "8   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "9   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "3           1234.0           12.34  \n",
      "4           4321.0           43.21  \n",
      "5              NaN             NaN  \n",
      "6           1234.0           12.34  \n",
      "7           4321.0           43.21  \n",
      "8              NaN             NaN  \n",
      "9           1234.0           12.34  \n"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame({\n",
    "    'expver': ['0001'] * 10,\n",
    "    'date@hdr': [int(datetime.datetime.now().strftime(\"%Y%m%d\"))] * 10,\n",
    "    'statid@hdr': ['stat{:02d}'.format(x) for x in range(10)],\n",
    "    'wigos@hdr': ['0-12345-0-678{:02d}'.format(x+90) for x in range(10)],\n",
    "    'obsvalue@body': [12.3456 * x for x in range(10)],\n",
    "    'integer_missing': list(islice(cycle([1234, 4321, None]), 10)),\n",
    "    'double_missing': list(islice(cycle([12.34, 43.21, None]), 10)),\n",
    "})\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "odc.encode_odb(df, 'example-file1.odb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see that this data has been correctly encoded by decoding it again directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "3   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "4   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "5   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "6   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "7   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "8   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "9   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "3           1234.0           12.34  \n",
      "4           4321.0           43.21  \n",
      "5              NaN             NaN  \n",
      "6           1234.0           12.34  \n",
      "7           4321.0           43.21  \n",
      "8              NaN             NaN  \n",
      "9           1234.0           12.34  \n"
     ]
    }
   ],
   "source": [
    "df_decoded = odc.read_odb('example-file1.odb', single=True)\n",
    "print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the encoding and decoding of ODB2 data work on file-like objects as well as on files. These have the advantage that you can encode multiple frames of data into the same file sequentially.\n",
    "\n",
    "In this case, we create an ODB file with frames of two different structures to demonstrate what can be done as a result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame({\n",
    "    'expver': ['0002'] * 10,\n",
    "    'date@hdr': [int(datetime.datetime.now().strftime(\"%Y%m%d\"))] * 10,\n",
    "    'statid@hdr': ['stat{:02d}'.format(20-x) for x in range(10)],\n",
    "    'obsvalue@body': [12.3456 * x for x in range(10)],\n",
    "})\n",
    "\n",
    "with open('example-file2.odb', 'wb') as f:\n",
    "    odc.encode_odb(df, f)\n",
    "    odc.encode_odb(df2, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The trivial decoder will now result in a `DataFrame` with a substantial number of missing values. In a later section we will see how to extract these different sections separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0    0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1    0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2    0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "3    0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "4    0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "5    0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "6    0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "7    0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "8    0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "9    0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "10   0002  20200602     stat20             None         0.0000   \n",
      "11   0002  20200602     stat19             None        12.3456   \n",
      "12   0002  20200602     stat18             None        24.6912   \n",
      "13   0002  20200602     stat17             None        37.0368   \n",
      "14   0002  20200602     stat16             None        49.3824   \n",
      "15   0002  20200602     stat15             None        61.7280   \n",
      "16   0002  20200602     stat14             None        74.0736   \n",
      "17   0002  20200602     stat13             None        86.4192   \n",
      "18   0002  20200602     stat12             None        98.7648   \n",
      "19   0002  20200602     stat11             None       111.1104   \n",
      "\n",
      "    integer_missing  double_missing  \n",
      "0            1234.0           12.34  \n",
      "1            4321.0           43.21  \n",
      "2               NaN             NaN  \n",
      "3            1234.0           12.34  \n",
      "4            4321.0           43.21  \n",
      "5               NaN             NaN  \n",
      "6            1234.0           12.34  \n",
      "7            4321.0           43.21  \n",
      "8               NaN             NaN  \n",
      "9            1234.0           12.34  \n",
      "10              NaN             NaN  \n",
      "11              NaN             NaN  \n",
      "12              NaN             NaN  \n",
      "13              NaN             NaN  \n",
      "14              NaN             NaN  \n",
      "15              NaN             NaN  \n",
      "16              NaN             NaN  \n",
      "17              NaN             NaN  \n",
      "18              NaN             NaN  \n",
      "19              NaN             NaN  \n"
     ]
    }
   ],
   "source": [
    "with open('example-file2.odb', 'rb') as f:\n",
    "    df_decoded = odc.read_odb(f, single=True)\n",
    "print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the details of encoding\n",
    "\n",
    "For most operational ODB2 data, the encoding used is a lossy. In particular, we encode most values as 4-byte REAL values rather than 8-byte DOUBLES.\n",
    "\n",
    "Typically the encoder will automatically select a datatype and corresponding encoder to use. This datatype can be overridden by supplying a types dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "odc.encode_odb(df, 'example-file3.odb', types={'obsvalue@body': odc.REAL})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see on interrogating the frame headers that the data type has changed for the newly encoded file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "r1 = odc.Reader('example-file1.odb', aggregated=False)\n",
    "r3 = odc.Reader('example-file3.odb', aggregated=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original: DataType.DOUBLE\n",
      "updated:  DataType.REAL\n"
     ]
    }
   ],
   "source": [
    "print(\"original:\", r1.frames[0].column_dict['obsvalue@body'].dtype)\n",
    "print(\"updated: \", r3.frames[0].column_dict['obsvalue@body'].dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And when we decode the data we can see that the precision of the data has been appropriately reduced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890       0.000000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891      12.345600   \n",
      "2   0001  20200602     stat02  0-12345-0-67892      24.691200   \n",
      "3   0001  20200602     stat03  0-12345-0-67893      37.036800   \n",
      "4   0001  20200602     stat04  0-12345-0-67894      49.382401   \n",
      "5   0001  20200602     stat05  0-12345-0-67895      61.728001   \n",
      "6   0001  20200602     stat06  0-12345-0-67896      74.073601   \n",
      "7   0001  20200602     stat07  0-12345-0-67897      86.419197   \n",
      "8   0001  20200602     stat08  0-12345-0-67898      98.764801   \n",
      "9   0001  20200602     stat09  0-12345-0-67899     111.110397   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "3           1234.0           12.34  \n",
      "4           4321.0           43.21  \n",
      "5              NaN             NaN  \n",
      "6           1234.0           12.34  \n",
      "7           4321.0           43.21  \n",
      "8              NaN             NaN  \n",
      "9           1234.0           12.34  \n"
     ]
    }
   ],
   "source": [
    "df_decoded = odc.read_odb('example-file3.odb', single=True)\n",
    "print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the frame structure\n",
    "\n",
    "ODB2 data is broken down into frames. By default a maximum of 10000 rows of data will be encoded into each frame. If more than 10000 rows are supplied then the data will be split into a sequence of frames with at maximum 10000 rows.\n",
    "\n",
    "This threshold can be modified by use of the `rows_per_frame` parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "odc.encode_odb(df, 'example-file4.odb', rows_per_frame=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examining the structure of this frame clearly shows that the file now contains multiple frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original frames: [<pyodc.frame.Frame object at 0x7f447a047d60>]\n",
      "updated  frames: [<pyodc.frame.Frame object at 0x7f4479fcba30>, <pyodc.frame.Frame object at 0x7f4479fcbd90>, <pyodc.frame.Frame object at 0x7f4479fcba60>, <pyodc.frame.Frame object at 0x7f447a047790>]\n",
      "original row counts: [10]\n",
      "updated  row counts: [3, 3, 3, 1]\n"
     ]
    }
   ],
   "source": [
    "r4 = odc.Reader('example-file4.odb', aggregated=False)\n",
    "\n",
    "print('original frames:', r1.frames)\n",
    "print('updated  frames:', r4.frames)\n",
    "\n",
    "print('original row counts:', [f.nrows for f in r1.frames])\n",
    "print('updated  row counts:', [f.nrows for f in r4.frames])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite these differences, the data is the same if decoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "3   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "4   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "5   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "6   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "7   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "8   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "9   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "3           1234.0           12.34  \n",
      "4           4321.0           43.21  \n",
      "5              NaN             NaN  \n",
      "6           1234.0           12.34  \n",
      "7           4321.0           43.21  \n",
      "8              NaN             NaN  \n",
      "9           1234.0           12.34  \n"
     ]
    }
   ],
   "source": [
    "df_decoded = odc.read_odb('example-file4.odb', single=True)\n",
    "print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoding a subset of the data\n",
    "\n",
    "Especially for large ODB2 files it can be very valuable to not decode all of the data. The decode functions accept a list or tuple specifying the `columns` to decode.\n",
    "\n",
    "This is especially helpful when the structure of ODB2 frames in a file is not constant, but all of the frames supply the data that is desired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   statid@hdr  obsvalue@body\n",
      "0      stat00         0.0000\n",
      "1      stat01        12.3456\n",
      "2      stat02        24.6912\n",
      "3      stat03        37.0368\n",
      "4      stat04        49.3824\n",
      "5      stat05        61.7280\n",
      "6      stat06        74.0736\n",
      "7      stat07        86.4192\n",
      "8      stat08        98.7648\n",
      "9      stat09       111.1104\n",
      "10     stat20         0.0000\n",
      "11     stat19        12.3456\n",
      "12     stat18        24.6912\n",
      "13     stat17        37.0368\n",
      "14     stat16        49.3824\n",
      "15     stat15        61.7280\n",
      "16     stat14        74.0736\n",
      "17     stat13        86.4192\n",
      "18     stat12        98.7648\n",
      "19     stat11       111.1104\n"
     ]
    }
   ],
   "source": [
    "df_decoded = odc.read_odb(\"example-file2.odb\", single=True, columns=('statid@hdr', 'obsvalue@body'))\n",
    "print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoding a sequence of frames\n",
    "\n",
    "If an ODB file is extremely large, it is undesirable to attempt to decode it into memory in its entirety. Further, if the frames within the file are not *compatible* it may be better to consider each of the frames separately.\n",
    "\n",
    "By default the `read_odb` function returns an iterable sequence that lazily decodes ODB2 frames as they are needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decoded data frame: 0\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "3   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "4   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "5   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "6   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "7   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "8   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "9   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "3           1234.0           12.34  \n",
      "4           4321.0           43.21  \n",
      "5              NaN             NaN  \n",
      "6           1234.0           12.34  \n",
      "7           4321.0           43.21  \n",
      "8              NaN             NaN  \n",
      "9           1234.0           12.34  \n",
      "\n",
      "Decoded data frame: 1\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat20         0.0000\n",
      "1   0002  20200602     stat19        12.3456\n",
      "2   0002  20200602     stat18        24.6912\n",
      "3   0002  20200602     stat17        37.0368\n",
      "4   0002  20200602     stat16        49.3824\n",
      "5   0002  20200602     stat15        61.7280\n",
      "6   0002  20200602     stat14        74.0736\n",
      "7   0002  20200602     stat13        86.4192\n",
      "8   0002  20200602     stat12        98.7648\n",
      "9   0002  20200602     stat11       111.1104\n"
     ]
    }
   ],
   "source": [
    "for idx, df_decoded in enumerate(odc.read_odb('example-file2.odb')):\n",
    "    if idx > 0: print()\n",
    "    print(\"Decoded data frame:\", idx)\n",
    "    print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregated or non-aggregated reading\n",
    "\n",
    "There are two different reasons that decoding to a sequence of dataframes may be useful.\n",
    "\n",
    "1. To page data through memory without consuming more resources than exist\n",
    "2. To handle a sequence of frames that do not have the same structure.\n",
    "\n",
    "Conceptually in the first case, a sequence of frames may be considered to be one frame that has been split for technical reasons. The library is able to logically group these frames together into one logical, aggregated frame (and, indeed, it does this by default). Decoding aggregated logical frames in one step significantly improves performance of the decoder if offloading to ODC.\n",
    "\n",
    "Note that frames do not have to have columns in the same *order* to be considered compatible.\n",
    "\n",
    "Both the `Reader` and `read_odb` functionality take two arguments:\n",
    "\n",
    "* `aggregated` - (default True) enables or disables aggregation of compatible frames.\n",
    "* `max_aggregated` - (default None) sets a maximum number of rows to be combined into one logical frame before the library will split them anyway (for pagination purposes).\n",
    "\n",
    "To demonstrate, first we build a decoder with several real and a smaller number of logical frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('example-file5.odb', 'wb') as f:\n",
    "    odc.encode_odb(df, f, rows_per_frame=3)\n",
    "    odc.encode_odb(df2, f, rows_per_frame=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can interrogate the structure using two different readers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aggregated row counts: [10, 10]\n",
      "separate   row counts: [3, 3, 3, 1, 3, 3, 3, 1]\n"
     ]
    }
   ],
   "source": [
    "r5a = odc.Reader('example-file5.odb')\n",
    "r5b = odc.Reader('example-file5.odb', aggregated=False)\n",
    "\n",
    "print('aggregated row counts:', [f.nrows for f in r5a.frames])\n",
    "print('separate   row counts:', [f.nrows for f in r5b.frames])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default we decode data in an aggregated fashion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decoded data frame: 0\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "0   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "1   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "2   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "0   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "1   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "2   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "0   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "0           1234.0           12.34  \n",
      "\n",
      "Decoded data frame: 1\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat20         0.0000\n",
      "1   0002  20200602     stat19        12.3456\n",
      "2   0002  20200602     stat18        24.6912\n",
      "0   0002  20200602     stat17        37.0368\n",
      "1   0002  20200602     stat16        49.3824\n",
      "2   0002  20200602     stat15        61.7280\n",
      "0   0002  20200602     stat14        74.0736\n",
      "1   0002  20200602     stat13        86.4192\n",
      "2   0002  20200602     stat12        98.7648\n",
      "0   0002  20200602     stat11       111.1104\n"
     ]
    }
   ],
   "source": [
    "for idx, df_decoded in enumerate(odc.read_odb('example-file5.odb')):\n",
    "    if idx > 0: print()\n",
    "    print(\"Decoded data frame:\", idx)\n",
    "    print(df_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can also decode the real frames separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decoded data frame: 0\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat00  0-12345-0-67890         0.0000   \n",
      "1   0001  20200602     stat01  0-12345-0-67891        12.3456   \n",
      "2   0001  20200602     stat02  0-12345-0-67892        24.6912   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "\n",
      "Decoded data frame: 1\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat03  0-12345-0-67893        37.0368   \n",
      "1   0001  20200602     stat04  0-12345-0-67894        49.3824   \n",
      "2   0001  20200602     stat05  0-12345-0-67895        61.7280   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "\n",
      "Decoded data frame: 2\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat06  0-12345-0-67896        74.0736   \n",
      "1   0001  20200602     stat07  0-12345-0-67897        86.4192   \n",
      "2   0001  20200602     stat08  0-12345-0-67898        98.7648   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0           1234.0           12.34  \n",
      "1           4321.0           43.21  \n",
      "2              NaN             NaN  \n",
      "\n",
      "Decoded data frame: 3\n",
      "  expver  date@hdr statid@hdr        wigos@hdr  obsvalue@body  \\\n",
      "0   0001  20200602     stat09  0-12345-0-67899       111.1104   \n",
      "\n",
      "   integer_missing  double_missing  \n",
      "0             1234           12.34  \n",
      "\n",
      "Decoded data frame: 4\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat20         0.0000\n",
      "1   0002  20200602     stat19        12.3456\n",
      "2   0002  20200602     stat18        24.6912\n",
      "\n",
      "Decoded data frame: 5\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat17        37.0368\n",
      "1   0002  20200602     stat16        49.3824\n",
      "2   0002  20200602     stat15        61.7280\n",
      "\n",
      "Decoded data frame: 6\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat14        74.0736\n",
      "1   0002  20200602     stat13        86.4192\n",
      "2   0002  20200602     stat12        98.7648\n",
      "\n",
      "Decoded data frame: 7\n",
      "  expver  date@hdr statid@hdr  obsvalue@body\n",
      "0   0002  20200602     stat11       111.1104\n"
     ]
    }
   ],
   "source": [
    "for idx, df_decoded in enumerate(odc.read_odb('example-file5.odb', aggregated=False)):\n",
    "    if idx > 0: print()\n",
    "    print(\"Decoded data frame:\", idx)\n",
    "    print(df_decoded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}