1-Basic-Datatypes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Basic Datatypes in HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Objectives:\n",
    "> * How to create (and read) HDF5 files with datasets of homogeneous and heterogenous datatypes\n",
    "> * See how h5py and PyTables achieves the same thing with their own APIs\n",
    "> * Be introduced to the `IsDescription` class in PyTables for declaring tables (instead of NumPy dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Homogeneous datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "arr_to_store = np.arange(10, dtype=np.int8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f = h5py.File(\"homogeneous1.h5\", \"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 dataset \"mydata\": shape (10,), type \"|i1\">"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.create_dataset(data=arr_to_store, name=\"mydata\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int8)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f['/mydata'][:]    # data can be accessed in a NumPy-like interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 file \"homogeneous1.h5\" (mode r+)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"homogeneous1.h5\" with sec2 driver.\r\n",
      "/                        Group\r\n",
      "    Location:  1:96\r\n",
      "    Links:     1\r\n",
      "/mydata                  Dataset {10/10}\r\n",
      "    Location:  1:800\r\n",
      "    Links:     1\r\n",
      "    Storage:   10 logical bytes, 10 allocated bytes, 100.00% utilization\r\n",
      "    Type:      native signed char\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -rv homogeneous1.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 faltet  staff  2154 May 18 10:04 homogeneous1.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l homogeneous1.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using PyTables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f2 = tables.open_file(\"homogeneous2.h5\", \"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "/mydata (Array(10,)) ''\n",
       "  atom := Int8Atom(shape=(), dflt=0)\n",
       "  maindim := 0\n",
       "  flavor := 'numpy'\n",
       "  byteorder := 'irrelevant'\n",
       "  chunkshape := None"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f2.create_array(f2.root, name=\"mydata\", obj=arr_to_store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int8)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f2.root.mydata[:]  # data can be accessed in a NumPy-like interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "File(filename=homogeneous2.h5, title='', mode='w', root_uep='/', filters=Filters(complevel=0, shuffle=False, bitshuffle=False, fletcher32=False, least_significant_digit=None))\n",
       "/ (RootGroup) ''\n",
       "/mydata (Array(10,)) ''\n",
       "  atom := Int8Atom(shape=(), dflt=0)\n",
       "  maindim := 0\n",
       "  flavor := 'numpy'\n",
       "  byteorder := 'irrelevant'\n",
       "  chunkshape := None"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f2.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"homogeneous2.h5\" with sec2 driver.\r\n",
      "mydata                   Dataset {10/10}\r\n",
      "    Attribute: CLASS scalar\r\n",
      "        Type:      5-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"ARRAY\"\r\n",
      "    Attribute: FLAVOR scalar\r\n",
      "        Type:      5-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"numpy\"\r\n",
      "    Attribute: TITLE null\r\n",
      "        Type:      1-byte null-terminated UTF-8 string\r\n",
      "\r\n",
      "    Attribute: VERSION scalar\r\n",
      "        Type:      3-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"2.4\"\r\n",
      "    Location:  1:1024\r\n",
      "    Links:     1\r\n",
      "    Storage:   10 logical bytes, 10 allocated bytes, 100.00% utilization\r\n",
      "    Type:      native signed char\r\n",
      "H5tools-DIAG: Error detected in HDF5:tools (1.8.17) thread 140736151184320:\r\n",
      "  #000: h5tools_dump.c line 1836 in h5tools_dump_mem(): H5Sis_simple failed\r\n",
      "    major: Failure in tools library\r\n",
      "    minor: error in function\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -v homogeneous2.h5/mydata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 faltet  staff  2154 May 18 10:04 homogeneous1.h5\r\n",
      "-rw-r--r--  1 faltet  staff  2154 May 18 10:04 homogeneous2.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l homogeneous*.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Compound Datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dtype = np.dtype([(\"myfield1\", np.int32), (\"myfield2\", np.float64), (\"myfield3\", \"S4\")])\n",
    "table_to_store = np.fromiter(((i, i**2, \"%s\"%i**3) for i in range(10)), dtype=dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0,   0., b'0'), (1,   1., b'1'), (2,   4., b'8'), (3,   9., b'27'),\n",
       "       (4,  16., b'64'), (5,  25., b'125'), (6,  36., b'216'),\n",
       "       (7,  49., b'343'), (8,  64., b'512'), (9,  81., b'729')], \n",
       "      dtype=[('myfield1', '<i4'), ('myfield2', '<f8'), ('myfield3', 'S4')])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_to_store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f = h5py.File(\"compound1.h5\", \"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 dataset \"mydata\": shape (10,), type \"|V16\">"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.create_dataset(data=table_to_store, name=\"mydata\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype([('myfield1', '<i4'), ('myfield2', '<f8'), ('myfield3', 'S4')])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f['mydata'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0,   0., b'0'), (1,   1., b'1'), (2,   4., b'8'), (3,   9., b'27'),\n",
       "       (4,  16., b'64'), (5,  25., b'125'), (6,  36., b'216'),\n",
       "       (7,  49., b'343'), (8,  64., b'512'), (9,  81., b'729')], \n",
       "      dtype=[('myfield1', '<i4'), ('myfield2', '<f8'), ('myfield3', 'S4')])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f['mydata'][:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"compound1.h5\" with sec2 driver.\r\n",
      "mydata                   Dataset {10/10}\r\n",
      "    Location:  1:800\r\n",
      "    Links:     1\r\n",
      "    Storage:   160 logical bytes, 160 allocated bytes, 100.00% utilization\r\n",
      "    Type:      struct {\r\n",
      "                   \"myfield1\"         +0    native int\r\n",
      "                   \"myfield2\"         +4    native double\r\n",
      "                   \"myfield3\"         +12   4-byte null-padded ASCII string\r\n",
      "               } 16 bytes\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -v compound1.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using PyTables (simple way)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f2 = tables.open_file(\"compound2.h5\", \"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "/mydata (Table(10,)) ''\n",
       "  description := {\n",
       "  \"myfield1\": Int32Col(shape=(), dflt=0, pos=0),\n",
       "  \"myfield2\": Float64Col(shape=(), dflt=0.0, pos=1),\n",
       "  \"myfield3\": StringCol(itemsize=4, shape=(), dflt=b'', pos=2)}\n",
       "  byteorder := 'little'\n",
       "  chunkshape := (4096,)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f2.create_table(f2.root, name=\"mydata\", obj=table_to_store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using PyTables (description way)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class MyTable(tables.IsDescription):\n",
    "    myfield1 = tables.Int32Col()\n",
    "    myfield2 = tables.Float64Col()\n",
    "    myfield3 = tables.StringCol(itemsize=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f3 = tables.open_file(\"compound3.h5\", \"w\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "t = f3.create_table(f3.root, \"mydata\", MyTable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "t.append(table_to_store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "f3.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"compound2.h5\" with sec2 driver.\r\n",
      "mydata                   Dataset {10/Inf}\r\n",
      "    Attribute: CLASS scalar\r\n",
      "        Type:      5-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"TABLE\"\r\n",
      "    Attribute: FIELD_0_FILL scalar\r\n",
      "        Type:      native int\r\n",
      "        Data:  0\r\n",
      "    Attribute: FIELD_0_NAME scalar\r\n",
      "        Type:      8-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"myfield1\"\r\n",
      "    Attribute: FIELD_1_FILL scalar\r\n",
      "        Type:      native double\r\n",
      "        Data:  0\r\n",
      "    Attribute: FIELD_1_NAME scalar\r\n",
      "        Type:      8-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"myfield2\"\r\n",
      "    Attribute: FIELD_2_FILL scalar\r\n",
      "        Type:      1-byte null-terminated ASCII string\r\n",
      "        Data:  \"\"\r\n",
      "    Attribute: FIELD_2_NAME scalar\r\n",
      "        Type:      8-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"myfield3\"\r\n",
      "    Attribute: NROWS scalar\r\n",
      "        Type:      native long\r\n",
      "        Data:  10\r\n",
      "    Attribute: TITLE null\r\n",
      "        Type:      1-byte null-terminated UTF-8 string\r\n",
      "\r\n",
      "    Attribute: VERSION scalar\r\n",
      "        Type:      3-byte null-terminated UTF-8 string\r\n",
      "        Data:  \"2.7\"\r\n",
      "    Location:  1:1024\r\n",
      "    Links:     1\r\n",
      "    Chunks:    {4096} 65536 bytes\r\n",
      "    Storage:   160 logical bytes, 65536 allocated bytes, 0.24% utilization\r\n",
      "    Type:      struct {\r\n",
      "                   \"myfield1\"         +0    native int\r\n",
      "                   \"myfield2\"         +4    native double\r\n",
      "                   \"myfield3\"         +12   4-byte null-terminated ASCII string\r\n",
      "               } 16 bytes\r\n",
      "H5tools-DIAG: Error detected in HDF5:tools (1.8.17) thread 140736151184320:\r\n",
      "  #000: h5tools_dump.c line 1836 in h5tools_dump_mem(): H5Sis_simple failed\r\n",
      "    major: Failure in tools library\r\n",
      "    minor: error in function\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -v compound2.h5/mydata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 faltet  staff   2.3K May 18 10:04 compound1.h5\r\n",
      "-rw-r--r--  1 faltet  staff    68K May 18 10:04 compound2.h5\r\n",
      "-rw-r--r--  1 faltet  staff    68K May 18 10:04 compound3.h5\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh compound*.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Hmm, it seems like PyTables files are larger than h5py ones, why?  Let's introspect a bit into the files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mydata                   Dataset {10}\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls compound1.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mydata                   Dataset {10/Inf}\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls compound2.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We see that the dimensionality of the table created with PyTables is `{10/Inf}`, indicating that the dataset is chunked, whereas the one created with h5py is just `{10}`, which means that it is not using chunking.  As chunked datasets take more space than non-chunked ones, this is why PyTables are larger.\n",
    "\n",
    "The reason why PyTables tables are chunked by default is that they can be enlarged and compressed, and chunking is required in order to allow that.  More on chunking later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}