Loading 4DSTEM datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading 4DSTEM datasets from ePSIC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the installation of the DE16 camera and the new Freescan arbitrary scan generator, there are now a number of different ways of collecting 4DSTEM data at ePSIC. This notebook demonstrates how to load all of the different types of data that you might encounter into the Hyperspy/pyxem ecosystem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MerlinEM data acquired with the JEOL scan generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Opening MerlinEM data acquired with the JEOL scan generator is straightforward as a data loader has been included in pyxem. The rows of the data are determined by picking out the flyback pixels from their longer exposure time. Data is automatically loaded lazily (that is, it is not loaded into RAM)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyxem as pxm\n",
    "import hyperspy.api as hs\n",
    "import numpy as np\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_jeol = pxm.load_mib('/dls/e02/data/2021/cm28158-1/Merlin/20210118_152746_data.mib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_jeol)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typically, the data has also been converted to HDF5 format using ePSIC's conversion scripts (which uses pyxem's data loader). The converted HDF5 files can be loaded using Hyperspy, either lazily or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_jeol_conv = hs.load('/dls/e02/data/2021/cm28158-1/processing/Merlin/20210118 145609/20210118_145609.hdf5',lazy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_jeol_conv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MerlinEM data acquired with the Freescan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading Freescan data is slightly more difficult due to the arbitrary nature of the scan. For regular rectangular scans, there remains straightforward approaches. The current version of pyxem (0.13.0) will read the mib file as TEM data because the Freescan doesn't include any flyback pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_freescan = pxm.load_mib('/dls/e02/data/2021/cm28158-1/Merlin/Freescan_cal/20210119_Freescan_600us_256X256_2.mib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_freescan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once loaded, it's then possible to reshape the data to the correct shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_freescan_reshaped = hs.signals.Signal2D(merlin_freescan.data.reshape((256,256,515,515)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then convert the data back into the electron diffraction datatype for pyxem to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_freescan_reshaped.set_signal_type('electron_diffraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have also modified the pyxem loader to input the data shape and load in the mib files to the correct shape. This will hopefully go into a later version of pyxem but feel free to ask for the code currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_freescan_mod = pxm.load_mib('/dls/e02/data/2021/cm28158-1/Merlin/Freescan_cal/20210119_Freescan_600us_256X256_2.mib', scan=(256,256))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_freescan_mod)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading non rectangular scans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading non-rectangular scans is more difficult, due to the arbitrary nature of the scan positions. The initial mib files can be loaded as usual, but will be assigned as TEM data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_arbitrary = pxm.load_mib('/dls/e02/data/2021/cm28158-1/Merlin/Freescan_cal/20210119_Freescan_600us_256_256_subsamples2X_2.mib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_arbitrary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is then necessary to import the scan positions from the xyz file, this can be done using the following function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def import_scan_pos(filename):\n",
    "    positions = []\n",
    "    import csv\n",
    "    with open(filename, newline='') as csvfile:\n",
    "        reader = csv.reader(csvfile, delimiter=',')\n",
    "        \n",
    "        #Skip lines\n",
    "        for i in range(4):\n",
    "            next(reader)\n",
    "\n",
    "        for row in reader:\n",
    "            positions.append((int(row[0]),int(row[1])))\n",
    "    return(positions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scan_positions = import_scan_pos('/dls/e02/data/2021/cm28158-1/DE16/Scan Coordinates XYZ Files/0256x0256/Subsampled_02x/0256x0256_Subsampled_02x_00001.xyz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scan_positions{0:10})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the scan positions are imported, it's then possible to reshape the data using the scan positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reshape_arbitrary_scan(data, scan_positions):\n",
    "    import numpy as np\n",
    "    \n",
    "    x_positions = []\n",
    "    y_positions = []\n",
    "    for pos in scan_positions:\n",
    "        x_positions.append(pos[0])\n",
    "        y_positions.append(pos[1])\n",
    "    x_max = max(x_positions)\n",
    "    y_max = max(y_positions)\n",
    "    \n",
    "    data_np = np.zeros((x_max,y_max,data.axes_manager[1].size,data.axes_manager[2].size))\n",
    "    \n",
    "    for i,pos in enumerate(scan_positions[:-1]):\n",
    "        data_np[pos[0]-1,pos[1]-1,:,:] = data.inav[i].data\n",
    "    \n",
    "    data_out = hs.signals.Signal2D(data_np)\n",
    "    \n",
    "    return(data_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_arbitrary.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merlin_arbitrary_reshaped = reshape_arbitrary_scan(merlin_arbitrary, scan_positions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(merlin_arbitrary_reshaped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DE16 acquired data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data from the DE16 comes in a variety of file formats. The raw data from the Streampix software is in .seq format. These files do not include any dark or gain correction but can be loaded using pims (note: get pims for epsic3.7?)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = pims.open('/dls/e02/data/2020/mg25140-9/DE_test_data/16-47-35.930.seq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pims_to_hs(images):\n",
    "    np_data = np.zeros((len(images),images.frame_shape[0],images.frame_shape[1]))\n",
    "    for i,image in enumerate(images):\n",
    "        np_data[i,:,:] = image\n",
    "    return(hs.signals.Signal2D(np_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_raw = pims_to_hs(images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(de_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the data is loaded, you can then reshape as shown for the Merlin data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_raw_reshaped = hs.signals.Signal2D(de_raw.data.reshape((100,100,256,256)))\n",
    "de_raw_reshaped.set_signal_type('electron_diffraction')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(de_raw_reshaped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing non-raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Direct Electron have their own software in which you can convert seq files to other formats. This software also performs dark/gain reference correction (with proprietary algorithms). There are a number of file formats to export to. The following code shows how to import the HDF5 files that have been reshaped to the correct dimensions. HDF5 files can only be exported for rectangular arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = h5py.File('/dls/e02/data/2021/cm28158-1/DE16/256X256 test.h5', 'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_hdf = np.array(f['4DSTEM_experiment']['data']['datacubes']['datacubes_0']['data'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_hdf_conv = hs.signals.Signal2D(de_hdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(de_hdf_conv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_hdf_conv.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of non-rectangular scan arrays, data can be saved in to one of the other formats. The MRC format does not currently work with the hyperspy loader..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "de_mrc = hs.load('/dls/e02/data/2021/cm28158-1/DE16/Test_256_subsampled_02x_01.mrc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 - EPSIC [DLS Conda]",
   "language": "python",
   "name": "conda-env-DLS_Conda-epsic3.7-kernel.json"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}