This repository contains code for reading Transition Region Explorer (TREx) All-Sky Imager (ASI) raw data. The data can be found at https://data.phys.ucalgary.ca.
Quick Links:
There exists readfile software for both IDL and Python. The datasets supported by these readfiles include:
- Blueline: stream0 PGM files
- Near-infrared: stream0 PGM files
- RGB: stream0 nominal 3s cadence colour images (HDF5), and stream0.burst 3Hz cadence colour images (PNG)
- Spectrograph: stream0 PGM files
This library can be installed for Python or IDL. Python installation is done using pip and IDL installation is done using ipm.
The trex-imager-readfile library is available on PyPI and officially supports Python 3.8+.
$ pip install trex-imager-readfile
$ python
>>> import trex_imager_readfileSince IDL 8.7.1, there exists an IDL package manager called ipm. We can use this to install the trex-imager-readfile library with a single command.
-
From the IDL command prompt, run the following:
IDL> ipm,/install,'https://aurora.phys.ucalgary.ca/public/trex-imager-readfile-idl/latest.zip'
-
Add the following to your startup file, or run the command manually using the IDL command prompt:
[ open your startup.pro file and put the following in it ] .run trex_imager_readfile_startup -
Reset your IDL session by either clicking the Reset button in the IDL editor or by typing
.resetinto the IDL command prompt. If you compiled the code manually in step 2 (instead of adding to your startup file), skip this step.
For further information, you can view what packages are installed using ipm,/list. You can also view the package details using ipm,/query,'trex-imager-readfile'. Previous releases are available here.
In Python, pip can be used to update the package.
$ pip install --upgrade trex-imager-readfileIn IDL, you can use the ipm command to update the package.
IDL> ipm,/update,'trex-imager-readfile'
IDL> .reset
[ instead of resetting, you can recompile manually ]
IDL> .run trex_imager_readfile_startupThe below text provides documentation for the available functions/procedures as part of the IDL and Python libraries.
Available functions:
trex_imager_readfile.read_blueline(file_list, workers=1, first_frame=False, no_metadata=False, quiet=False)trex_imager_readfile.read_nir(file_list, workers=1, first_frame=False, no_metadata=False, quiet=False)trex_imager_readfile.read_rgb(file_list, workers=1, first_frame=False, no_metadata=False, tar_tempdir=None, quiet=False)trex_imager_readfile.read_spectrograph(file_list, workers=1, first_frame=False, no_metadata=False, quiet=False)
Parameters:
file_list: filename or list of filenames --> type strworkers: number of worker processes to spawn, defaults to 1 --> type int, optionalfirst_frame: only read the first frame of a 1-min file (H5, stacked PGM, PNG tarball), defaults to False --> type bool, optionalno_metadata: skip reading of metadata, defaults to False -> type bool, optionaltar_tempdir: path to untar files to, defaults to '~/.trex_imager_readfile' --> type str, optionalquiet: reduce output while reading data, defaults to False --> type bool, optional
Return values:
- return variables:
images, metadata dictionaries, and problematic files - return types:
numpy.ndarray, list[dict], list[dict]
Warning: On Windows, be sure to put any read_* calls into a main() method. This is because we utilize the multiprocessing library and the method of forking processes in Windows requires it. Note that if you're using Jupyter or other IPython-based interfaces, this is not required.
For full documentation, see the main source file here.
; CALLING SEQUENCE:
; TREX_IMAGER_READFILE, filename, images, metadata, /KEYWORDS
;
; INPUTS:
; filename - a string OR array of strings containing valid TREx image filenames
;
; OUTPUTS:
; images - PGM files (TREx NIR, Blueline, Spectrograph)
; --> a WIDTH x HEIGHT x NFRAMES array of unsigned integers or bytes
; - H5 files (TREx RGB nominal cadence)
; --> a CHANNELS x WIDTH x HEIGHT x NFRAMES array of unsigned integers or bytes
; - PNG files (TREx RGB burst cadence)
; --> a CHANNELS x WIDTH x HEIGHT x NFRAMES array of unsigned integers or bytes
; metadata - a NFRAMES element array of structures
;
; KEYWORDS:
; FIRST_FRAME - only read the first frame of a 1-min file (H5, stacked PGM, PNG tarball)
; NO_METADATA - don't read or process metadata (use if file has no metadata or you don't
; want to read it)
; MINIMAL_METADATA - set the least required metadata fields (slightly faster)
; ASSUME_EXISTS - assume that the filename(s) exist (slightly faster)
; COUNT - returns the number of image frames (usage ex. COUNT=nframes)
; VERBOSE - set verbosity to level 1
; VERY_VERBOSE - set verbosity to level 2
; SHOW_DATARATE - show the read datarate stats for each file processed (usually used
; with /VERBOSE keyword)
; UNTAR_DIR - specify the directory to untar RGB colour PNG files to, default
; is IDL_TMPDIR on Windows and '~/.trex_imager_readfile' on
; Linux (usage ex. UNTAR_DIR='path\for\files')
; NO_UNTAR_CLEANUP - don't remove files after untarring to the UNTAR_DIR and readingBelow are a few quick examples of using the readfile library in Python and IDL.
Available functions:
- trex_imager_readfile.read_blueline() -- Jupyter notebook example
- trex_imager_readfile.read_nir() -- Jupyter notebook example
- trex_imager_readfile.read_rgb() -- Jupyter notebook example
- trex_imager_readfile.read_spectrograph() -- Jupyter notebook example
Further, some quick examples are below.
>>> import trex_imager_readfile
>>> filename = "path/to/rgb_data/2022/02/01/fsmi_rgb-01/ut06/20220201_0600_fsmi_rgb-01_full.h5"
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(filename)>>> import trex_imager_readfile, glob
>>> file_list = glob.glob("path/to/files/2020/01/01/fsmi_rgb-01/ut06/*full.h5")
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(file_list)>>> import trex_imager_readfile, glob
>>> file_list = glob.glob("path/to/files/2020/01/01/fsmi_rgb-01/ut06/*full.h5")
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(file_list, workers=4)If a file has issues being read in, it is placed into the problematic_files variable and each error message is written to stdout. If you'd like the read function to not output print messages to stdout, you can use the quiet=True parameter.
>>> import trex_imager_readfile, glob
>>> file_list = glob.glob("path/to/files/2020/01/01/fsmi_rgb-01/ut06/*full.h5")
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(file_list, workers=4, quiet=True)>>> import trex_imager_readfile, glob
>>> file_list = glob.glob("path/to/files/2020/01/01/fsmi_rgb-01/ut06/*full.h5")
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(file_list, first_frame=True)
>>> import trex_imager_readfile, glob
>>> file_list = glob.glob("path/to/files/2020/01/01/fsmi_rgb-01/ut06/*full.h5")
>>> img, meta, problematic_files = trex_imager_readfile.read_rgb(file_list, no_metadata=True)
IDL> trex_imager_readfile,filename,img,meta
IDL> help,img
IMG UINT = Array[256, 256, 20]
IDL> help,meta
META STRUCT = -> TREX_IMAGER_METADATA Array[20]
IDL> f=file_search("C:\path\to\files\for\an\hour\*")
IDL> trex_imager_readfile,f,img,meta
IDL> help,img
IMG UINT = Array[256, 256, 1200]
IDL> help,meta
META STRUCT = -> TREX_IMAGER_METADATA Array[1200]
Read only the first frame of a file (can be used to speed up performance if you only need the first frame)
IDL> trex_imager_readfile,filename,img,meta,/first_frame
IDL> help,img
IMG UINT = Array[256, 256]
IDL> help,meta
String STRUCT = -> TREX_IMAGER_METADATA
IDL> trex_imager_readfile,filename,img,meta,/no_metadata
You can alternatively install the trex-imager-readfile library manually by downloading the ZIP file and extracting it into, or adding it to, your IDL path.
-
Download the latest release here
-
Extract the zip file into your IDL path (or add it as a directory to your IDL path)
-
Add the following to your startup file (or run the command manually using the IDL command prompt).
[ open your startup.pro file and put the following in it ] .run trex_imager_readfile_startup -
Reset your IDL session by either clicking the Reset button in the IDL editor or by typing
.resetinto the IDL command prompt.
$ git clone https://github.com/ucalgary-aurora/trex-imager-readfile.git
$ cd trex-imager-readfile/python
$ pip install poetry
$ poetry install$ cd python
$ make test-linting
$ make test-pytestThe PyTest functionality tests include several categories of tests. You can run each category separately if you want using the "markers" feature of PyTest. All markers are found in the pytest.ini file at the root of the repository.
Below are some more commands for advanced usages of PyTest.
poetry run pytest --collect-onlyList all available testspoetry run pytest --markersList all markers (includes builtin, plugin and per-project ones)poetry run pytest -v -m nirPerform only the tests for the "nir" markercat pytest.iniList custom markers
When a new release is ready for deployment, there are a few tasks that need to be done.
-
Increment the version number and change the date in
idlpackage.json,trex_imager_readfile.pro, andREADME.md. -
Generate a new distributable Zip file (more info)
IDL> ipm,/create,'path_to_code',name='trex-imager-readfile'
-
Upload the generated Zip file to https://aurora.phys.ucalgary.ca/public/trex-imager-readfile-idl, and update the symlink for latest.zip