Skip to content

Commit

Permalink
updates for crib sheets
Browse files Browse the repository at this point in the history
  • Loading branch information
@dchaddock
dchaddock committed Dec 9, 2024
1 parent ad4e45b commit 864292d
Show file tree
Hide file tree
Showing 10 changed files with 485 additions and 527 deletions.
6 changes: 6 additions & 0 deletions RELEASE_NOTES.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
Version 1.6.0
-------------------

- added dataset_name optional parameter to the `purge_download_output_root_path()` function


Version 1.5.0
-------------------

Expand Down
457 changes: 218 additions & 239 deletions examples/notebooks/data/download_and_read_data.ipynb
View file

Large diffs are not rendered by default.

211 changes: 92 additions & 119 deletions examples/notebooks/data/download_skymaps_and_calibrations.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,44 +1,5 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style>\n",
".cell-output-ipywidget-background {\n",
" background-color: transparent !important;\n",
"}\n",
":root {\n",
" --jp-widgets-color: var(--vscode-editor-foreground);\n",
" --jp-widgets-font-size: var(--vscode-editor-font-size);\n",
"} \n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%html\n",
"<style>\n",
".cell-output-ipywidget-background {\n",
" background-color: transparent !important;\n",
"}\n",
":root {\n",
" --jp-widgets-color: var(--vscode-editor-foreground);\n",
" --jp-widgets-font-size: var(--vscode-editor-font-size);\n",
"} \n",
"</style>"
]
},
{
"cell_type": "code",
"execution_count": 2,
Expand All @@ -56,7 +17,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Skymaps"
"# Skymaps\n",
"\n",
"Skymap files are used for projecting all-sky image data on a map. Skymaps are created for each of the ASI datasets we provide, and generate new ones every year, or when the deployed instrument is serviced. \n",
"\n",
"A detailed description of the skymaps can be found [here](https://data.phys.ucalgary.ca/sort_by_project/other/documentation/skymap_file_description.pdf).\n",
"\n",
"If you find that you are projecting data onto a map with multiple imagers and an auroral arc is not lining up between two sites, this is normally resolved by using a different skymap (a good first try is the one before or after what you are using). If you continue to have issues, reach out to the [dataset contact](https://data.phys.ucalgary.ca/about_datasets)."
]
},
{
Expand All @@ -74,12 +41,13 @@
"THEMIS_ASI_SKYMAP_IDLSAV UCalgary L3 None THEMIS All Sky Imagers skymap data (IDL save format) \n",
"TREX_BLUE_SKYMAP_IDLSAV UCalgary L3 https://commons.datacite.org/doi.org/10.11575/80pf-0p02 TREx Blueline All Sky Imagers skymap data (IDL save format)\n",
"TREX_NIR_SKYMAP_IDLSAV UCalgary L3 https://commons.datacite.org/doi.org/10.11575/98w7-jp47 TREx NIR All Sky Imagers skymap data (IDL save format) \n",
"TREX_RGB_SKYMAP_IDLSAV UCalgary L3 https://commons.datacite.org/doi.org/10.11575/4p8e-1k65 TREx RGB All Sky Imagers skymap data (IDL save format) \n"
"TREX_RGB_SKYMAP_IDLSAV UCalgary L3 https://commons.datacite.org/doi.org/10.11575/4p8e-1k65 TREx RGB All Sky Imagers skymap data (IDL save format) \n",
"TREX_SPECT_SKYMAP_IDLSAV UCalgary L3 https://commons.datacite.org/doi.org/10.11575/2wnp-yc80 TREx Spectrograph skymap data (IDL save format) \n"
]
}
],
"source": [
"# search datasets for skymaps\n",
"# list all available skymap datasets\n",
"aurorax.data.list_datasets_in_table(name=\"SKYMAP\")"
]
},
Expand All @@ -89,8 +57,8 @@
"metadata": {},
"outputs": [],
"source": [
"# we'll set our dataset for use later\n",
"dataset = aurorax.data.list_datasets(name=\"THEMIS_ASI_SKYMAP_IDLSAV\")[0]"
"# set the dataset that we'll use later\n",
"dataset = aurorax.data.get_dataset(name=\"THEMIS_ASI_SKYMAP_IDLSAV\")"
]
},
{
Expand All @@ -99,8 +67,8 @@
"source": [
"When selecting a skymap to use for projecting an image on a map, we have two methods available to us:\n",
"\n",
"1) choosing manually\n",
"2) using the `find_best_skymap()` function to choose automatically\n",
"1) using the `find_best_skymap()` function to choose automatically\n",
"2) choosing manually\n",
"\n",
"Skymaps are generated for each site, and for a given time period. It is important to choose a skymap that is valid for the date you're looking at data for, otherwise the image may not appear accurately when projected on a map.\n"
]
Expand All @@ -109,18 +77,62 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Choosing a skymap manually"
"## Automatically choosing a skymap\n",
"\n",
"The easiest way to choose a skymap is to lean on the `download_best_skymap()` function to let PyAuroraX figure out what's best to use. It takes the dataset name, site/observatory unique identifier, and a timestamp. Since skymaps change over the course of an imaging season, it is important to supply the timestamp of the data that you are plotting.\n",
"\n",
"To explore all available skymaps, you can see them in the data tree for each instrument array (ie. [THEMIS ASI](https://data.phys.ucalgary.ca/sort_by_project/THEMIS/asi/skymaps/), [TREx RGB](https://data.phys.ucalgary.ca/sort_by_project/TREx/RGB/skymaps/)).\n",
"\n",
"You can also let the library choose the skymap for you using the `download_best_skymap()` function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[PosixPath('/home/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20210308/themis_skymap_gill_20210308-+_v02.sav')]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# let's say we are working on data from the Gillam THEMIS ASI on 2021-11-04\n",
"#\n",
"# set params\n",
"dataset_name = \"THEMIS_ASI_SKYMAP_IDLSAV\"\n",
"site_uid = \"gill\"\n",
"dt = datetime.datetime(2021, 11, 4)\n",
"\n",
"# download the recommended skymap\n",
"r = aurorax.data.ucalgary.download_best_skymap(dataset_name, site_uid, dt)\n",
"r.filenames"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Choosing a skymap manually\n",
"\n",
"Choosing a skymap manually"
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a9c344f8ac6d46078b4087fc83291613",
"model_id": "96c1a042d8344e3bb56caff20f4554ba",
"version_major": 2,
"version_minor": 0
},
Expand All @@ -134,20 +146,18 @@
{
"data": {
"text/plain": [
"[WindowsPath('C:/Users/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20210308/themis_skymap_gill_20210308-+_v02.sav'),\n",
" WindowsPath('C:/Users/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20220301/themis_skymap_gill_20220301-+_v02.sav')]"
"[PosixPath('/home/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20210308/themis_skymap_gill_20210308-+_v02.sav'),\n",
" PosixPath('/home/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20220301/themis_skymap_gill_20220301-+_v02.sav')]"
]
},
"execution_count": 5,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# First, let's choose the skymap we want manually. Let's assume we are working on data\n",
"# from the Gillam THEMIS ASI on 2021-11-04.\n",
"#\n",
"# We'll download the skymaps for a few years around that time, and then we'll choose which one we want after\n",
"# Next, let's see how you'd manually choose the skymap. We'll download the skymaps \n",
"# for a few years around that time, and then we'll go and decide which one we want\n",
"r = aurorax.data.ucalgary.download(\n",
" \"THEMIS_ASI_SKYMAP_IDLSAV\",\n",
" datetime.datetime(2021, 1, 1, 0, 0),\n",
Expand All @@ -163,12 +173,12 @@
"source": [
"Looks like we have a couple skymaps to choose from around 2021-11-04. We'll choose the first one since the date for it is before, and the second one's date is after. \n",
"\n",
"The date indicates the first date it is valid for. There are some cases where a later or earlier skymap can be used. That is a situation where you can play around and try different skymaps, looking for which one works best for you. Most skymaps have small differences, but some have large ones that you'll notice very quickly when working with the projected data on a map.\n"
"The date indicates the first date it is valid for. There are some cases where a later or earlier skymap can be used. That is a situation where you can play around and try different skymaps. Most skymaps have small differences, but some have large ones that you'll notice very quickly when working with the projected data on a map (usually happens after the instrument has been serviced).\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 9,
"metadata": {},
"outputs": [
{
Expand All @@ -178,35 +188,34 @@
"Data(data=[1 Skymap object], timestamp=[], metadata=[], problematic_files=[], calibrated_data=None, dataset=Dataset(name=THEMIS_ASI_SKYMAP_IDLSAV, short_description='THEMIS All Sky Im...))\n",
"\n",
"Data:\n",
" data : [1 Skymap object]\n",
" timestamp : []\n",
" metadata : []\n",
" problematic_files : []\n",
" calibrated_data : None\n",
" dataset : Dataset(name=THEMIS_ASI_SKYMAP_IDLSAV, short_description='THEMIS All Sky Im...)\n",
" data : [1 Skymap object]\n",
" timestamp : []\n",
" metadata : []\n",
" problematic_files : []\n",
" calibrated_data : None\n",
" dataset : Dataset(name=THEMIS_ASI_SKYMAP_IDLSAV, short_description='THEMIS All Sky Im...)\n",
"\n",
"Skymap:\n",
" filename : C:\\Users\\darrenc\\pyaurorax_data\\THEMIS_ASI_SKYMAP_IDLSAV\\gill\\gill_20210308\\themis_skymap_gill_20210308-+_v02.sav\n",
" full_azimuth : array(dims=(256, 256), dtype=>f4)\n",
" full_elevation : array(dims=(256, 256), dtype=>f4)\n",
" full_map_altitude : array(dims=(3,), dtype=>f4)\n",
" full_map_latitude : array(dims=(3, 257, 257), dtype=>f4)\n",
" full_map_longitude : array(dims=(3, 257, 257), dtype=>f4)\n",
" generation_info : SkymapGenerationInfo(...)\n",
" imager_uid : themis19\n",
" project_uid : themis\n",
" site_map_altitude : 0.0\n",
" site_map_latitude : 56.376724\n",
" site_map_longitude : 265.35632\n",
" site_uid : gill\n",
" version : v02\n"
" filename : /home/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20210308/themis_skymap_gill_20210308-+_v02.sav\n",
" full_azimuth : array(dims=(256, 256), dtype=>f4)\n",
" full_elevation : array(dims=(256, 256), dtype=>f4)\n",
" full_map_altitude : array(dims=(3,), dtype=>f4)\n",
" full_map_latitude : array(dims=(3, 257, 257), dtype=>f4)\n",
" full_map_longitude : array(dims=(3, 257, 257), dtype=>f4)\n",
" generation_info : SkymapGenerationInfo(...)\n",
" imager_uid : themis19\n",
" project_uid : themis\n",
" site_map_altitude : 0.0\n",
" site_map_latitude : 56.376724\n",
" site_map_longitude : 265.35632\n",
" site_uid : gill\n",
" version : v02\n"
]
}
],
"source": [
"# Now that we know which one we'll use, we can read it in.\n",
"#\n",
"# You can also read in all of them and choose later using the resulting Data object.\n",
"# Now that we know which one we'll use, we can read it in. You can also read \n",
"# in all of them and choose later using the resulting Data object.\n",
"skymap_data = aurorax.data.ucalgary.read(dataset, r.filenames[0])\n",
"\n",
"print(skymap_data)\n",
Expand All @@ -217,45 +226,9 @@
"skymap_data.data[0].pretty_print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Automatically choosing a skymap\n",
"\n",
"You can also let the library choose the skymap for you using the `download_best_skymap()` function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[WindowsPath('C:/Users/darrenc/pyaurorax_data/THEMIS_ASI_SKYMAP_IDLSAV/gill/gill_20210308/themis_skymap_gill_20210308-+_v02.sav')]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# set params\n",
"dataset_name = \"THEMIS_ASI_SKYMAP_IDLSAV\"\n",
"site_uid = \"gill\"\n",
"dt = datetime.datetime(2021, 11, 4)\n",
"\n",
"# get the recommendation\n",
"r = aurorax.data.ucalgary.download_best_skymap(dataset_name, site_uid, dt)\n",
"r.filenames\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -573,7 +546,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.19"
"version": "3.9.20"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit 864292d

Please sign in to comment.