butler_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e82b4c3-2faf-43c8-9a98-065f4d157409",
   "metadata": {},
   "source": [
    "<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=170 style=\"padding: 10px\"> \n",
    "<b>Little Demo: Butler 2</b> <br>\n",
    "Contact author(s): Melissa Graham <br>\n",
    "Last verified to run: 2024-06-27 <br>\n",
    "LSST Science Pipelines version: Weekly 2024_16 <br>\n",
    "Container Size: medium"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9801707b-ac96-4bef-8746-6ff8ec4c9e55",
   "metadata": {},
   "source": [
    "The `butler` is powerful middleware to query and retrieve LSST data.\n",
    "\n",
    "This little demo shows how to explore the bulter and find out what data is available.\n",
    "\n",
    "For more information about the `butler`, see the [butler documentation](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/index.html), the [butler FAQ](https://pipelines.lsst.io/middleware/faq.html#frequently-asked-questions), and [tutorial notebooks](https://github.com/rubin-dp0/tutorial-notebooks) 04a and 04b.\n",
    "\n",
    "## 1. Set up\n",
    "\n",
    "Import packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d94698a-400b-41c4-956b-5b59f30fac76",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lsst.daf.butler as dafButler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad18341-94f4-404f-b479-9f29a63f3c20",
   "metadata": {},
   "source": [
    "Instantiate the `butler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26db757c-a922-4c0a-ade2-b5e05b5021d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "butler = dafButler.Butler('dp02', collections='2.2i/runs/DP0.2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca33f2b-77ff-4d29-ba57-2bb02089715b",
   "metadata": {},
   "source": [
    "## 2. The registry\n",
    "\n",
    "The `registry` contains entries for all data products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "587e2f34-1fc1-44f0-84fa-46dca304f80c",
   "metadata": {},
   "outputs": [],
   "source": [
    "registry = butler.registry"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac9ff90-b524-483c-a75f-b864edffddd3",
   "metadata": {},
   "source": [
    "### 2.1. Collections\n",
    "\n",
    "Data sets in the `registry` are organized into [collections](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/organizing.html#collections).\n",
    "\n",
    "Find all collections with the word \"raw\" in their name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb508cc-3175-4dc4-b815-df9572766cba",
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in sorted(registry.queryCollections(\"*raw*\")):\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95318d5-2384-4a3b-8be8-8f33b9c28930",
   "metadata": {},
   "source": [
    "### 2.2. Dataset types\n",
    "\n",
    "The `butler` contains a variety of [dataset types](https://pipelines.lsst.io/middleware/faq.html#querydatasettypes).\n",
    "\n",
    "For example, print all dataset types with names that start with `calexp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e56ad1b2-49ea-4aef-a269-3b23dc1419a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for dt in sorted(registry.queryDatasetTypes('calexp*')):\n",
    "    print(dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45c92efc-e6bf-44b4-b534-baeb9fa92833",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-27T17:46:31.642166Z",
     "iopub.status.busy": "2024-06-27T17:46:31.641922Z",
     "iopub.status.idle": "2024-06-27T17:46:31.668916Z",
     "shell.execute_reply": "2024-06-27T17:46:31.668470Z",
     "shell.execute_reply.started": "2024-06-27T17:46:31.642150Z"
    }
   },
   "source": [
    "## 3. The dataId\n",
    "\n",
    "The `dataId` is a dictionary-like identifier ([dataId documentation](https://pipelines.lsst.io/modules/lsst.daf.butler/dimensions.html#data-ids)).\n",
    "\n",
    "The `DatasetType` (above) will show the `dimensions` of the `dataId`.\n",
    "\n",
    "For example, print the `dataId` dimensions for a `calexp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e05f55c-6955-4e5f-9fe7-750334463d84",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = registry.getDatasetType('calexp')\n",
    "print(\"Name:\", dt.name)\n",
    "print(\"Dimensions:\", dt.dimensions)\n",
    "print(\"Storage Class:\", dt.storageClass)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c6aa9aa-0ff5-4b89-9c17-d328dfa6991f",
   "metadata": {},
   "source": [
    "### 3.1. Query datasets\n",
    "\n",
    "A `calexp` `dataId` can be uniquely specified by only visit and detector.\n",
    "\n",
    "For a `calexp`, define a minimal `dataId` and get the band from the full `dataId`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9fe23ab-8a2f-441e-a0dd-4575a19ed902",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataId = {'visit': 192350, 'detector': 175}\n",
    "for i, ref in enumerate(registry.queryDatasets('calexp', dataId=dataId)):\n",
    "    print(\"band: \", ref.dataId['band'])\n",
    "    print(' ')\n",
    "    print('full dataId')\n",
    "    print(ref.dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b2f6bf6-2374-4f6e-b3ec-d7c73934d33f",
   "metadata": {},
   "source": [
    "Visit alone is insufficient to uniquely specify a `calexp.\n",
    "\n",
    "Define a `dataId` with only visit and print the number of `calexps` (detectors) for that visit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf083e08-c8f4-4586-8b60-96f783af17dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataId = {'visit': 192350}\n",
    "datasetRefs = set(registry.queryDatasets('calexp', dataId=dataId))\n",
    "print(f\"Found {len(datasetRefs)} detectors for this visit.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86820430-88ce-4dba-87a6-8a46cc69c7cc",
   "metadata": {},
   "source": [
    "Print the first two `dataId`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "404c5cd0-254f-44c6-8ba9-e57bf58e3bb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, ref in enumerate(datasetRefs):\n",
    "    if i < 2:\n",
    "        print(ref.dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c56fdab8-536a-4c93-a393-b7a60850bb0f",
   "metadata": {},
   "source": [
    "## 4. Retrieving data\n",
    "\n",
    "With `datasetRefs` already defined, retrieve the first two `calexp` in the list.\n",
    "\n",
    "As an example of accessing metadata, simply print their detector number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be64b0f2-5293-41ea-a754-d6c90bdfc457",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, ref in enumerate(datasetRefs):\n",
    "    if i < 2:\n",
    "        calexp = butler.get(ref)\n",
    "        print(i, ' calexp.detector.getId(): ', calexp.detector.getId())\n",
    "        del calexp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57788ab1-4573-4522-8374-d6d46bbf80e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}