diff --git a/_toc.yml b/_toc.yml index 6029de926..35a4881a3 100644 --- a/_toc.yml +++ b/_toc.yml @@ -106,3 +106,5 @@ parts: title: IFU Products with NSClean - file: notebooks/NIRSpec_NSClean/MOS_NSClean_example.ipynb title: MOS Products with NSClean + - file: notebooks/NIRSpec_NSClean/BOTS_NSClean_example.ipynb + title: BOTS Products with NSClean \ No newline at end of file diff --git a/notebooks/NIRSpec_NSClean/BOTS_NSClean_example.ipynb b/notebooks/NIRSpec_NSClean/BOTS_NSClean_example.ipynb new file mode 100644 index 000000000..729c8554d --- /dev/null +++ b/notebooks/NIRSpec_NSClean/BOTS_NSClean_example.ipynb @@ -0,0 +1,1290 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "f9edb434", + "metadata": {}, + "source": [ + "\n", + "\n", + "# Cleaning Residual 1/f Noise in NIRSpec BOTS Products with NSClean \n", + "
\n", + "\n", + "## Notebook Goal\n", + "\n", + "The goal of this notebook is to generate cleaned BOTS (*_rateint.fits*) files by removing residual 1/f noise. These cleaned files will be used as input for the 3 (`TSO3Pipeline`) pipeline.\n", + "\n", + "\n", + "## Table of Contents\n", + "\n", + "* 1. [Introduction](#introduction)\n", + "* 2. [Import Library](#imports)\n", + "* 3. [Download the Data](#data)\n", + "* 4. [Running `Spec2Pipeline` without NSClean (Original Data)](#nsclean_skipped)\n", + " * 4.1 [Modifying the EXTRACT1D Reference File](#extract1d_mod)\n", + "* 5. [Clean up 1/f Noise with NSClean (Default Pipeline Mask)](#nsclean_default)\n", + " * 5.1 [Verify the Mask (Default Pipeline Mask)](#verify_default_mask)\n", + " * 5.2 [Comparing Original vs. Cleaned Data (Default Pipeline Mask)](#nsclean_default_compare)\n", + "* 6. [Clean up 1/f Noise with NSClean (Alternate Mask)](#nsclean_alternate)\n", + " * 6.1 [Verify the Mask (Alternate Mask)](#verify_alternate_mask)\n", + " * 6.2 [Comparing Original vs. Cleaned Data (Alternate Mask)](#nsclean_alternate_compare)\n", + "* 7. [Clean up 1/f Noise with NSClean (Hand-Modified Mask)](#nsclean_modified)\n", + " * 7.1 [Verify the Mask (Hand-Modified Mask)](#verify_modified_mask)\n", + " * 7.2 [Comparing Original vs. Cleaned Data (Hand-Modified Mask)](#nsclean_modified_compare)\n", + "* 8. [Conclusion](#conclusion)\n", + "* [About the Notebook](#about)\n", + "\n", + "\n", + "## 1. Introduction \n", + "
\n", + "\n", + "The JWST NIRSpec instrument has a number of features and characteristics that observers should be aware of when planning observations and interpreting data. One notable feature seen in NIRSpec pipeline products is negative and/or surplus flux in the extracted 1-D spectrum, typically with an irregular wavelength-dependent undulation. The cause of this artifact is correlated noise, known as [1/f noise](https://jwst-docs.stsci.edu/jwst-near-infrared-spectrograph/nirspec-instrument-features-and-caveats#NIRSpecInstrumentFeaturesandCaveats-1/fnoise), from low-level detector thermal instabilities, seen as vertical banding in 2-D count rate images, particularly in exposures of the NRS2 detector. While the IRS2 readout mode reduces this effect, it is not completely eliminated.\n", + "\n", + "To address this issue, the JWST Science Calibration Pipeline has integrated an external package developed by Bernard Rauscher, known as [NSClean](https://webb.nasa.gov/content/forScientists/publications.html#NSClean), within the `Spec2Pipeline` under [NSCleanStep](https://jwst-pipeline.readthedocs.io/en/latest/jwst/nsclean/main.html). This algorithm uses dark areas of the detector to fit a background model to the data in Fourier space. It requires an input mask to identify all dark areas of the detector. The more thorough and complete this mask is, the better the background fit.\n", + "\n", + "In this notebook, we will use the NSClean algorithm integrated into the pipeline, utilizing a mask generated on-the-fly with default parameters to remove 1/f noise. In some cases, this mask may not be complete enough/too restrictive for the best possible noise removal. To address this, we demonstrate how one can manually modify the default mask, as well as how to create an alternative mask by adjusting the [NSCleanStep parameters](https://jwst-pipeline.readthedocs.io/en/latest/jwst/nsclean/arguments.html). If needed, see the [NSClean documentation](https://iopscience.iop.org/article/10.1088/1538-3873/ad1b36/pdf) for some suggestions on manually creating a custom mask.\n", + "\n", + "This notebook utilizes transit data of WASP-39b observed in the BOTS G395H grism as part of the [JWST Early Release Science program ERS-1366](https://www.stsci.edu/jwst/science-execution/approved-programs/dd-ers/program-1366) observation 3, as an example." + ] + }, + { + "cell_type": "markdown", + "id": "f8e576d0", + "metadata": {}, + "source": [ + "## 2. Import Library \n", + "
\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7fc8dee-052e-4eda-898b-9ee30b40eeaf", + "metadata": {}, + "outputs": [], + "source": [ + "# ---------- Set CRDS environment variables ----------\n", + "import os\n", + "import jwst\n", + "os.environ['CRDS_CONTEXT'] = 'jwst_1215.pmap'\n", + "os.environ['CRDS_PATH'] = os.environ['HOME']+'/crds_cache'\n", + "os.environ['CRDS_SERVER_URL'] = 'https://jwst-crds.stsci.edu'\n", + "print(f'CRDS cache location: {os.environ[\"CRDS_PATH\"]}')\n", + "\n", + "print(\"JWST Calibration Pipeline Version={}\".format(jwst.__version__))\n", + "# print(\"Current Operational CRDS Context = {}\".format(crds.get_default_context()))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3ead3ba", + "metadata": {}, + "outputs": [], + "source": [ + "# ------ General Imports ------\n", + "import numpy as np\n", + "import time as tt\n", + "import logging\n", + "import warnings\n", + "import json\n", + "\n", + "# ------ JWST Calibration Pipeline Imports ------\n", + "from crds.client import api\n", + "from stpipe import crds_client\n", + "from jwst.pipeline.calwebb_spec2 import Spec2Pipeline\n", + "\n", + "# ------ Plotting/Stats Imports ------\n", + "from matplotlib import pyplot as plt\n", + "from astropy.io import fits\n", + "\n", + "from utils import get_jwst_file, plot_dark_data, plot_cleaned_data, plot_spectra\n", + "\n", + "# Hide all log and warning messages.\n", + "logging.disable(logging.ERROR)\n", + "warnings.simplefilter(\"ignore\", RuntimeWarning)" + ] + }, + { + "cell_type": "markdown", + "id": "e6fecded", + "metadata": {}, + "source": [ + "## 3. Download the Data \n", + "
\n", + "\n", + "The input data for this notebook features a transit of WASP-39b observed through the NIRSpec BOTS mode with the G395H grism. The dataset is part of the [JWST Early Release Science program ERS-1366](https://www.stsci.edu/jwst/science-execution/approved-programs/dd-ers/program-1366), specifically observation 3. It consists of 456 integrations with 70 groups each, using the SUB2048 subarray (array size of 2048 by 32 pixels per integration, per group). The entire observation spans 10.56 hours and is divided into three segments. This notebook focuses on the second segment (seg002) as an example. However, it's important to note that before proceeding to the `TSO3Pipeline`, all segments must first be processed through the `Spec2Pipeline`.\n", + "\n", + "For a comprehensive overview of current and planned JWST Time-Series Observation (TSO) programs, particularly those involving transiting exoplanets with the NIRSpec BOTS mode, refer to [TrExoLiSTS](https://www.stsci.edu/~nnikolov/TrExoLiSTS/JWST/trexolists.html)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ebe84f8f", + "metadata": {}, + "outputs": [], + "source": [ + "# Define a downloads directory.\n", + "mast_products_dir = \"./mast_products/\"\n", + "# Check if the directory exists.\n", + "if not os.path.exists(mast_products_dir):\n", + " # Create the directory if it doesn't exist.\n", + " os.makedirs(mast_products_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0d2f0aae", + "metadata": {}, + "outputs": [], + "source": [ + "# This notebook focuses on the second segment.\n", + "obs_ids = [\"jw01366003001_04101_00001-seg002\"]\n", + "detectors = [1, 2] # Both Detectors NRS1 and NRS2.\n", + "\n", + "\n", + "# Specify the countrate products.\n", + "rateints_names = []\n", + "\n", + "for obs_id in obs_ids:\n", + " for detector in detectors:\n", + " rateints_names.append(f\"{obs_id}_nrs{detector}_rateints.fits\")\n", + "\n", + "# Download all the FITS files.\n", + "for name in rateints_names:\n", + " print(f\"Downloading {name}\")\n", + " get_jwst_file(\n", + " name,\n", + " mast_api_token=None,\n", + " save_directory=mast_products_dir,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "edb427ae", + "metadata": {}, + "source": [ + "## 4. Running `Spec2Pipeline` without NSClean (Original Data) \n", + "
\n", + "\n", + "The cell below executes the `Spec2Pipeline`, explicitly skipping the NSClean step during processing. The level 2 products generated will serve as a reference point to illustrate how the countrate images and final extracted spectra appear without the removal of 1/f noise." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "84fc4501", + "metadata": {}, + "outputs": [], + "source": [ + "# Set up directory for running the pipeline without NSClean.\n", + "stage2_nsclean_skipped_dir = \"./stage2_nsclean_skipped/\"\n", + "if not os.path.exists(stage2_nsclean_skipped_dir):\n", + " os.makedirs(stage2_nsclean_skipped_dir) # Create the directory if it doesn't exist." + ] + }, + { + "cell_type": "markdown", + "id": "375a2711", + "metadata": {}, + "source": [ + "### 4.1 Modifying the EXTRACT1D Reference File \n", + "
\n", + "\n", + "We adjust the default parameter reference file for the `extract_1d` step, specifically modifying the extraction width aperture. This modification is necessary because the pipeline currently does not resample the 2D spectra to rectify the spectra (remove the curvature). Consequently, we rely on a broad extraction aperture, especially for NRS2." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5bf6ac1b", + "metadata": {}, + "outputs": [], + "source": [ + "# EXTRACT1D reference file for BOTS data NRS1.\n", + "refs = api.dump_references(crds_client.get_context_used('jwst'),\n", + " ['jwst_nirspec_extract1d_0006.json'])\n", + "extract_1d_ref = refs['jwst_nirspec_extract1d_0006.json']\n", + "\n", + "# Open EXTRACT1D reference file in read-mode.\n", + "with open(extract_1d_ref, \"r\") as ref_file:\n", + " params = json.load(ref_file)\n", + "\n", + " params[\"apertures\"][0][\n", + " \"extract_width\"\n", + " ] = 27 # Extraction aperture radius in pixels, can be modified.\n", + " params[\"apertures\"][0][\"xstart\"] = 100\n", + "\n", + " params[\"apertures\"][0].pop(\"nod2_offset\") # Remove.\n", + " params[\"apertures\"][0].pop(\"nod3_offset\") # Remove.\n", + " params[\"apertures\"][0].pop(\"nod5_offset\") # Remove.\n", + "\n", + " params[\"apertures\"][1][\n", + " \"extract_width\"\n", + " ] = 27 # Extraction aperture radius in pixels, can be modified.\n", + " params[\"apertures\"][1][\"xstart\"] = 100\n", + " params[\"apertures\"][1].pop(\"nod2_offset\") # Remove.\n", + " params[\"apertures\"][1].pop(\"nod3_offset\") # Remove.\n", + " params[\"apertures\"][1].pop(\"nod5_offset\") # Remove.\n", + "\n", + "# Write changes to a new file.\n", + "newData = json.dumps(params, indent=4)\n", + "# Add the suffix '_bots' to distinguish the file from the default version.\n", + "with open(os.path.basename(extract_1d_ref)[:-5] + \"_nrs1_bots.json\", \"w\") as file:\n", + " file.write(newData)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "935c6ef6", + "metadata": {}, + "outputs": [], + "source": [ + "# EXTRACT1D reference file for BOTS data NRS2.\n", + "refs = api.dump_references(crds_client.get_context_used('jwst'),\n", + " ['jwst_nirspec_extract1d_0006.json'])\n", + "extract_1d_ref = refs['jwst_nirspec_extract1d_0006.json']\n", + "\n", + "# Open EXTRACT1D reference file in read-mode.\n", + "with open(extract_1d_ref, \"r\") as ref_file:\n", + " params = json.load(ref_file)\n", + "\n", + " params[\"apertures\"][0][\n", + " \"extract_width\"\n", + " ] = 27 # Extraction aperture radius in pixels, can be modified.\n", + " params[\"apertures\"][0].pop(\"nod2_offset\") # Remove.\n", + " params[\"apertures\"][0].pop(\"nod3_offset\") # Remove.\n", + " params[\"apertures\"][0].pop(\"nod5_offset\") # Remove.\n", + "\n", + " params[\"apertures\"][1][\n", + " \"extract_width\"\n", + " ] = 27 # Extraction aperture radius in pixels, can be modified.\n", + " params[\"apertures\"][1].pop(\"nod2_offset\") # Remove.\n", + " params[\"apertures\"][1].pop(\"nod3_offset\") # Remove.\n", + " params[\"apertures\"][1].pop(\"nod5_offset\") # Remove.\n", + "\n", + "# Write changes to a new file.\n", + "newData = json.dumps(params, indent=4)\n", + "# Add the suffix '_bots' to distinguish the file from the default version.\n", + "with open(os.path.basename(extract_1d_ref)[:-5] + \"_nrs2_bots.json\", \"w\") as file:\n", + " file.write(newData)" + ] + }, + { + "cell_type": "markdown", + "id": "a41cc40b", + "metadata": {}, + "source": [ + "
\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "796f247d", + "metadata": {}, + "outputs": [], + "source": [ + "# Original data (no NSClean Applied).\n", + "# Estimated run time: 10 minutes (may vary).\n", + "start = tt.time()\n", + "\n", + "for i in rateints_names:\n", + " print(f\"Processing {i}...\")\n", + "\n", + " if \"nrs1\" in i:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs1_bots.json\"\n", + " else:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs2_bots.json\"\n", + "\n", + " Spec2Pipeline.call(\n", + " mast_products_dir + i,\n", + " save_results=True,\n", + " steps={\n", + " # Assigns wavelength solution for spectroscopic data.\n", + " \"assign_wcs\": {\"skip\": False},\n", + " # Removes correlated read noise (1/f noise) from NIRSpec images.\n", + " \"nsclean\": {\"skip\": True},\n", + " \"extract_2d\": {\"skip\": False}, # Obtains 2D cutouts.\n", + " \"srctype\": {\"skip\": False}, # Assigns source type: 'point' or 'extended'.\n", + " # We opt to skip this step, because it can introduce spurious pixels\n", + " # that can affect the light curve scatter.\n", + " \"flat_field\": {\"skip\": True},\n", + " # We also skip this step, because BOTS exoplanet observations are relative.\n", + " \"photom\": {\"skip\": True},\n", + " # Estimates the flux values of pixels flagged as DO_NOT_USE in 2-D spectra.\n", + " \"pixel_replace\": {\n", + " \"skip\": False,\n", + " \"n_adjacent_cols\": 5,\n", + " \"algorithm\": \"fit_profile\",\n", + " },\n", + " \"extract_1d\": {\n", + " \"skip\": False,\n", + " \"override_extract1d\": extract1dref,\n", + " \"apply_apcorr\": False,\n", + " \"subtract_background\": False,\n", + " },\n", + " # 'dispaxis': 1, 'xstart': 100, 'xstop': 150, 'extract_width': 10},\n", + " },\n", + " output_dir=stage2_nsclean_skipped_dir,\n", + " )\n", + "\n", + " print(f\"Saved {i[:-13]}\" + \"calints.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"x1dints.fits\")\n", + "\n", + "\n", + "end = tt.time()\n", + "print(\"Run time: \", round(end - start, 1) / 60.0, \" min\")" + ] + }, + { + "cell_type": "markdown", + "id": "a90d6ed0", + "metadata": {}, + "source": [ + "## 5. Clean up 1/f Noise with NSClean (Default Pipeline Mask) \n", + "
\n", + "\n", + "If a user-supplied mask file is not provided to the [NSClean step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/nsclean/index.html) in the `Spec2Pipeline`, the pipeline will generate a mask based on default parameters. This mask will identify any pixel that is unilluminated. That is, the mask must contain True and False values, where True indicates that the pixel is dark, and False indicates that the pixel is illuminated (not dark).\n", + "\n", + "By default, the pipeline marks the following detector areas as illuminated, non-dark areas (False):\n", + "\n", + "* Pixels designated as science areas for FS data.\n", + "* 5-sigma outliers (default value)\n", + "* Any pixel set to NaN in the rate data.\n", + "\n", + "To tune the outlier detection in the mask, try modifying the `n_sigma` parameter (explored in the next section). A higher value will identify fewer outliers. A lower value will identify more. \n", + "\n", + "The default generated mask is saved and analyzed below. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ff0eaf0c", + "metadata": {}, + "outputs": [], + "source": [ + "# Set up directory for running NSClean with default parameters.\n", + "stage2_nsclean_default_dir = \"./stage2_nsclean_default/\"\n", + "if not os.path.exists(stage2_nsclean_default_dir):\n", + " os.makedirs(stage2_nsclean_default_dir) # Create the directory if it doesn't exist." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "019de8fb", + "metadata": {}, + "outputs": [], + "source": [ + "# 1/f noise cleaned data (default NSClean pipeline mask).\n", + "# Estimated run time: 36 minutes (may vary).\n", + "start = tt.time()\n", + "\n", + "for i in rateints_names:\n", + " print(f\"Processing {i}...\")\n", + "\n", + " if \"nrs1\" in i:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs1_bots.json\"\n", + " else:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs2_bots.json\"\n", + "\n", + " Spec2Pipeline.call(\n", + " mast_products_dir + i,\n", + " save_results=True,\n", + " steps={\n", + " # Assigns wavelength solution for spectroscopic data.\n", + " \"assign_wcs\": {\"skip\": False, \"save_results\": True},\n", + " # Removes correlated read noise (1/f noise) from NIRSpec images.\n", + " \"nsclean\": {\"skip\": False, \"save_mask\": True, \"save_results\": True},\n", + " \"extract_2d\": {\"skip\": False}, # Obtains 2D cutouts.\n", + " \"srctype\": {\"skip\": False}, # Assigns source type: 'point' or 'extended'.\n", + " # We opt to skip this step, because it can introduce spurious pixels\n", + " # that can affect the light curve scatter.\n", + " \"flat_field\": {\"skip\": True},\n", + " # We also skip this step, because BOTS exoplanet observations are relative.\n", + " \"photom\": {\"skip\": True},\n", + " # Estimates the flux values of pixels flagged as DO_NOT_USE in 2-D spectra.\n", + " \"pixel_replace\": {\n", + " \"skip\": False,\n", + " \"n_adjacent_cols\": 5,\n", + " \"algorithm\": \"fit_profile\",\n", + " },\n", + " \"extract_1d\": {\n", + " \"skip\": False,\n", + " \"override_extract1d\": extract1dref,\n", + " \"apply_apcorr\": False,\n", + " \"subtract_background\": False,\n", + " },\n", + " # 'dispaxis': 1, 'xstart': 100, 'xstop': 150, 'extract_width': 10},\n", + " },\n", + " output_dir=stage2_nsclean_default_dir,\n", + " )\n", + "\n", + " print(f\"Saved {i[:-13]}\" + \"assign_wcs.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"mask.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"nsclean.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"calints.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"x1dints.fits\")\n", + "\n", + "\n", + "end = tt.time()\n", + "print(\"Run time: \", round(end - start, 1) / 60.0, \" min\")" + ] + }, + { + "cell_type": "markdown", + "id": "fd030dd0", + "metadata": {}, + "source": [ + "
\n", + "Warning: \n", + " \n", + "In some situations, the NSClean step may fail to find a fit to the background noise. This failure may occur if the mask does not contain enough dark data (marked True). In particular, every column in the mask except for the first and last 4 columns must contain some pixels marked True. The background fitting procedure considers each column, one at a time, so it will crash if there is no data in a column to fit. If failure occurs, check that your mask in the image below has at least some True values in every column.\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "id": "45d04156", + "metadata": {}, + "source": [ + "### 5.1 Verify the Mask (Default Pipeline Mask) \n", + "
\n", + "\n", + "Check the mask against the rate data to make sure it keeps only dark areas of the detector.\n", + "\n", + "Note that there are still some remaining illuminated areas, primarily due to transient artifacts like cosmic rays and snowballs.\n", + "\n", + "Also, for some data sets, there may be several illuminated regions of the detector that are not masked by the WCS bounding boxes. In this data, note that the mask does not cover the bluest end of the spectral trace. This region is illuminated, but is not calibrated.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f11ba7d3", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the rate data with masked areas blocked.\n", + "\n", + "# List of on-the-fly built masks from the pipeline.\n", + "nsclean_default_masks = [\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs1_mask.fits\",\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs2_mask.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and mask file.\n", + "for rateint_file, mask_file in zip(rateints_names, nsclean_default_masks):\n", + " plot_dark_data(\n", + " mast_products_dir + rateint_file,\n", + " mask_file,\n", + " slice_index=100,\n", + " axis=0,\n", + " layout=\"rows\",\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "103144e5-a7b9-4f4b-9573-13b5947d1d73", + "metadata": {}, + "source": [ + "The default pipeline mask for NRS2 masks the entire countrate image. " + ] + }, + { + "cell_type": "markdown", + "id": "af681c11", + "metadata": {}, + "source": [ + "### 5.2 Comparing Original vs. Cleaned Data (Default Pipeline Mask) \n", + "
\n", + "\n", + "We can now compare the cleaned data (with the default pipeline mask) to the original rate file and verify that the 1/f noise has been reduced.\n", + "\n", + "In many cases, the cleaning process introduces new artifacts to the rate file. These should be carefully examined and weighed against the benefits of noise reduction. If transient artifacts, like snowballs, are interfering with the cleaning process, it may be beneficial to manually edit the mask to remove these areas from consideration in the background fit. To do so, try varying the outlier detection threshold or editing specific pixels in the mask array directly (explored in the next few sections). Otherwise, refer to the [NSClean documentation](https://iopscience.iop.org/article/10.1088/1538-3873/ad1b36/pdf) for additional suggestions on manual editing.\n", + "\n", + "Note that in the images below, there are scattered values with large relative differences from the original rate file (shown in the relative difference image below). These are artifacts of the cleaning process.\n", + "\n", + "There are also broader low-level residual background effects (shown in the relative difference image on the right, below, with scattered outliers, identified with sigma clipping, hidden by masking). These include the background patterns we are trying to remove: the 1/f noise variations in the dispersion direction and the picture frame effect at the top and bottom of the frame (for full-frame data). However, there may also be low-level artifacts introduced by over-fitting the dark data in the cleaning process.\n", + "\n", + "Check both residual images carefully to understand the impact of the cleaning process on your data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f9975416", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the original and cleaned data, as well as a residual map.\n", + "\n", + "cleaned_default_masks = [\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs1_nsclean.fits\",\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs2_nsclean.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and cleaned file.\n", + "for rateint_file, cleaned_file in zip(rateints_names, cleaned_default_masks):\n", + " plot_cleaned_data(\n", + " mast_products_dir + rateint_file, cleaned_file, slice_index=100, layout=\"rows\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "6ab37ea9", + "metadata": {}, + "source": [ + "The default pipeline mask for NRS2 masks the entire countrate image so there is no difference when NSClean is applied. \n", + "\n", + "Compare the extracted spectrum from the cleaned data to the spectrum extracted from the original rate file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "09c5046b", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the entire spectra.\n", + "\n", + "# 1D extracted spectra.\n", + "x1d_nsclean_skipped = [\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "x1d_nsclean_default = [\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_default_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_default):\n", + " plot_spectra([original, cleaned], ext_num=100)\n", + "\n", + "# Wavelength/flux region of interest\n", + "wavelength_range = {\"nrs1\": [3.4, 3.45], \"nrs2\": [4.25, 4.275]}\n", + "\n", + "flux_range = {\"nrs1\": [1100, 1300], \"nrs2\": [500, 550]}\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_default):\n", + " plot_spectra(\n", + " [original, cleaned],\n", + " ext_num=100,\n", + " scale_percent=0,\n", + " wavelength_range=wavelength_range,\n", + " flux_range=flux_range,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "7c016065", + "metadata": {}, + "source": [ + "
\n", + "Notes: \n", + " \n", + "* The bright spectrum in NRS1 shows little difference after cleaning.\n", + "\n", + "\n", + "* The default mask generated on-the-fly by the pipeline in NRS2 masks the entire science region, potentially over-masking dark areas that could enhance the background fit. This results in no discernible differences between the original and cleaned data for NRS2. In the following section, we explore the process of generating an alternative mask with the pipeline to address this issue.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "fca43729", + "metadata": {}, + "source": [ + "## 6. Clean up 1/f Noise with NSClean (Alternate Mask) \n", + "
\n", + "\n", + "\n", + "For some data sets, masking the entire science region may excessively mask dark areas of the detector that could be used to improve the background fit. Excessive masking can introduce some high frequency noise in the cleaning process that appears as vertical striping over the spectral traces. \n", + "\n", + "Also, for some data sets, there may be several illuminated regions of the detector that are not masked by the WCS bounding boxes. In the subarray data, note that the mask does not cover the bluest end of the spectral trace. This region is illuminated, but is not calibrated.\n", + "\n", + "In some cases, it may be beneficial to build the mask with an alternate algorithm. Here, we do not use the bounding boxes and instead iteratively mask any data more than 0.1 sigma above the background. For bright sources, this leaves more dark data near the spectral trace and may improve the background fit. \n", + "\n", + "Note, however, that excessive cleaning may impact the continuum level for the spectrum, if too much or too little illuminated data is included in the mask. Again, the generated mask and output spectra should be carefully examined to weigh the benefits of cleaning against the impact on the spectra.\n", + "\n", + "To tune the illumination detection in this mask, try modifying the `n_sigma` parameter below. A higher value will identify less illumination. A lower value will identify more." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a684adcc", + "metadata": {}, + "outputs": [], + "source": [ + "# Set up directory for running NSClean with alternate parameters.\n", + "stage2_nsclean_alternate_dir = \"./stage2_nsclean_alternate/\"\n", + "if not os.path.exists(stage2_nsclean_alternate_dir):\n", + " # Create the directory if it doesn't exist.\n", + " os.makedirs(stage2_nsclean_alternate_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "603f3f1a", + "metadata": {}, + "outputs": [], + "source": [ + "# 1/f noise cleaned data (alternate NSClean pipeline mask).\n", + "# Estimated run time: 66 minutes (may vary).\n", + "start = tt.time()\n", + "\n", + "for indx, i in enumerate(rateints_names):\n", + "\n", + " print(f\"Processing {i}...\")\n", + "\n", + " if \"nrs1\" in i:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs1_bots.json\"\n", + " else:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs2_bots.json\"\n", + "\n", + " Spec2Pipeline.call(\n", + " mast_products_dir + i,\n", + " save_results=True,\n", + " steps={\n", + " # Assigns wavelength solution for spectroscopic data.\n", + " \"assign_wcs\": {\"skip\": False, \"save_results\": True},\n", + " # Removes correlated read noise (1/f noise) from NIRSpec images.\n", + " \"nsclean\": {\n", + " \"skip\": False,\n", + " \"save_mask\": True,\n", + " \"n_sigma\": 0.1,\n", + " \"mask_spectral_regions\": False,\n", + " \"save_results\": True,\n", + " },\n", + " \"extract_2d\": {\"skip\": False}, # Obtains 2D cutouts.\n", + " \"srctype\": {\"skip\": False}, # Assigns source type: 'point' or 'extended'.\n", + " # We opt to skip this step, because it can introduce spurious pixels\n", + " # that can affect the light curve scatter.\n", + " \"flat_field\": {\"skip\": True},\n", + " # We also skip this step, because BOTS exoplanet observations are relative.\n", + " \"photom\": {\"skip\": True},\n", + " # Estimates the flux values of pixels flagged as DO_NOT_USE in 2-D spectra.\n", + " \"pixel_replace\": {\n", + " \"skip\": False,\n", + " \"n_adjacent_cols\": 5,\n", + " \"algorithm\": \"fit_profile\",\n", + " },\n", + " \"extract_1d\": {\n", + " \"skip\": False,\n", + " \"override_extract1d\": extract1dref,\n", + " \"apply_apcorr\": False,\n", + " \"subtract_background\": False,\n", + " },\n", + " # 'dispaxis': 1, 'xstart': 100, 'xstop': 150, 'extract_width': 10},\n", + " },\n", + " output_dir=stage2_nsclean_alternate_dir,\n", + " )\n", + "\n", + " print(f\"Saved {i[:-13]}\" + \"assign_wcs.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"mask.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"nsclean.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"calints.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"x1dints.fits\")\n", + "\n", + "end = tt.time()\n", + "print(\"Run time: \", round(end - start, 1) / 60.0, \" min\")" + ] + }, + { + "cell_type": "markdown", + "id": "7d6e4f6f", + "metadata": {}, + "source": [ + "### 6.1 Verify the Mask (Alternate Mask) \n", + "
\n", + "\n", + "Check the mask against the rate data to make sure it keeps only dark areas of the detector.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a30fb181-981f-4b72-b87c-15a70f4b2c89", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the rate data with masked areas blocked.\n", + "\n", + "# List of on-the-fly built masks from the pipeline.\n", + "nsclean_alternate_masks = [\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs1_mask.fits\",\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs2_mask.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and mask file.\n", + "for rateint_file, mask_file in zip(rateints_names, nsclean_alternate_masks):\n", + " plot_dark_data(\n", + " mast_products_dir + rateint_file, mask_file, slice_index=100, axis=0, scale=9\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "a0c87bac", + "metadata": {}, + "source": [ + "### 6.2 Comparing Original vs. Cleaned Data (Alternate Mask) \n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "caaab2cc", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the original and cleaned data, as well as a residual map.\n", + "\n", + "cleaned_alternate_masks = [\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs1_nsclean.fits\",\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs2_nsclean.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and cleaned file.\n", + "for rateint_file, cleaned_file in zip(rateints_names, cleaned_alternate_masks):\n", + " plot_cleaned_data(\n", + " mast_products_dir + rateint_file, cleaned_file, slice_index=100, layout=\"rows\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "f820fa5a", + "metadata": {}, + "source": [ + "Compare the extracted spectrum from the cleaned data to the spectrum extracted from the original rate file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b155b77e", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# 1D extracted spectra\n", + "x1d_nsclean_skipped = [\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "x1d_nsclean_alternate = [\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_alternate_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_alternate):\n", + " plot_spectra([original, cleaned], ext_num=100, scale_percent=0)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2f2e3d32", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Wavelength region of interest.\n", + "wavelength_range = {\"nrs1\": [3.4, 3.45], \"nrs2\": [4.25, 4.275]}\n", + "flux_range = {\"nrs1\": [1100, 1300], \"nrs2\": [500, 550]}\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_alternate):\n", + " plot_spectra(\n", + " [original, cleaned],\n", + " ext_num=100,\n", + " scale_percent=0,\n", + " wavelength_range=wavelength_range,\n", + " flux_range=flux_range,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "c3086f4c-f868-45e1-a29e-03bf1a0e848e", + "metadata": {}, + "source": [ + "
\n", + "Note: The bright spectrum in NRS1/NRS2 shows little difference after cleaning. However, according to the ratio, the spectrum in NRS1 indicates that the cleaned data may have a slightly higher flux, whereas in NRS2, the cleaned data exhibits a slightly lower flux compared to the original data.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "19c6fa96", + "metadata": {}, + "source": [ + "## 7. Clean up 1/f Noise with NSClean (Hand-Modified Mask) \n", + "
\n", + "\n", + "In certain scenarios, manual generation of a mask may be required. Here, we present **one** approach to manually modify the mask starting with the default mask output from the pipeline. It is worth noting that the mask modified using this method may not necessarily outperform the two previous options." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "523e5d5d", + "metadata": {}, + "outputs": [], + "source": [ + "# Set up directory for running NSClean with user-supplied mask.\n", + "stage2_nsclean_modified_dir = \"./stage2_nsclean_modified/\"\n", + "if not os.path.exists(stage2_nsclean_modified_dir):\n", + " # Create the directory if it doesn't exist.\n", + " os.makedirs(stage2_nsclean_modified_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d950a205-e699-4019-9525-cfbb856c82bb", + "metadata": {}, + "outputs": [], + "source": [ + "# Hand-modify certain mask regions.\n", + "\n", + "# Define the list to store paths of modified masks.\n", + "nsclean_modified_masks = []\n", + "\n", + "# Iterate through the list of original masks.\n", + "for mask in nsclean_default_masks:\n", + " # New mask file name.\n", + " output_file = os.path.basename(mask)[:-5] + \"_modified.fits\"\n", + "\n", + " # Open the FITS file.\n", + " with fits.open(mask) as hdul:\n", + " # Extract the mask data from the science extension.\n", + " mask_data = hdul[\"SCI\"].data.copy() # Make a copy.\n", + "\n", + " # Step 1: Set the default masked regions back to True.\n", + " mask_data[:, :, :] = True\n", + "\n", + " # Step 2: Re-define masked regions by hand.\n", + " if \"nrs1\" in mask:\n", + " mask_data[:, 11:29, 400:700] = False\n", + " mask_data[:, 10:27, 700:800] = False\n", + " mask_data[:, 8:26, 800:1000] = False\n", + " mask_data[:, 7:24, 1000:1200] = False\n", + " mask_data[:, 4:23, 1200:1400] = False\n", + " mask_data[:, 4:22, 1400:1600] = False\n", + " mask_data[:, 3:21, 1600:2043] = False\n", + " else:\n", + " mask_data[:, :16, :290] = False\n", + " mask_data[:, 1:18, 290:740] = False\n", + " mask_data[:, 2:22, 740:1290] = False\n", + " mask_data[:, 8:29, 1290:1840] = False\n", + " mask_data[:, 11:, 1840:] = False\n", + "\n", + " # Update the data within the science extension.\n", + " hdul[\"SCI\"].data = mask_data\n", + " # Save the modified FITS file.\n", + " output_path = os.path.join(stage2_nsclean_modified_dir, output_file)\n", + " hdul_modified = hdul.copy() # Make a copy.\n", + " hdul_modified.writeto(output_path, overwrite=True)\n", + " nsclean_modified_masks.append(output_path)\n", + " print(f\"Saved modified mask as: {output_path}\")" + ] + }, + { + "cell_type": "markdown", + "id": "991ea8e0-9fd4-4217-bbc9-9ec195d697dc", + "metadata": {}, + "source": [ + "### 7.1 Verify the Mask (Hand-Modified Mask) \n", + "
\n", + "\n", + "Check the mask against the rate data to make sure it keeps only dark areas of the detector.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2cce9efc-c5de-4c04-9dd2-7059d459309b", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the rate data with masked areas blocked.\n", + "\n", + "# List of modified masks for the pipeline.\n", + "nsclean_modified_masks = [\n", + " stage2_nsclean_modified_dir\n", + " + \"jw01366003001_04101_00001-seg002_nrs1_mask_modified.fits\",\n", + " stage2_nsclean_modified_dir\n", + " + \"jw01366003001_04101_00001-seg002_nrs2_mask_modified.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and mask file.\n", + "for rateint_file, mask_file in zip(rateints_names, nsclean_modified_masks):\n", + " plot_dark_data(\n", + " mast_products_dir + rateint_file, mask_file, slice_index=100, axis=0, scale=9\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "a5b5a355-ae43-4d4b-a52c-67cda0fb98bd", + "metadata": {}, + "source": [ + "
\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dcdb68f7", + "metadata": {}, + "outputs": [], + "source": [ + "# 1/f noise cleaned data (user-supplied mask).\n", + "# Estimated run time: 57 minutes (may vary).\n", + "start = tt.time()\n", + "\n", + "for indx, i in enumerate(rateints_names):\n", + "\n", + " print(f\"Processing {i}...\")\n", + "\n", + " if \"nrs1\" in i:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs1_bots.json\"\n", + " else:\n", + " extract1dref = os.path.basename(extract_1d_ref)[:-5] + \"_nrs2_bots.json\"\n", + "\n", + " Spec2Pipeline.call(\n", + " mast_products_dir + i,\n", + " save_results=True,\n", + " steps={\n", + " # Assigns wavelength solution for spectroscopic data.\n", + " \"assign_wcs\": {\"skip\": False, \"save_results\": True},\n", + " # Removes correlated read noise (1/f noise) from NIRSpec images.\n", + " \"nsclean\": {\n", + " \"skip\": False,\n", + " \"save_mask\": True,\n", + " \"save_results\": True,\n", + " \"user_mask\": nsclean_modified_masks[indx],\n", + " },\n", + " \"extract_2d\": {\"skip\": False}, # Obtains 2D cutouts.\n", + " \"srctype\": {\"skip\": False}, # Assigns source type: 'point' or 'extended'.\n", + " # We opt to skip this step, because it can introduce spurious pixels\n", + " # that can affect the light curve scatter.\n", + " \"flat_field\": {\"skip\": True},\n", + " # We also skip this step, because BOTS exoplanet observations are relative.\n", + " \"photom\": {\"skip\": True},\n", + " # Estimates the flux values of pixels flagged as DO_NOT_USE in 2-D spectra.\n", + " \"pixel_replace\": {\n", + " \"skip\": False,\n", + " \"n_adjacent_cols\": 5,\n", + " \"algorithm\": \"fit_profile\",\n", + " },\n", + " \"extract_1d\": {\n", + " \"skip\": False,\n", + " \"override_extract1d\": extract1dref,\n", + " \"apply_apcorr\": False,\n", + " \"subtract_background\": False,\n", + " },\n", + " # 'dispaxis': 1, 'xstart': 100, 'xstop': 150, 'extract_width': 10},\n", + " },\n", + " output_dir=stage2_nsclean_modified_dir,\n", + " )\n", + "\n", + " print(f\"Saved {i[:-13]}\" + \"assign_wcs.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"mask.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"nsclean.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"calints.fits\")\n", + " print(f\"Saved {i[:-13]}\" + \"x1dints.fits\")\n", + "\n", + "end = tt.time()\n", + "print(\"Run time: \", round(end - start, 1) / 60.0, \" min\")" + ] + }, + { + "cell_type": "markdown", + "id": "0ae87af7", + "metadata": {}, + "source": [ + "### 7.2 Comparing Original vs. Cleaned Data (Hand-Modified Mask) \n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7cda5133", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Plot the original and cleaned data, as well as a residual map.\n", + "\n", + "cleaned_modified_masks = [\n", + " stage2_nsclean_modified_dir + \"jw01366003001_04101_00001-seg002_nrs1_nsclean.fits\",\n", + " stage2_nsclean_modified_dir + \"jw01366003001_04101_00001-seg002_nrs2_nsclean.fits\",\n", + "]\n", + "\n", + "# Plot each associated set of rateint data and cleaned file\n", + "for rateint_file, cleaned_file in zip(rateints_names, cleaned_modified_masks):\n", + " plot_cleaned_data(\n", + " mast_products_dir + rateint_file, cleaned_file, slice_index=100, layout=\"rows\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "dedd55ae", + "metadata": {}, + "source": [ + "Compare the extracted spectrum from the cleaned data to the spectrum extracted from the original rate file. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f752a0ab", + "metadata": {}, + "outputs": [], + "source": [ + "# 1D extracted spectra.\n", + "x1d_nsclean_skipped = [\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_skipped_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "x1d_nsclean_modified = [\n", + " stage2_nsclean_modified_dir + \"jw01366003001_04101_00001-seg002_nrs1_x1dints.fits\",\n", + " stage2_nsclean_modified_dir + \"jw01366003001_04101_00001-seg002_nrs2_x1dints.fits\",\n", + "]\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_modified):\n", + " plot_spectra([original, cleaned], ext_num=100, scale_percent=0)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dc9927d5", + "metadata": {}, + "outputs": [], + "source": [ + "# Wavelength Region of interest.\n", + "wavelength_range = {\"nrs1\": [3.4, 3.45], \"nrs2\": [4.25, 4.275]}\n", + "flux_range = {\"nrs1\": [1100, 1300], \"nrs2\": [500, 550]}\n", + "\n", + "for original, cleaned in zip(x1d_nsclean_skipped, x1d_nsclean_modified):\n", + " plot_spectra(\n", + " [original, cleaned],\n", + " ext_num=100,\n", + " scale_percent=0,\n", + " wavelength_range=wavelength_range,\n", + " flux_range=flux_range,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "d9f6c22a-3594-4bf9-83ae-e40efaa27011", + "metadata": {}, + "source": [ + "
\n", + "Note: Once more, the bright spectrum in NRS1/NRS2 shows little difference after cleaning. However, according to the ratio, the spectrum in NRS1 indicates that the cleaned data may have a slightly higher flux, whereas in NRS2, the cleaned data exhibits a slightly lower flux compared to the original data.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "9080bb96", + "metadata": {}, + "source": [ + "## 8. Conclusion \n", + "
\n", + "\n", + "The final plots below show the countrate images and the resulting 1D extracted spectra side-by-side to compare the different cleaning methods: the original (no NSClean applied), the cleaned countrate image (with the default pipeline mask), the cleaned countrate image (with an alternate pipeline mask), and finally, the cleaned countrate image (with the hand-modified mask).\n", + "\n", + "Please note that the results presented in this notebook may vary for different datasets (e.g., targets of different brightness, spatial extent, etc.). Users are encouraged to explore NSClean using different masking methods to determine the optimal results.\n", + "\n", + "The output from the cleaning algorithm is now ready for further processing. The (*_calint.fits*) files produced by the above `Spec2Pipeline` run may be used as input to the `TSO3Pipeline`, for generating final combined spectra." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "567a247c-db7f-473a-89e7-8de96dafcf47", + "metadata": {}, + "outputs": [], + "source": [ + "# Not cleaned vs. cleaned (default mask) vs. cleaned (alternate mask) rate data\n", + "original_rate_data = [\n", + " fits.open(mast_products_dir + rate_name)[1].data[100, :, :]\n", + " for rate_name in rateints_names\n", + "]\n", + "cleaned_rate_default_data = [\n", + " fits.open(cleaned_default_mask)[1].data[100, :, :]\n", + " for cleaned_default_mask in cleaned_default_masks\n", + "]\n", + "cleaned_rate_alternate_data = [\n", + " fits.open(cleaned_alternate_mask)[1].data[100, :, :]\n", + " for cleaned_alternate_mask in cleaned_alternate_masks\n", + "]\n", + "cleaned_rate_modified_data = [\n", + " fits.open(cleaned_modified_mask)[1].data[100, :, :]\n", + " for cleaned_modified_mask in cleaned_modified_masks\n", + "]\n", + "\n", + "# For plotting visualization\n", + "for data_list in [\n", + " original_rate_data,\n", + " cleaned_rate_default_data,\n", + " cleaned_rate_alternate_data,\n", + " cleaned_rate_modified_data,\n", + "]:\n", + " for data in data_list:\n", + " data[np.isnan(data)] = 0\n", + "\n", + "# Original vs. cleaned data (with default mask)\n", + "fig, axs = plt.subplots(2, 4, figsize=(25, 8))\n", + "\n", + "# Set y-axis titles and plot the data\n", + "titles = [\n", + " \"Original Rate Data\",\n", + " \"Cleaned Rate Data (Default Mask)\",\n", + " \"Cleaned Rate Data (Alternate Mask)\",\n", + " \"Cleaned Rate Data (Hand-Modified Mask)\",\n", + "]\n", + "for i, (data_list, title) in enumerate(\n", + " zip(\n", + " [\n", + " original_rate_data,\n", + " cleaned_rate_default_data,\n", + " cleaned_rate_alternate_data,\n", + " cleaned_rate_modified_data,\n", + " ],\n", + " titles,\n", + " )\n", + "):\n", + " for j, data in enumerate(data_list):\n", + " ax = axs[j, i]\n", + " if j == 0:\n", + " ax.set_title(f'{title} \\n {\"Integration[100,:,:] | NRS1\"}', fontsize=12)\n", + " else:\n", + " ax.set_title(f'{title} \\n {\"Integration[100,:,:] | NRS2\"}', fontsize=12)\n", + " im = ax.imshow(data, origin=\"lower\", aspect=30, clim=(-1e-1, 1e-1))\n", + " fig.colorbar(im, ax=ax, pad=0.05, shrink=0.7, label=\"DN/s\")\n", + " ax.set_xlabel(\"Pixel Column\", fontsize=10)\n", + " ax.set_ylabel(\"Pixel Row\", fontsize=10)\n", + "\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b841db95", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Final Comparison\n", + "plot_spectra(\n", + " [\n", + " x1d_nsclean_skipped[0],\n", + " x1d_nsclean_default[0],\n", + " x1d_nsclean_alternate[0],\n", + " x1d_nsclean_modified[0],\n", + " ],\n", + " ext_num=100,\n", + " wavelength_range=wavelength_range,\n", + " flux_range=flux_range,\n", + ")\n", + "plot_spectra(\n", + " [\n", + " x1d_nsclean_skipped[1],\n", + " x1d_nsclean_default[1],\n", + " x1d_nsclean_alternate[1],\n", + " x1d_nsclean_modified[1],\n", + " ],\n", + " ext_num=100,\n", + " wavelength_range=wavelength_range,\n", + " flux_range=flux_range,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "65b997a0-6cd9-409c-b20a-b65d5a650b8f", + "metadata": {}, + "source": [ + "
\n", + "Final Note: Both the alternate (clip-based) and hand-modified masks produced similar results. However, the hand-modified mask appears to introduce less high-frequency noise.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "d3f1c98f-b929-46bc-aa98-38ae1f7c47c7", + "metadata": {}, + "source": [ + "## About the Notebook \n", + "\n", + "**Authors:** Melanie Clarke, Kayli Glidic; NIRSpec Instrument Team\n", + "\n", + "**Updated On**: Feburary 29, 2024.\n", + "\n", + "
\n", + "\n", + "[Top of Page](#top)\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.8" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/NIRSpec_NSClean/requirements.txt b/notebooks/NIRSpec_NSClean/requirements.txt index c037a19b3..bad3b89cc 100644 --- a/notebooks/NIRSpec_NSClean/requirements.txt +++ b/notebooks/NIRSpec_NSClean/requirements.txt @@ -1,7 +1,7 @@ -numpy == 1.26.4 +numpy == 1.25.2 requests == 2.31.0 -jwst == 1.13.4 +jwst == 1.14.0 crds == 11.17.19 -stpipe == 0.5.1 -matplotlib == 3.8.3 -astropy == 6.0.0 \ No newline at end of file +stpipe == 0.5.2 +matplotlib == 3.8.4 +astropy == 6.0.1 \ No newline at end of file