Skip to content

Commit

Permalink
Merge branch 'develop' into dark-monitor-integ-limits-and-dates
Browse files Browse the repository at this point in the history
  • Loading branch information
bhilbert4 authored Dec 20, 2023
2 parents 6ac77eb + 3c27420 commit 790db8c
Show file tree
Hide file tree
Showing 89 changed files with 5,416 additions and 3,514 deletions.
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "monthly"
89 changes: 32 additions & 57 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,78 +2,53 @@ name: JWQL CI

on: [push, pull_request]

jobs:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

JWQL-CI:
defaults:
run:
shell: bash -l {0}

name: Python - ${{ matrix.python-version }}
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- run: pip install bandit
- run: bandit ./jwql/ -c .bandit

test:
name: test (Python ${{ matrix.python-version }}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
max-parallel: 5
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: [3.8, 3.9]
python-version: [3.9, "3.10"]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- uses: conda-incubator/setup-miniconda@v2
- uses: mamba-org/provision-with-micromamba@v15
with:
activate-environment: jwql-${{ matrix.python-version }}
environment-file: ./environment_python_${{ matrix.python-version }}.yml
cache-env: true
cache-downloads: true

- name: Bandit Check
uses: jpetrucciani/bandit-check@master
with:
path: "./jwql/"
bandit_flags: "-c .bandit"
continue-on-error: false
if: runner.os == 'Linux'
- run: pip install -e .[test]

- name: Start Redis
uses: supercharge/[email protected]
- run: conda env export

- uses: supercharge/[email protected]
with:
redis-version: 5.0
if: runner.os == 'Linux'

- name: Build jwql conda environment and run tests
shell: bash -l {0}
run: |
echo "Setting useful environment variables:"
echo " "
export PYTHONVERSION=${{ matrix.python-version }}
export MAJOR="$(echo $PYTHONVERSION | head -c 1)"
export MINOR="$(echo $PYTHONVERSION | tail -c 2)"
echo "PYTHONVERSION: $PYTHONVERSION"
echo " "
echo "Installing jwql conda environment"
echo " "
cd $RUNNER_WORKSPACE/
cd jwql/
$CONDA/bin/conda env update -f environment_python_${MAJOR}_${MINOR}.yml -n jwql-${{ matrix.python-version }}
echo " "
echo "Installing jwql package"
echo " "
python setup.py develop
echo " "
echo "Testing package installation"
echo " "
python -c "import jwql; print('Version: ' + jwql.__version__); print('Path: ' + jwql.__path__[0])"
# echo " "
# echo "Reinstall numpy to fix numpy.core.multiarray error"
# echo " "
pip install -U numpy
echo " "
echo "The conda environment being used:"
echo " "
$CONDA/bin/conda env export
- run: python -c "import jwql; print('Version ' + jwql.__version__); print('Path ' + jwql.__path__[0])"

echo " "
echo "Running pytests"
echo " "
pytest jwql/tests/
- run: pytest jwql/tests/
89 changes: 89 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,92 @@
## What's Changed

1.2.1 (2023-12-20)
==================

Web Application
~~~~~~~~~~~~~~~
- Visit Filter on query and archive thumbnails pages by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1412

Project & API Documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- NIRCam Claw Monitor by @bsunnquist in https://github.com/spacetelescope/jwql/pull/1152
- Stop storing absolute paths in the database by @york-stsci in https://github.com/spacetelescope/jwql/pull/1394


1.2.0 (2023-11-21)
==================

Web Application
~~~~~~~~~~~~~~~
- Hover over proposal thumbnails by @shanosborne in https://github.com/spacetelescope/jwql/pull/927
- Not found images replaced with default image by @BradleySappington in https://github.com/spacetelescope/jwql/pull/949
- Remove second set of colorbar tick labels from preview images by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/963
- Split program page by obsnum by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/948
- Interactive preview images by @BradleySappington in https://github.com/spacetelescope/jwql/pull/965
- Alert user to non-existant proposal number by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/994
- Fix bad thumbnails by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/997
- Explore by group type by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1000
- Default scaling params for preview images where all pix are NaN by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1006
- Explore int grp difference by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1020
- Add inst select home search by @mengesser in https://github.com/spacetelescope/jwql/pull/1010
- Allow proposal thumbnails to come from dark thumbnails by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1001
- Adding Next and Previous Buttons by @mfixstsci in https://github.com/spacetelescope/jwql/pull/978
- Build archive page from database by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1030
- Select anomaly name formatting by @rcooper295 in https://github.com/spacetelescope/jwql/pull/1053
- Implement "Viewed" Button and Filtering by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1049
- fixes for no data found on server by @penaguerrero in https://github.com/spacetelescope/jwql/pull/1042
- Filter archive page by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1065
- Exposure page: Default to show rate img, keep order of radio buttons constant by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1111
- Filter type by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1092
- Sort obs date by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1113
- added scattered light, light saber anomalies for niriss by @rcooper295 in https://github.com/spacetelescope/jwql/pull/1133
- Add new NIRCam anomalies by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1136
- Add Logging View by @mfixstsci in https://github.com/spacetelescope/jwql/pull/1080
- Remove bokeh templating from Dark monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1153
- Fix API view for filenames per proposal and preview images by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1046
- Create archive page asynchronously by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1203
- Adding sort by category by @mfixstsci in https://github.com/spacetelescope/jwql/pull/1155
- Improve bad pixel monitor plots by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1202
- List viewed/new status by instrument by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1197
- Display all detectors by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1213
- Remove bokeh templating from Bias Monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1230
- Reformat query page functionality by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1240
- Add download links to image pages by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1283
- Improve explore image by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1282
- Remove Bokeh Templating from Readnoise Monitor by @mfixstsci in https://github.com/spacetelescope/jwql/pull/1238
- Move Date Range selection to Query Page by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1396

Project & API Documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Make Readnoise Monitor less memory-intensive by @bsunnquist in https://github.com/spacetelescope/jwql/pull/922
- Speed up MAST queries by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/942
- Lock python by @BradleySappington in https://github.com/spacetelescope/jwql/pull/955
- Reduce memory usage by the dark monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/908
- Fix clipping logic in preview image scaling by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1011
- Cosmic ray monitor (via mengesser:develop) by @bourque in https://github.com/spacetelescope/jwql/pull/700
- nirspec TA monitors: wata and msata by @penaguerrero in https://github.com/spacetelescope/jwql/pull/888
- allow per-instrument database table reset by @york-stsci in https://github.com/spacetelescope/jwql/pull/1031
- Celery monitors by @york-stsci in https://github.com/spacetelescope/jwql/pull/921
- Raise the upper limit on number of returned Mast entries by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1048
- Added chunked iteration to CR monitor by @york-stsci in https://github.com/spacetelescope/jwql/pull/1066
- Created a clone_tables script by @york-stsci in https://github.com/spacetelescope/jwql/pull/1059
- Add EDB telemetry monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/832
- Tweaks to mnemonics queried by the EDB monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1115
- Fix failing MIRI dark montior by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1129
- Unit test cleanup by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1167
- Catch missing obs in navigation data by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1206
- Update preview image generator to handle new NaN pixels by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1212
- drop support for Python 3.8 by @zacharyburnett in https://github.com/spacetelescope/jwql/pull/1249
- Adding Python 3.10 Support to Testing Matrix by @mfixstsci in https://github.com/spacetelescope/jwql/pull/1257
- Remove mast monitor by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1256
- Improve general JWQL queries by @melanieclarke in https://github.com/spacetelescope/jwql/pull/1281
- Prepare models for external postgres db by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1287
- Pipeline skip already run steps by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1262
- Include migrations in source control by @BradleySappington in https://github.com/spacetelescope/jwql/pull/1364
- Add step_args to calwebb_detector1_save_jump by @bhilbert4 in https://github.com/spacetelescope/jwql/pull/1358

--Full Changelog--: https://github.com/spacetelescope/jwql/compare/1.1.0...1.2.0

Unreleased
==========

Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ include LICENSE
include CHANGES.rst
include environment_python_3_7.yml
include environment_python_3_8.yml
include setup.py
include pyproject.toml
include requirements.txt
include rtd_requirements.txt
include jwql/example_config.json
Expand Down
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@ Getting `jwql` up and running on your own computer requires four steps, detailed

### Prerequisites

It is highly suggested that contributors have a working installation of `anaconda` or `miniconda` for Python 3.8. Downloads and installation instructions are available here:
It is highly suggested that contributors have a working installation of `anaconda` or `miniconda` for Python 3.9+. Downloads and installation instructions are available here:

- [Miniconda](https://conda.io/miniconda.html)
- [Anaconda](https://www.continuum.io/downloads)

Requirements for contributing to the `jwql` package will be included in the `jwql` `conda` environment, which is included in our installation instructions below. Further package requirements will be provided for `jwql` by a `setup.py` script included in the repository.
Requirements for contributing to the `jwql` package will be included in the `jwql` `conda` environment, which is included in our installation instructions below. Further package requirements will be provided for `jwql` by a `pyproject.toml` file included in the repository.

### Clone the `jwql` repo

Expand Down Expand Up @@ -83,16 +83,16 @@ source activate base/root

**Note:** If you have added a step activating conda to your default terminal/shell (e.g. the `.bashrc`, `.zshrc`, or `.profile` file) then you don't need to do the above step.

Lastly, create the `jwql` environment via one of the `environment.yml` files (currently `environment_python_3_8.yml`, for python 3.8, and `environment_python_3.9.yml`, for python 3.9, are supported by `jwql`):
Lastly, create the `jwql` environment via one of the `environment.yml` files (currently `environment_python_3.9.yml`, for python 3.9, and `environment_python_3.10.yml`, for python 3.10, are supported by `jwql`):

```
conda env create -f environment_python_3_8.yml
conda env create -f environment_python_3.9.yml
```

or

```
conda env create -f environment_python_3_9.yml
conda env create -f environment_python_3.10.yml
```

### Configuration File
Expand Down Expand Up @@ -154,10 +154,11 @@ Any questions about the `jwql` project or its software can be directed to `jwql@
- Mees Fix (Technical Lead, INS) [@mfixstsci](https://github.com/mfixstsci)
- Misty Cracraft (INS) [@cracraft](https://github.com/cracraft)
- Mike Engesser (INS) [@mengesser](https://github.com/mengesser)
- Shannon Osborne (INS) [@shanosborne](https://github.com/shanosborne)
- Maria Pena-Guerrero [@penaguerrero](https://github.com/penaguerrero)
- Ben Sunnquist (INS) [@bsunnquist](https://github.com/bsunnquist)
- Brian York (INS) [@york-stsci](https://github.com/york-stsci)
- Bradley Sappington (INS) [@bradleysappington](https://github.com/bradleysappington)
- Melanie Clarke (INS) [@melanieclarke](https://github.com/melanieclarke)

## Past Development Team Members
- Matthew Bourque (INS) [@bourque](https://github.com/bourque)
Expand All @@ -168,7 +169,7 @@ Any questions about the `jwql` project or its software can be directed to `jwql@
- Sara Ogaz (DMD) [@SaOgaz](https://github.com/SaOgaz)
- Catherine Martlin (INS) [@catherine-martlin](https://github.com/catherine-martlin)
- Johannes Sahlmann (INS) [@Johannes-Sahlmann](https://github.com/johannes-sahlmann)

- Shannon Osborne (INS) [@shanosborne](https://github.com/shanosborne)

## Acknowledgments:
- Faith Abney (DMD)
Expand Down
6 changes: 0 additions & 6 deletions docs/source/jwql_monitors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,3 @@ monitor_filesystem.py
.. automodule:: jwql.jwql_monitors.monitor_filesystem
:members:
:undoc-members:

monitor_mast.py
---------------
.. automodule:: jwql.jwql_monitors.monitor_mast
:members:
:undoc-members:
4 changes: 2 additions & 2 deletions docs/source/tests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,9 @@ test_logging_functions.py
:members:
:undoc-members:

test_monitor_mast.py
test_mast_utils.py
--------------------
.. automodule:: jwql.tests.test_monitor_mast
.. automodule:: jwql.tests.test_mast_utils
:members:
:undoc-members:

Expand Down
6 changes: 0 additions & 6 deletions docs/source/website.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,6 @@ data_containers.py
:members:
:undoc-members:

db.py
-----
.. automodule:: jwql.website.apps.jwql.db
:members:
:undoc-members:

forms.py
--------
.. automodule:: jwql.website.apps.jwql.forms
Expand Down
Loading

0 comments on commit 790db8c

Please sign in to comment.