Refactored and improved crds_setup_env into 3 pieces:

crds_setup_all  -- installs both base conda and CRDS s/w + dependencies

crds_setup_env  -- installs base conda and creates crds-env

crds_setup_crds -- installs CRDS + dependencies into current env

The new setup scripts normally require no parameters.

The new env setup enables selection of OS shell (bash,csh,zsh) and
ienstall location.

The new env setup enables selection of the install directory.

Pip installs now also --upgrade.

Refectored dependency groups some to streamline submission+test install.
Added bandit and jupyterlab as test + dev dependencies.
Updated README.rst

README.rst

@@ -22,10 +22,11 @@ CRDS is also available for installation as part of AstroConda Contrib:
 Basic CRDS Installation
 -----------------------
 
-For many applications CRDS is automatically installed as a dependency of the
+For many roles, CRDS is *automatically installed as a dependency* of the
 calibration software.  This default installation supports running calibrations
-but not more advanced CRDS activities like submitting files or development. You
-can test for the installation of CRDS like this::
+but not more advanced CRDS activities like submitting files or development.
+
+You can test for an existing installation of CRDS like this::
 
   $ crds list --status
   CRDS Version = '7.4.0, b7.4.0, daf308e24c8dd37e70c89012e464058861417245'
@@ -42,85 +43,114 @@ can test for the installation of CRDS like this::
 This output indicates CRDS is installed and configured for processing onsite
 using a pre-built cache of CRDS rules and references at */grp/crds/cache*.
 
-Full Environment Install
-++++++++++++++++++++++++
-While the Advanced Installation instructions below detail the installation
-of a complete CRDS evironment,  you may find it more convenient to clone
-CRDS from github and run the *crds_setup_env* script which performs the same
-steps::
+File Submission Installation
+----------------------------
+
+For performing the file submission role,  CRDS includes additional dependencies
+and can be trickier to install.
+
+Addding CRDS to an Existing Environment
++++++++++++++++++++++++++++++++++++++++
+
+You can install/upgrade CRDS and it's dependencies in your current environment
+like this::
 
   git clone https://github.com/spacetelescope/crds.git
   cd crds
-  ./crds_setup_env os-x  # or linux
+  ./crds_setup_crds
 
-Accept all default responses to installation script prompts.
+It is recommended that you only do this in an environment dedicated to file
+submissions.   This may be suitable for e.g. installing/upgrading CRDS in
+an active *redcatconda* environment.
 
-Advanced CRDS Installation
---------------------------
+Full Environment Install
+++++++++++++++++++++++++
 
-Users performing more advanced activities like CRDS file submissions or
-development will need additional CRDS dependencies.
+Sometimes it's expedient to install an entirely new environment including a
+baseline conda,  CRDS,  and all of it's dependencies.  To start from scratch,
+you can::
 
-Install Conda
-+++++++++++++
+  git clone https://github.com/spacetelescope/crds.git
+  cd crds
+  ./crds_setup_all
 
-Typical CRDS installations are based on installing a barebones Miniconda
-environment and then adding most packages via pip::
+  # open a new terminal window
+  conda activate crds-env
 
-  if [ "$1" == "os-x" ]; then
-    export CONDA_INSTALLER=Miniconda3-latest-MacOSX-x86_64.sh
-  elif [ "$1" == "linux" ]; then
-      export CONDA_INSTALLER=Miniconda3-latest-Linux-x86_64.sh
-  else
-      echo "usage:  crds_setup_env  [os-x|linux]"
-  fi
-  echo "Installing $CONDA_INSTALLER"
+To customize a bit more, *crds_setup_all* and *crds_setup_env* support
+parameters which can be used to specify OS, shell, and install location.
+Substitute the below to specify Linux, c-shell, and a non-default install
+location::
 
-  rm -rf ~/miniconda3.old
-  mv ~/miniconda3 ~/miniconda3.old
-  rm -f Miniconda3-latest-*.sh*
+  ./crds_setup_all   Linux  csh   $HOME/miniconda_crds
 
-  curl https://repo.anaconda.com/miniconda/${CONDA_INSTALLER}  >${CONDA_INSTALLER}
-  sh ${CONDA_INSTALLER}
-  rm -f ${CONDA_INSTALLER}
-  conda update --yes -n base -c  conda
+Advanced Install
+++++++++++++++++
 
-Accept all default responses to the install script's quesions.
+Below are the current sub-tasks used conceptually for a full featured CRDS
+install.    These can serve as an alternative to cloning the CRDS repo and
+running the install script(s).  If you already have a python environment
+supporting pip,
 
-Add Astroconda channel
-++++++++++++++++++++++
-Some CRDS dependencies are only available through the conda channel Astroconda.
-Configure the conda environment to use Astroconda::
+1. Installing Conda / Astroconda
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-  conda config --add channels http://ssb.stsci.edu/astroconda
+Alternate / definitive installation instructions for installing a baseline conda
+can be found here::
 
-Create crds-env Environment
-+++++++++++++++++++++++++++
+  https://spacetelescope.github.io/training-library/computer_setup.html#installing-conda
+
+2. Create crds-env Environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The CRDS software and basic conda dependencies should be installed in an
 isolated conda environment::
 
-  conda create -n crds-env python=3.7
+  conda create -n crds-env
   conda activate crds-env
 
-Add JWST CAL S/W and Dependencies
-+++++++++++++++++++++++++++++++++
-Installing the JWST CAL S/W will also automatically install many dendencies
-of a numerical computing environment::
+You can substitute the environment name of your choice, e.g. *redcatconda* vs. *crds-env*.
+
+3. Add JWST CAL S/W and Dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Installing the JWST CAL S/W will also automatically install many dendencies of
+a numerical computing environment::
 
-  pip install numpy
-  pip install git+https://github.com/spacetelescope/jwst
+  pip install --upgrade numpy
+  pip install --upgrade git+https://github.com/spacetelescope/jwst
+
+Note that these commands also install the latest version of CRDS from pip which
+may not be current enough for ongoing reference file testing and
+troubleshooting.
+
+4. Install CRDS and Dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Install CRDS and Dependencies
-+++++++++++++++++++++++++++++
 This sequence first removes the CRDS installed automatically as part of
 installing the *jwst* package and then installs the latest available CRDS
 from github with advanced dependencies not needed for basic operation::
 
   pip uninstall --yes crds
-  pip install git+https://github.com/spacetelescope/crds.git#egg=crds["submission","test","dev","docs"]
+  pip install --upgrade  git+https://github.com/spacetelescope/crds.git#egg=crds["submission","test"]
+
+A more full featured CRDS install is::
+
+  pip install --upgrade  git+https://github.com/spacetelescope/crds.git#egg=crds["submission","dev","test","docs"]
+
+5. Install Fitsverify
+^^^^^^^^^^^^^^^^^^^^^
+
+Since it is a C-based package fitsverify is not available using pip but is
+available via conda on the astroconda channel::
+
+  conda config --add channels http://ssb.stsci.edu/astroconda
   conda install --yes fitsverify
 
+As part of an end-user setup installation of fitsverify is optional, CRDS
+certify will run without it after issuing a warning, the CRDS server will run
+fitsverify as part of its checks unless/until we stop using it altogether.
+
 User's Guide
 ------------
 

crds_setup_all

@@ -0,0 +1,10 @@
+#! /bin/bash
+
+./crds_setup_env $*  ||  exit -1
+
+
+export WHERE=${3:-"$HOME/miniconda3"}
+source $WHERE/etc/profile.d/conda.sh
+conda activate crds-env
+
+source ./crds_setup_crds

crds_setup_crds

@@ -0,0 +1,11 @@
+#! /bin/bash
+
+pip install --upgrade numpy
+pip install --upgrade git+https://github.com/spacetelescope/jwst
+pip uninstall --yes crds
+
+# pip install git+https://github.com/spacetelescope/crds.git#egg=crds["submission","test"]
+pip install --upgrade .["submission","test"]
+
+conda config --add channels http://ssb.stsci.edu/astroconda
+conda install --yes fitsverify

crds_setup_env

@@ -1,31 +1,47 @@
 #! /bin/bash
 
-if [ "$1" == "os-x" ]; then
-    export CONDA_INSTALLER=Miniconda3-latest-MacOSX-x86_64.sh
-elif [ "$1" == "linux" ]; then
-    export CONDA_INSTALLER=Miniconda3-latest-Linux-x86_64.sh
-else
-    echo "usage:  crds_setup_env  [os-x|linux]"
-    exit -1
-fi
-echo "Installing $CONDA_INSTALLER"
+export OS=${1:-`uname -s`}
+export SHELL=${2:-bash}
+export WHERE=${3:-"$HOME/miniconda3"}
 
+case $OS in
+    Darwin)
+        export CONDA_INSTALLER=Miniconda3-latest-MacOSX-x86_64.sh
+        ;;
+    Linux)
+        export CONDA_INSTALLER=Miniconda3-latest-Linux-x86_64.sh
+        ;;
+    *)
+        echo "usage:  $0  [Darwin|Linux] [bash|csh|zsh] [path_to_install_dir]"
+        exit -1
+        ;;
+esac
+
+case $SHELL in
+    bash)
+    ;;
+    csh)
+    ;;
+    zsh)
+    ;;
+    *)
+        echo "Invalid SHELL selected = $SHELL"
+        exit -1
+        ;;
+esac
+
+echo "Installing   OS=$OS  SHELL=$SHELL  WHERE=$WHERE"
+
+rm -rf $WHERE
 rm -f ${CONDA_INSTALLER}*
 wget --no-check-certificate https://repo.anaconda.com/miniconda/${CONDA_INSTALLER}
 
-bash ${CONDA_INSTALLER}  -b -p $HOME/miniconda3
+bash ${CONDA_INSTALLER}  -b -p $WHERE
 rm -f ${CONDA_INSTALLER}
 
-conda update --yes -n base -c  conda
-conda config --add channels http://ssb.stsci.edu/astroconda
+source $WHERE/etc/profile.d/conda.sh
+conda update --yes -n base -c defaults conda
+conda init $SHELL
+
 conda create --yes -n crds-env python=3.7 # conda
 conda activate crds-env
-
-pip install numpy
-pip install git+https://github.com/spacetelescope/jwst
-
-pip uninstall --yes crds
-# pip install git+https://github.com/spacetelescope/crds.git#egg=crds["submission","test","dev","docs"]
-pip install .["submission","test","dev","docs"]
-
-conda install --yes fitsverify

setup.py

@@ -78,6 +78,8 @@
     "scripts" : glob.glob("scripts/*"),
     }
 
+TEST_DEPS = ["lockfile", "mock", "nose", "pytest", "pylint", "flake8", "bandit"]
+
 setup(name="crds",
       provides=["crds"],
       version = '7.4.1.1',
@@ -91,12 +93,12 @@
       extras_require={
           "jwst": ["jwst"],
           "submission": ["requests", "lxml", "parsley"],
-          "dev" : ["ipython", "jupyter"],
-          "test" : ["lockfile", "mock", "nose", "pytest", "pylint", "flake8"],
+          "dev" : ["ipython","jupyterlab"],
+          "test" : TEST_DEPS,
           "docs" : ["sphinx","sphinx_rtd_theme","docutils"],
           "aws" : ["boto3"]
       },
-      tests_require=["lockfile", "mock", "nose", "pytest", "pylint", "flake8"],
+      tests_require=TEST_DEPS,
       zip_safe=False,
       classifiers=[
           'Intended Audience :: Science/Research',