Added URL checker and corresponding CI job. Removed travis related fi…

…les. Replaced hard-coded links in documentation by cross-linking directives.

Signed-off-by: Daniel Scheffler <[email protected]>

.gitlab-ci.yml

@@ -45,6 +45,15 @@ test_styles:
     when: always
 
 
+test_urls:
+  stage: test
+  script:
+    - source /root/miniconda3/bin/activate ci_env
+    - pip install -U urlchecker
+    - make urlcheck
+  when: always
+
+
 test_spechomo_install:
   stage: test
   script:

Makefile

@@ -58,6 +58,9 @@ lint: ## check style with flake8
 	pycodestyle spechomo --exclude="*.ipynb,*.ipynb*" --max-line-length=120 > ./tests/linting/pycodestyle.log
 	-pydocstyle spechomo > ./tests/linting/pydocstyle.log
 
+urlcheck: ## check for dead URLs
+	urlchecker check . --file-types .py,.rst,.md,.json
+
 test: ## run tests quickly with the default Python
 	python setup.py test
 

README.rst

@@ -21,17 +21,6 @@ SpecHomo - Spectral homogenization of multispectral satellite data
 Status
 ------
 
-.. .. image:: https://img.shields.io/travis/danschef/spechomo.svg
-        :target: https://travis-ci.org/danschef/spechomo
-
-.. .. image:: https://readthedocs.org/projects/spechomo/badge/?version=latest
-        :target: https://spechomo.readthedocs.io/en/latest/?badge=latest
-        :alt: Documentation Status
-
-.. .. image:: https://pyup.io/repos/github/danschef/spechomo/shield.svg
-     :target: https://pyup.io/repos/github/danschef/spechomo/
-     :alt: Updates
-
 .. image:: https://gitext.gfz-potsdam.de/geomultisens/spechomo/badges/master/pipeline.svg
         :target: https://gitext.gfz-potsdam.de/geomultisens/spechomo/commits/master
 .. image:: https://gitext.gfz-potsdam.de/geomultisens/spechomo/badges/master/coverage.svg

docs/conf.py

@@ -47,7 +47,8 @@
     'sphinx.ext.todo',
     'sphinxarg.ext',
     'sphinx_autodoc_typehints',
-    'sphinx.ext.intersphinx'
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.autosectionlabel'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -136,10 +137,14 @@ def setup(app):
 
 # Add mappings for intersphinx extension (allows to link to the API reference of other sphinx documentations)
 intersphinx_mapping = {
-    'geoarray': ('http://danschef.gitext.gfz-potsdam.de/geoarray/doc/', None),
-    'python': ('http://docs.python.org/3', None),
+    'geoarray': ('https://danschef.gitext-pages.gfz-potsdam.de/geoarray/doc/', None),
+    'python': ('https://docs.python.org/3', None),
 }
 
+# Configure the autosectionlabel extension
+autosectionlabel_prefix_document = True
+autosectionlabel_maxdepth = 10
+
 
 # -- Options for HTML output -------------------------------------------
 

docs/installation.rst

@@ -65,4 +65,3 @@ you through the process.
 .. _pip: https://pip.pypa.io
 .. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
 .. _conda: https://conda.io/docs
-.. _`dependencies of SpecHomo`: https://gitext.gfz-potsdam.de/danschef/arosics/-/blob/master/requirements.txt

docs/usage/available_transformations.rst

@@ -5,13 +5,12 @@ The classifiers included in the SpecHomo package
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The SpecHomo package provides a set of classifiers that can be used for spectral homogenization of various sensor
-combinations (see `here <https://geomultisens.gitext-pages.gfz-potsdam.de/spechomo/doc/usage/
-available_transformations.html#which-sensor-transformations-are-available>`__). These classifiers have been trained
-based on simulated multispectral remote sensing images created by spectral convolution from hyperspectral airborne
-datasets. These hyperspectral datasets were chosen with the intention to include as many different land-cover types
-as possible in order to maximize spectral variability. This makes these classifiers suitable for many application
-fields. Details on the underlying datasets and the used algorithms to create the classifiers can be found
-in `Scheffler et al. 2020 <https://doi.org/10.1016/j.rse.2020.111723>`__.
+combinations (see :ref:`here <usage/available_transformations:Which sensor transformations are available?>`). These
+classifiers have been trained based on simulated multispectral remote sensing images created by spectral convolution
+from hyperspectral airborne datasets. These hyperspectral datasets were chosen with the intention to include as many
+different land-cover types as possible in order to maximize spectral variability. This makes these classifiers suitable
+for many application fields. Details on the underlying datasets and the used algorithms to create the classifiers can
+be found in `Scheffler et al. 2020 <https://doi.org/10.1016/j.rse.2020.111723>`__.
 
 
 Which sensor transformations are available?

docs/usage/homogenization.rst

@@ -7,11 +7,11 @@ class. Please see the linked content for a full documentation of this class.
 
 For the sake of simplicity, the usage of this class is described below, at the
 **example of Landsat-8 data, spectrally adapted to Sentinel-2A**. Transformations between various other sensors are
-possible, see `here <https://geomultisens.gitext-pages.gfz-potsdam.de/spechomo/doc/usage/available_transformations.html
-#which-sensor-transformations-are-available>`__.
+possible, see :ref:`usage/available_transformations:Which sensor transformations are available?`.
 
-First, load the Landsat-8 `surface reflectance`_ image that you want to transform to the spectral domain of Sentinel-2A
-(we use the `geoarray`_ library for this - it is installed with SpecHomo):
+First, load the Landsat-8 :ref:`surface-reflectance <usage/input_data_requirements:Surface Reflectance>` image that you
+want to transform to the spectral domain of Sentinel-2A (we use the `geoarray`_ library for this - it is installed with
+SpecHomo):
 
 .. code-block:: python
 
@@ -23,8 +23,7 @@ First, load the Landsat-8 `surface reflectance`_ image that you want to transfor
 
     Please make sure, that the Landsat-8 input image contains the right bands in the correct order before you run the
     homogenization! By running the :func:`list_available_transformations<spechomo.utils.list_available_transformations>`
-    function as described
-    `here <https://geomultisens.gitext-pages.gfz-potsdam.de/spechomo/doc/usage/available_transformations.html>`__, you can
+    function as described :doc:`here <./available_transformations>`, you can
     find out, that the needed band list is ['1', '2', '3', '4', '5', '6', '7']. These band numbers refer to the
     official provider band-names as described for Landsat at the
     `USGS website <https://www.usgs.gov/faqs/what-are-band-designations-landsat-satellites>`__.
@@ -66,9 +65,7 @@ follows:
     * You can directly copy/paste possible input parameters for the
       :meth:`predict_by_machine_learner<spechomo.prediction.SpectralHomogenizer.predict_by_machine_learner>` method
       from the :func:`list_available_transformations<spechomo.utils.list_available_transformations>`
-      function as described
-      `here <https://geomultisens.gitext-pages.gfz-potsdam.de/spechomo/doc/usage/available_transformations.html
-      #which-sensor-transformations-are-available>`__.
+      function as described :ref:`here <usage/available_transformations:Which sensor transformations are available?>`.
     * You may also save the homogenization results to other GDAL compatible image formats
       (see :meth:`geoarray.GeoArray.save` for details).
     * Further explanation on input parameters like `method`, `n_clusters`, `classif_alg` or `global_clf_threshold` is
@@ -77,5 +74,4 @@ follows:
       evaluation of the different homogenization algorithms available in the SpecHomo library.
 
 
-.. _`surface reflectance`: https://geomultisens.gitext-pages.gfz-potsdam.de/spechomo/doc/usage/input_data_requirements.html#surface-reflectance
 .. _`geoarray`: https://gitext.gfz-potsdam.de/danschef/geoarray

setup.py

@@ -59,7 +59,7 @@
 
 req_setup = ['setuptools-git']  # needed for package_data version-controlled by GIT
 
-req_test = ['coverage', 'nose', 'nose2', 'nose-htmloutput', 'rednose']
+req_test = ['coverage', 'nose', 'nose2', 'nose-htmloutput', 'rednose', 'urlchecker']
 
 req_doc = ['sphinx-argparse', 'sphinx_rtd_theme', 'sphinx-autodoc-typehints']
 

spechomo/classifier.py

@@ -251,7 +251,7 @@ def predict_weighted_averages(self, im_src, cmap_3D, weights_3D=None, nodataVal=
         """Predict target satellite spectral information using separate prediction coefficients for spectral clusters.
 
         NOTE:   This version of the prediction function uses the prediction coefficients of multiple spectral clusters
-                and computes the result as weighted average of them. Therefore, the classifcation map must assign
+                and computes the result as weighted average of them. Therefore, the classification map must assign
                 multiple spectral cluster to each input pixel.
 
         # NOTE:   At unclassified pixels (cmap_3D[y,x,z>0] == -1) the prediction result using global coefficients
@@ -454,7 +454,6 @@ def __getitem__(self, item):
         except KeyError:
             raise(KeyError("The classifier has no key '%s'. Available keys are: %s"
                            % (item, repr(self))))
-
     # def save_to_json(self, filepath):
     #     a = 1
     #     pass

spechomo/training_data.py

@@ -278,7 +278,7 @@ def add_spectra(self, spectra, src_imname, LayerBandsAssignment):
             "%s != %s" % (LayerBandsAssignment, self.LayerBandsAssignment)
 
         # reshape 2D spectra array to one image column (refcube is an image with spectral information in the 3rd dim.)
-        im_col = spectra.reshape(spectra.shape[0], 1, spectra.shape[1])
+        im_col = spectra.reshape((spectra.shape[0], 1, spectra.shape[1]))
 
         meta = self.data.metadata  # needs to be copied to the new GeoArray
 
@@ -382,7 +382,7 @@ def _get_spectra_by_label_imname(self, cluster_label, image_basename, n_spectra2
 
     def plot_sample_spectra(self, image_basename, cluster_label='all', include_mean_spectrum=True,
                             include_median_spectrum=True, ncols=5, **kw_fig):
-        # type: (Union[str, int, List], str, bool, bool, int, dict) -> plt.figure
+        # type: (Union[str, int, List], str, bool, bool, int, dict) -> 'plt.figure'
         from matplotlib import pyplot as plt
 
         if isinstance(cluster_label, int):

tests/gitlab_CI_docker/context/environment_spechomo.yml

@@ -35,6 +35,7 @@ dependencies:
     - pydocstyle
     - rednose
     - sphinx-argparse
+    - urlchecker
 
     # setup requirements
     - setuptools-git

tests/test_classifier.py

@@ -35,6 +35,7 @@
 import os
 import json
 from unittest import TestCase
+from tempfile import TemporaryDirectory
 
 from spechomo.classifier import Cluster_Learner
 from spechomo import __path__
@@ -71,9 +72,9 @@ def test_to_jsonable_dict(self):
         outstr = json.dumps(jsonable_dict, sort_keys=True, indent=4)
         self.assertIsInstance(outstr, str)
 
-    # def test_save_to_json(self):
-    #     with TemporaryDirectory() as tmpDir:
-    #         self.clf.save_to_json(os.path.join(tmpDir, 'clf.json'))
+    def test_save_to_json(self):
+        with TemporaryDirectory() as tmpDir:
+            self.clf.save_to_json(os.path.join(tmpDir, 'clf.json'))
 
 
 # class Test_ClassifierCollection(TestCase):

tests/test_classifier_creation.py

@@ -93,7 +93,8 @@ def test_resample_spectra(self):
         src_im = GeoArray(self.SHC.ims_ref[0])
         unif_random_spectra = self.SHC.cluster_image_and_get_uniform_spectra(src_im)
 
-        tgt_rsr = RSR(satellite='Sentinel-2A', sensor='MSI')
+        # tgt_rsr = RSR(satellite='Sentinel-2A', sensor='MSI')
+        tgt_rsr = RSR(satellite='Terra', sensor='MODIS', sort_by_cwl=True)
         unif_random_spectra_rsp = \
             self.SHC.resample_spectra(unif_random_spectra,
                                       src_cwl=np.array(src_im.meta.band_meta['wavelength'], dtype=np.float).flatten(),