waveform indexer: skips files with unchanged timestamp even if option "--force-reindex" is requested

[megies]

Currently the option --force-reindex is not behaving as expected. Intuitively I would assume that any file encountered by the crawler is reindexed, but currently the file is not reindexed, if the file's timestamp is unchanged from when it was last reindexed (because option --force-reindex is not passed on from src/jane/waveforms/management/commands/index_waveforms.py to src/jane/waveforms/process_waveforms.py).

This patch solves the issue:

diff --git a/src/jane/waveforms/management/commands/index_waveforms.py b/src/jane/waveforms/management/commands/index_waveforms.py
index 98b2126..29c33e0 100644
--- a/src/jane/waveforms/management/commands/index_waveforms.py
+++ b/src/jane/waveforms/management/commands/index_waveforms.py
@@ -282,7 +282,7 @@ class WaveformFileCrawler(object):
         self.input_queue[filepath] = (path, file)
 
 
-def worker(_i, input_queue, work_queue, log_queue):
+def worker(_i, input_queue, work_queue, log_queue, force_reindex=False):
     try:
         # loop through input queue
         while True:
@@ -298,7 +298,7 @@ def worker(_i, input_queue, work_queue, log_queue):
 
             # process the file.
             try:
-                process_waveforms.process_file(filepath)
+                process_waveforms.process_file(filepath, force_reindex=force_reindex)
             except Exception as e:
                 log_queue.append("Error indexing '%s': '%s' - %s" % (
                     filepath, str(type(e)), str(e)))
@@ -388,7 +388,8 @@ def _run_indexer(options):
         # spawn processes
         connection.close()
         for i in range(options["number_of_cpus"]):
-            args = (i, in_queue, work_queue, log_queue)
+            args = (i, in_queue, work_queue, log_queue,
+                    options["force_reindex"])
             p = multiprocessing.Process(target=worker, args=args)
             p.daemon = True
             p.start()
diff --git a/src/jane/waveforms/process_waveforms.py b/src/jane/waveforms/process_waveforms.py
index 0a4beea..ef00c7c 100644
--- a/src/jane/waveforms/process_waveforms.py
+++ b/src/jane/waveforms/process_waveforms.py
@@ -13,7 +13,7 @@ from . import models
 from .utils import to_datetime
 
 
-def process_file(filename):
+def process_file(filename, force_reindex=False):
     """
     Process a single waveform file.
 
@@ -38,8 +38,9 @@ def process_file(filename):
         ctime = to_datetime(stats.st_ctime)
         size = int(stats.st_size)
 
-        # Nothing to do if nothing changed.
-        if file.size == size and file.mtime == mtime and file.ctime == ctime:
+        # Nothing to do if nothing changed.. unless force reindex is requested
+        if file.size == size and file.mtime == mtime and file.ctime == ctime \
+                and not force_reindex:
             return
 
     # If it does not exist, create it in the next step.

But actually the current behavior is also a valid use case, so it might even be better to introduce two levels of "forcing", one that still skips on unchanged timestamp (like it is now) and one that does a force reindex even on unchanged timestamps..