Streamlining some scripts after changing imports (issue #79).

docs/notes/python3.md

@@ -200,21 +200,11 @@ This made many changes, importing from future and changing other imports. One we
 
 There are no occurrences of "import *" in functions and no import cycles that I am aware of.
 
+See commit [038ec101](https://github.com/tarsqi/ttk/commit/038ec101a197840f92ba492a064a7d452b0ed501).
 
+After this I made some changes to streamline some of the utilities and testing scripts, including removing the annoying *path* module and updating the documentation on running the scripts. Previsously, scripts like `testing/run_tests.py` ran from the directory they are in and they relied on separate scrappy module to manipulate the path. That started to fail after the import updates so I removed the path cludge and changed the documentation to reflect the new way of calling the scripts from the parent directory: 
 
 ```
-$ python regression.py --evita
-Traceback (most recent call last):
-  File "regression.py", line 46, in <module>
-    from . import path
-ValueError: Attempted relative import in non-package
-```
-
-No idea what is going on here because there is an init file and the directory should be considered a package. Okay, got it, see [stackoverflow.com/questions/11536764](https://stackoverflow.com/questions/11536764/how-to-fix-attempted-relative-import-in-non-package-even-with-init-py) and [PEP 338](https://www.python.org/dev/peps/pep-0338/). When you run the regressions script it is the main script and then `__name__` is set to `__main__` and all the imports go awry.
-
-Could not figure out exactly how to use `__package__` to deal with this, but you can do the following from the parent directory:
-
-```
-$ python -m testing.regression --evita
+$ python -m testing.run_tests
 ```
 

testing/regression.py

@@ -3,24 +3,25 @@
 Regression test for the Tarsqi components. Currently only for Evita and within
 evita only loads some verbal event cases.
 
-Usage:
+USAGE, this needs to be run from the parent directory with the -m option:
 
-$ python regression.py --evita
+$ python -m testing.regression --evita
 
    Runs all available Evita tests and stores the results in directories
    results/evita-XX where the XX denotes a particular Evita test. Files in the
    results directories are timestamped.
 
-$ python regression.py --report
+$ python -m testing.regression --report
 
-   Generate HTML reports of all available tests.
+   Generate HTML reports of all available tests. The reports are written to
+   results/html.
 
-$ python regression.py --purge TEST_CASE TIMESTAMP
+$ python -m testing.regression --purge TEST_CASE TIMESTAMP
 
    Purge the resuts data for a particular test case at a particular timestamp
    and update the report. An example test case would be evita-vg.
 
-$ python regression.py --create-event-vg-cases
+$ python -m testing.regression --create-event-vg-cases
 
    Creates test cases from cases/input/timebank-events-vg.txt and puts them in
    cases/cases-evita-vg.tab. This needs to be run only once.
@@ -41,11 +42,12 @@
 
 
 from __future__ import absolute_import
+
 import os, sys, getopt, time, glob
 
-from . import path
 import tarsqi
 
+
 def load_cases(fname):
     """return a list of all test cases in fname."""
     cases = []

testing/run_tests.py

@@ -49,10 +49,11 @@
 
 
 from __future__ import absolute_import
+
 import sys, getopt, traceback, types
 
-from . import path
 import tarsqi
+
 from components.merging.sputlink.main import ConstraintPropagator
 
 from .cases.test_cases_timex import TIMEX_TESTS

utilities/convert.py

@@ -1,10 +1,11 @@
 """convert.py
 
-Some format conversion utilities.
+Some format conversion utilities. Run all commands below from the parent
+directory using the -m option.
 
 1. Convert LDC TimeBank into a modern TimeBank in the TTK format.
 
-   $ python convert.py --timebank2ttk TIMEBANK_DIR TTK_DIR
+   $ python -m utilities.convert --timebank2ttk TIMEBANK_DIR TTK_DIR
 
    Converts TimeBank 1.2 as released by LDC into a version without makeinstance
    tags using the TTK format. This should be run on the data/extra files in the
@@ -13,7 +14,7 @@
 
 2. Convert Thyme format into TTK.
 
-   $ python convert.py --thyme2ttk THYME_TEXT_DIR THYME_ANNO_DIR TTK_DIR
+   $ python -m utilities.convert --thyme2ttk THYME_TEXT_DIR THYME_ANNO_DIR TTK_DIR
 
    Note that in the Thyme corpus we have annotation directories like
    AnnotationData/coloncancer/Dev, whereas in the text directories we find
@@ -22,16 +23,16 @@
 
 3. Convert the TTK format into HTML.
 
-   $ python convert.py --ttk2html TTK_DIR HTML_DIR
-   $ python convert.py --ttk2html --show-links TTK_DIR HTML_DIR
+   $ python -m utilities.convert --ttk2html TTK_DIR HTML_DIR
+   $ python -m utilities.convert --ttk2html --show-links TTK_DIR HTML_DIR
 
    Converts TTK files in TTK_DIR into HTML files in HTML_DIR, if --show-links is
    used links are shown in addition to the timexes and events.
 
 4. Convert Knowtator format into TTK.
 
-   $ python convert.py --knowtator2ttk KNOWTATOR_DIR TTK_DIR
-   $ python convert.py --knowtator2ttk --tarsqi KNOWTATOR_DIR TTK_DIR
+   $ python -m utilities.convert --knowtator2ttk KNOWTATOR_DIR TTK_DIR
+   $ python -m utilities.convert --knowtator2ttk --tarsqi KNOWTATOR_DIR TTK_DIR
 
    This is not a general conversion from any Knowtator output, it assumes that
    the input annotations are all events, timexes and tlinks. If the --tarsqi
@@ -40,20 +41,20 @@
    considered Tarsqi results), using the --tarsqi option can be useful for
    evaluation.
 
-   $ python convert.py --knowtator2ttk --tarsqi TEXT_FILE TTK_FILE
+   $ python -m utilities.convert --knowtator2ttk --tarsqi TEXT_FILE TTK_FILE
 
    Version for processing a single file. You only supply the text file, the code
    assumes that there is a file TEXT_FILE.knowtator.xml with the annotations.
 
 5. Convert TTK into Knowtator format.
 
-   $ python convert.py --ttk2knowtator TTK_FILE TEXT_FILE ANNO_FILE
+   $ python -m utilities.convert --ttk2knowtator TTK_FILE TEXT_FILE ANNO_FILE
 
    IN PROGRESS.
 
 6. Convert from ECB into TTK
 
-   $ python convert.py --ecb2ttk ECB_DIR OUT_DIR
+   $ python -m utilities.convert --ecb2ttk ECB_DIR OUT_DIR
 
    THE ECB_DIR directory should be the top-level directory of the ECB
    distribution (which has a README file and a data directory which includes
@@ -74,7 +75,6 @@
 import os, sys, getopt, codecs, time, glob
 from xml.dom import minidom, Node
 
-import path
 import tarsqi
 from docmodel.main import create_source_parser
 from docmodel.main import create_metadata_parser

utilities/make_documentation.py

@@ -9,6 +9,13 @@
 notably those that import the treetagger.
 
 
+USAGE:
+
+Run this from the parent directory with the -m option
+
+$ python -m utilities.make_documentation
+
+
 TODO:
 
 - the code does not deal with class methods, see for example TagId in
@@ -19,6 +26,8 @@
   functions that are not in the code, but are imported functions. This has
   seized to be the case for that module, but it is not clear why.
 
+- Check whether pydoc still chokes in tarsqi code
+
 """
 
 
@@ -40,7 +49,7 @@
 PRINT_PRIVATE_FUNCTIONS = True
 
 # directory where the documentation is written
-DOCUMENTATION_DIR = os.path.join('..', 'docs', 'code')
+DOCUMENTATION_DIR = os.path.join('docs', 'code')
 
 
 javascript_code = """<script language="JavaScript" type="text/JavaScript">

utilities/modules.py

@@ -8,7 +8,7 @@
 """
 
 from __future__ import absolute_import
-from . import path
+
 import tarsqi
 
 import components.blinker.compare