Open PR for Streaming Partitioning

.github/PULL_REQUEST_TEMPLATE.md

@@ -4,13 +4,13 @@
   `scripts/` is it tested?
 - [ ] `make format diff_pylint_report cppcheck doc pydocstyle` Is it well
   formatted?
-- [ ] Did it change the command-line interface? Only additions are allowed
-  without a major version increment. Changing file formats also requires a
-  major version number increment.
-- [ ] Is it documented in the `ChangeLog`?
-  http://en.wikipedia.org/wiki/Changelog#Format
+- [ ] Did it change the command-line interface? Only backwards-compatible
+  additions are allowed without a major version increment. Changing file
+  formats also requires a major version number increment.
+- [ ] For substantial changes or changes to the command-line interface, is it
+  documented in `CHANGELOG.md`? See [keepachangelog](http://keepachangelog.com/)
+  for more details.
 - [ ] Was a spellchecker run on the source code and documentation after
   changes were made?
 - [ ] Do the changes respect streaming IO? (Are they
   tested for streaming IO?)
-- [ ] Is the Copyright year up to date?

.gitignore

@@ -52,3 +52,4 @@ pylint_report.txt
 pep8_report.txt
 pep257_report.txt
 .cache/
+khmer/_oxli/*.cpp

.travis.yml

@@ -7,6 +7,13 @@ dist: trusty
 notifications:
   email: false
 
+# with building pushes enabled on travis we would get
+# two builds for every new commit pushed to a branch
+# for a PR. So we disable build-on-push for all branches
+# except the ones whitelisted here.
+branches:
+  only:
+    - master
 
 matrix:
   fast_finish: true
@@ -62,6 +69,7 @@ script:
 
 # generate all the diagnostic reports
 after_success:
+  - make clean
   - PYTEST_ADDOPTS=-qqq make coverage-gcovr.xml coverage.xml
   # Fix suggested by http://diff-cover.readthedocs.io/en/latest/#troubleshooting
   - git fetch origin master:refs/remotes/origin/master

CHANGELOG.md

@@ -0,0 +1,65 @@
+# Change Log
+All notable changes to the khmer project will be documented in this file.
+See [keepachangelog](http://keepachangelog.com/) for more info.
+
+The khmer project's command line scripts adhere to
+[Semantic Versioning](http://semver.org/). The Python and C++ APIs are not yet
+under semantic versioning, but will be in future versions of khmer.
+
+## [Unreleased]
+### Added
+- New `--no-reformat` option for `interleave-reads.py` script disables default
+  read name correction behavior.
+- New `HashSet` data structure for managing collections of k-mer hashes and
+  tags.
+- khmer package version now included in `.info` files.
+- New `-o|--outfile` option for `filter-abund-single.py` script.
+- New sandbox script `extract-compact-dbg.py` for computing a compact de Bruijn
+  graph from sequence data.
+- New `--quiet` flag to several scripts, silencing diagnostic messages in
+  terminal output.
+- Support for human-friendly memory requests (2G instead of 2000000000 or 2e9).
+- Support for variable-coverage trimming in the `filter-abund-single.py` script.
+- A simple example of the C++ API in `examples/c++-api`.
+- New `assemble_linear_path` function for baiting unambiguous contigs with a
+  seed k-mer from a hashtable.
+- Support for assembling directly from k-mer graphs, and a new
+  JunctionCountAssembler class.
+- Add --info flag for obtaining citation information.
+- Added Counttable and Nodetable to support non-reversible hashing
+  functionality and k > 32.
+- Add a new storage class using half a byte per entry. Exposed as
+  SmallCounttable and SmallCountgraph.
+- Added `cleaned_seq` attribute to `khmer.Read` class which provides a cleaned
+  version of the sequence of each read.
+
+### Changed
+- Switch from nose to py.test as the testing framework.
+- Switch from internally managed Jenkins setup to Travis CI for continuous
+  integration testing.
+- Renamed core data structures: CountingHash --> Countgraph,
+  Hashbits --> Nodegraph.
+- Replaced the IParser and FastxParser classes with a single ReadParser class.
+  Different input formats are supported by templating ReadParser with a reader
+  class.
+- Renamed `consume_fasta` and related functions to `consume_seqfile`, with
+  support for reading sequences from additional formats pending.
+
+### Fixed
+- Bug in compressed(gzip) streaming output from scripts
+- The hashbits `update_from` function to correctly track occupied bins for
+  calculating FPR.
+- Bug in the `filter-abund.py` script when `--gzip` and `-o` flags are used
+  simultaneously.
+- Bug in the hashtable `get_kmers` function based on incorrect usage of the
+  `substr` function.
+- Bug in `broken_paired_reader` related to dropping short reads when
+  `require_paired` is set.
+- Bug related to handling lowercase [acgtn] characters in input data.
+- Bug in `load-graph.py` that calculated required graph space incorrectly.
+- Fix loading of empty partion map files
+
+## [2.0] - 2015-10-08
+
+Previous to the khmer 2.1 release, all changes were documented in a file named
+`ChangeLog`. This file is now at `legacy/ChangeLog` for posterity.

Doxyfile.in

@@ -753,8 +753,10 @@ WARN_LOGFILE           =
 # spaces.
 # Note: If this tag is empty the current directory is searched.
 
-INPUT                  = lib \
-                         khmer \
+INPUT                  = include/oxli \
+                         include/khmer \
+                         src/oxli \
+                         src/khmer \
                          scripts
 
 # This tag can be used to specify the character encoding of the source files

MANIFEST.in

@@ -1,7 +1,8 @@
 include ChangeLog ez_setup.py IDEAS Makefile README.rst setup.cfg
 include versioneer.py MANIFEST.in CITATION CONTRIBUTING.md Doxyfile.in
 include LICENSE TODO .ycm_extra_conf.py
-recursive-include lib *.hh *.cc [Mm]akefile* get_version.py
+recursive-include include *.hh [Mm]akefile* get_version.py
+recursive-include khmer/_oxli *.pxd *.cpp
 recursive-include third-party *.cc *.1 *.xsl README* sample* words* *.sh *.c
 recursive-include third-party manual* [Mm]akefile* *.pl *.dsp CHANGES *.txt *.h
 recursive-include third-party ChangeLog FAQ INDEX configure *.xsl

Makefile

@@ -37,13 +37,16 @@
 
 # `SHELL=bash` Will break Titus's laptop, so don't use BASH-isms like
 # `[[` conditional expressions.
-CPPSOURCES=$(wildcard lib/*.cc lib/*.hh khmer/_khmer.cc) setup.py
+#
+PREFIX=/usr/local
+CPPSOURCES=$(wildcard src/oxli/*.cc include/oxli/*.hh src/khmer/_cpy_*.cc include/khmer/_cpy_*.hh) setup.py
+CYSOURCES=$(wildcard khmer/_oxli/*.pxd khmer/_oxli/*.pyx)
 PYSOURCES=$(filter-out khmer/_version.py, \
 	  $(wildcard khmer/*.py scripts/*.py oxli/*.py) )
-SOURCES=$(PYSOURCES) $(CPPSOURCES) setup.py
+SOURCES=$(PYSOURCES) $(CPPSOURCES) $(CYSOURCES) setup.py
 
 DEVPKGS=pep8==1.6.2 diff_cover autopep8 pylint coverage gcovr pytest \
-	pydocstyle screed pyenchant
+	    pydocstyle screed pyenchant Cython==0.25.2
 GCOVRURL=git+https://github.com/nschum/gcovr.git@never-executed-branches
 
 VERSION=$(shell ./setup.py version | grep Version | awk '{print $$4}' \
@@ -65,11 +68,11 @@ INCLUDEOPTS=$(shell gcc -E -x c++ - -v < /dev/null 2>&1 >/dev/null \
 PYINCLUDE=$(shell python -c \
 	  "import sysconfig;print(sysconfig.get_path('include'))")
 
-CPPCHECK=ls lib/*.cc khmer/_khmer.cc | grep -v test | cppcheck -DNDEBUG \
+CPPCHECK=ls src/oxli/*.cc src/khmer/*.cc | grep -v test | cppcheck -DNDEBUG \
 	 -DVERSION=0.0.cppcheck -DSEQAN_HAS_BZIP2=1 -DSEQAN_HAS_ZLIB=1 \
 	 -UNO_UNIQUE_RC --enable=all --suppress='*:/usr/*' \
 	 --suppress='*:$(PYINCLUDE)/*' --file-list=- --platform=unix64 \
-	 --std=c++11 --inline-suppr --quiet -Ilib -Ithird-party/bzip2 \
+	 --std=c++11 --inline-suppr --quiet --verbose -Iinclude -Ithird-party/bzip2 \
 	 -Ithird-party/zlib -Ithird-party/smhasher -I$(PYINCLUDE) \
 	 $(DEFINES) $(INCLUDEOPTS)
 
@@ -83,6 +86,7 @@ endif
 MODEXT=$(shell python -c \
        "import sysconfig;print(sysconfig.get_config_var('SO'))")
 EXTENSION_MODULE = khmer/_khmer$(MODEXT)
+CY_MODULES = $($(wildcard khmer/_oxli/*.pyx): .pyx=.$(MODEXT))
 
 PYLINT_TEMPLATE="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}"
 
@@ -102,9 +106,12 @@ install-dependencies:
 	pip install --upgrade --requirement doc/requirements.txt
 
 ## sharedobj   : build khmer shared object file
-sharedobj: $(EXTENSION_MODULE)
+sharedobj: $(EXTENSION_MODULE) $(CYTHON_MODULE)
+
+$(EXTENSION_MODULE): $(CPPSOURCES) $(CYSOURCES)
+	./setup.py build_ext --inplace
 
-$(EXTENSION_MODULE): $(CPPSOURCES)
+$(CYTHON_MODULE): $(CPPSOURCES) $(CYSOURCES)
 	./setup.py build_ext --inplace
 
 coverage-debug: $(CPPSOURCES)
@@ -124,21 +131,28 @@ dist/khmer-$(VERSION).tar.gz: $(SOURCES)
 
 ## clean       : clean up all temporary / machine-generated files
 clean: FORCE
-	cd lib && $(MAKE) clean || true
+	cd src/oxli && $(MAKE) clean || true
 	cd tests && rm -rf khmertest_* || true
+	rm -f pytest_runner-*.egg pytests.xml
 	rm -f $(EXTENSION_MODULE)
-	rm -f khmer/*.pyc lib/*.pyc scripts/*.pyc tests/*.pyc oxli/*.pyc \
-		sandbox/*.pyc khmer/__pycache__/* sandbox/__pycache__/*
+	rm -f khmer/*.pyc scripts/*.pyc tests/*.pyc oxli/*.pyc \
+		sandbox/*.pyc khmer/__pycache__/* sandbox/__pycache__/* \
+		khmer/_oxli/*.cpp
 	./setup.py clean --all || true
 	rm -f coverage-debug
-	rm -Rf .coverage
+	rm -Rf .coverage coverage-gcovr.xml coverage.xml
 	rm -f diff-cover.html
 	rm -Rf build dist
+	rm -rf __pycache__/ .eggs/ khmer.egg-info/
+	@find ./ -type d -name __pycache__ -exec rm -rf {} +
+	@find ./khmer/ -type f -name *$(MODEXT) -exec rm -f {} +
+	@find ./khmer/_oxli/ -type f -name *.so -exec rm -f {} +
+	-rm -f *.gcov
 
 debug: FORCE
 	export CFLAGS="-pg -fprofile-arcs -D_GLIBCXX_DEBUG_PEDANTIC \
 		-D_GLIBCXX_DEBUG -DDEBUG_ASSEMBLY=1 -DDEBUG_FILTERS=1"; python setup.py build_ext --debug \
-		--inplace 
+		--inplace
 
 ## doc         : render documentation in HTML
 doc: build/sphinx/html/index.html
@@ -262,32 +276,37 @@ doxygen: doc/doxygen/html/index.html
 
 doc/doxygen/html/index.html: $(CPPSOURCES) $(PYSOURCES)
 	mkdir -p doc/doxygen
-	sed "s=\$${VERSION}=$$(python ./lib/get_version.py)=" Doxyfile.in | \
+	sed "s=\$${VERSION}=$(VERSION)=" Doxyfile.in | \
 		sed "s=\$${INCLUDES}=$(INCLUDESTRING)=" > Doxyfile
 	doxygen
 
-lib: FORCE
-	cd lib && \
+liboxli: FORCE
+	cd src/oxli && \
 	$(MAKE)
 
-# Runs a test of ./lib
+install-liboxli: liboxli
+	cd src/oxli && $(MAKE) install PREFIX=$(PREFIX)
+	mkdir -p $(PREFIX)/include/khmer
+	cp -r include/khmer/_cpy_*.hh $(PREFIX)/include/khmer/
+
+# Runs a test of liboxli
 libtest: FORCE
 	rm -rf install_target
 	mkdir -p install_target
-	cd lib && \
+	cd src/oxli && \
 	 $(MAKE) clean && \
 	 $(MAKE) all && \
 	 $(MAKE) install PREFIX=../install_target
 	test -d install_target/include
-	test -f install_target/include/oxli/khmer.hh
+	test -f install_target/include/oxli/oxli.hh
 	test -d install_target/lib
 	test -f install_target/lib/liboxli.a
 	$(CXX) -std=c++11 -o install_target/test-prog-static \
-		-I install_target/include lib/test-compile.cc \
+		-I install_target/include src/oxli/test-compile.cc \
 		install_target/lib/liboxli.a
 	$(CXX) -std=c++11 -o install_target/test-prog-dynamic \
 		-I install_target/include -L install_target/lib \
-		lib/test-compile.cc -loxli
+		src/oxli/test-compile.cc -loxli
 	rm -rf install_target
 
 ## test        : run the khmer test suite
@@ -296,12 +315,12 @@ test: FORCE
 	py.test -m ${TESTATTR}
 
 sloccount.sc: $(CPPSOURCES) $(PYSOURCES) $(wildcard tests/*.py) Makefile
-	sloccount --duplicates --wide --details lib khmer scripts tests \
+	sloccount --duplicates --wide --details include src khmer scripts tests \
 		setup.py Makefile > sloccount.sc
 
 ## sloccount   : count lines of code
 sloccount:
-	sloccount lib khmer scripts tests setup.py Makefile
+	sloccount src include khmer scripts tests setup.py Makefile
 
 coverity-build:
 	if [ -x "${cov_analysis_dir}/bin/cov-build" ]; \

README.rst

@@ -62,38 +62,47 @@ them, please visit the following URLs:
 
 -  Announcements: http://lists.idyll.org/listinfo/khmer-announce
 
-We chat at https://gitter.im/dib-lab/khmer and the maintainers can be
-contacted at [email protected].
+The maintainers can be contacted at [email protected].
 
 For getting help please see this guide: http://khmer.readthedocs.io/en/stable/user/getting-help.html
+Your first port of call when looking for help should be: https://github.com/dib-lab/khmer/issues
 
-IMPORTANT NOTE: CITE US!
+
+Important note: cite us!
 ------------------------
 
 khmer is *research software*, so you should cite us when you use it
 in scientific publications!  Please see the `CITATION
 <http://khmer.readthedocs.io/en/stable/citations.html>`__ file for
 citation information.
 
-INSTALL INSTRUCTIONS:
----------------------
 
-khmer requires a 64-bit operating system and Python version 2.7.x, 3.3.x, or
-3.4.x. Linux users will need the Python development libraries and gcc. OS X
-users may need XCode installed to build from source.
+Install
+-------
+
+khmer works with Python version 2.7 or 3.5. Create a
+`virtual environemnt <http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_
+and install khmer with:
+
+```
+pip install khmer
+```
 
-In short:
+You are ready to go. For more details see `doc/install.rst
+<https://khmer.readthedocs.io/en/stable/user/install.html>`_
 
-``pip install khmer`` to download, build, and install the latest stable
-version.
 
-For more details see `doc/install.rst <https://khmer.readthedocs.io/en/stable/user/install.html>`_
+Contributing
+------------
 
-The use of a virtualenv is recommended, see
-https://virtualenv.readthedocs.io/en/latest/installation.html
+If you want to modify khmer or contribute to its development read the
+`developer instructions
+<https://khmer.readthedocs.io/en/stable/dev/getting-started.html>`_
 
-khmer is under the BSD license; see doc/LICENSE.txt. Distribution,
+
+License
+-------
+
+khmer is distributed under the BSD license; see doc/LICENSE.txt. Distribution,
 modification and redistribution, incorporation into other software, and
 pretty much everything else is allowed.
-
-MRC 2015-09-07