Major literal optimization refactoring.

The principle change in this commit is a complete rewrite of how
literals are detected from a regular expression. In particular, we now
traverse the abstract syntax to discover literals instead of the
compiled byte code. This permits more tuneable control over which and
how many literals are extracted, and is now exposed in the
`regex-syntax` crate so that others can benefit from it.

Other changes in this commit:

* The Boyer-Moore algorithm was rewritten to use my own concoction based
  on frequency analysis. We end up regressing on a couple benchmarks
  slightly because of this, but gain in some others and in general should
  be faster in a broader number of cases. (Principally because we try to
  run `memchr` on the rarest byte in a literal.) This should also greatly
  improve handling of non-Western text.
* A "reverse suffix" literal optimization was added. That is, if suffix
  literals exist but no prefix literals exist, then we can quickly scan
  for suffix matches and then run the DFA in reverse to find matches.
  (I'm not aware of any other regex engine that does this.)
* The mutex-based pool has been replaced with a spinlock-based pool
  (from the new `mempool` crate). This reduces some amount of constant
  overhead and improves several benchmarks that either search short
  haystacks or find many matches in long haystacks.
* Search parameters have been refactored.
* RegexSet can now contain 0 or more regular expressions (previously, it
  could only contain 2 or more). The InvalidSet error variant is now
  deprecated.
* A bug in computing start states was fixed. Namely, the DFA assumed the
  start states was always the first instruction, which is trivially
  wrong for an expression like `^☃$`. This bug persisted because it
  typically occurred when a literal optimization would otherwise run.
* A new CLI tool, regex-debug, has been added as a non-published
  sub-crate. The CLI tool can answer various facts about regular
  expressions, such as printing its AST, its compiled byte code or its
  detected literals.

Closes #96, #188, #189

Author: BurntSushi

Cargo.toml

@@ -17,6 +17,8 @@ finite automata and guarantees linear time matching on all inputs.
 aho-corasick = "0.5"
 # For skipping along search text quickly when a leading byte is known.
 memchr = "0.1"
+# For managing regex caches quickly across multiple threads.
+mempool = "0.2"
 # For parsing regular expressions.
 regex-syntax = { path = "regex-syntax", version = "0.3.0" }
 # For compiling UTF-8 decoding into automata.

HACKING.md

@@ -32,7 +32,8 @@ The code to find prefixes and search for prefixes is in src/literals.rs. When
 more than one literal prefix is found, we fall back to an Aho-Corasick DFA
 using the aho-corasick crate. For one literal, we use a variant of the
 Boyer-Moore algorithm. Both Aho-Corasick and Boyer-Moore use `memchr` when
-appropriate.
+appropriate. The Boyer-Moore variant in this library also uses elementary
+frequency analysis to choose the write byte to run `memchr` with.
 
 Of course, detecting prefix literals can only take us so far. Not all regular
 expressions have literal prefixes. To remedy this, we try another approach to
@@ -53,10 +54,12 @@ text results in at most one new DFA state. It is made fast by caching states.
 DFAs are susceptible to exponential state blow up (where the worst case is
 computing a new state for every input byte, regardless of what's in the state
 cache). To avoid using a lot of memory, the lazy DFA uses a bounded cache. Once
-the cache is full, it is wiped and state computation starts over again.
+the cache is full, it is wiped and state computation starts over again. If the
+cache is wiped too frequently, then the DFA gives up and searching falls back
+to one of the aforementioned algorithms.
 
-All of the above matching engines expose precisely the matching semantics. This
-is indeed tested. (See the section below about testing.)
+All of the above matching engines expose precisely the same matching semantics.
+This is indeed tested. (See the section below about testing.)
 
 The following sub-sections describe the rest of the library and how each of the
 matching engines are actually used.
@@ -70,6 +73,9 @@ encountered. Parsing is done in a separate crate so that others may benefit
 from its existence, and because it is relatively divorced from the rest of the
 regex library.
 
+The regex-syntax crate also provides sophisticated support for extracting
+prefix and suffix literals from regular expressions.
+
 ### Compilation
 
 The compiler is in src/compile.rs. The input to the compiler is some abstract
@@ -162,7 +168,7 @@ knows what the caller wants. Using this information, we can determine which
 engine (or engines) to use.
 
 The logic for choosing which engine to execute is in src/exec.rs and is
-documented on the Exec type. Exec values collection regular expression
+documented on the Exec type. Exec values contain regular expression
 Programs (defined in src/prog.rs), which contain all the necessary tidbits
 for actually executing a regular expression on search text.
 
@@ -172,6 +178,14 @@ of src/exec.rs by far is the execution of the lazy DFA, since it requires a
 forwards and backwards search, and then falls back to either the NFA algorithm
 or backtracking if the caller requested capture locations.
 
+The parameterization of every search is defined in src/params.rs. Among other
+things, search parameters provide storage for recording capture locations and
+matches (for regex sets). The existence and nature of storage is itself a
+configuration for how each matching engine behaves. For example, if no storage
+for capture locations is provided, then the matching engines can give up as
+soon as a match is witnessed (which may occur well before the leftmost-first
+match).
+
 ### Programs
 
 A regular expression program is essentially a sequence of opcodes produced by
@@ -268,48 +282,46 @@ N.B. To run tests for the `regex!` macro, use:
 
 The benchmarking in this crate is made up of many micro-benchmarks. Currently,
 there are two primary sets of benchmarks: the benchmarks that were adopted at
-this library's inception (in `benches/bench.rs`) and a newer set of benchmarks
+this library's inception (in `benches/src/misc.rs`) and a newer set of benchmarks
 meant to test various optimizations. Specifically, the latter set contain some
-analysis and are in `benches/bench_sherlock.rs`. Also, the latter set are all
+analysis and are in `benches/src/sherlock.rs`. Also, the latter set are all
 executed on the same lengthy input whereas the former benchmarks are executed
 on strings of varying length.
 
 There is also a smattering of benchmarks for parsing and compilation.
 
+Benchmarks are in a separate crate so that its dependencies can be managed
+separately from the main regex crate.
+
 Benchmarking follows a similarly wonky setup as tests. There are multiple
 entry points:
 
-* `bench_native.rs` - benchmarks the `regex!` macro
-* `bench_dynamic.rs` - benchmarks `Regex::new`
-* `bench_dynamic_nfa.rs` benchmarks `Regex::new`, forced to use the NFA
-  algorithm on every regex. (N.B. This can take a few minutes to run.)
+* `bench_rust_plugin.rs` - benchmarks the `regex!` macro
+* `bench_rust.rs` - benchmarks `Regex::new`
+* `bench_rust_bytes.rs` benchmarks `bytes::Regex::new`
 * `bench_pcre.rs` - benchmarks PCRE
+* `bench_onig.rs` - benchmarks Oniguruma
 
-The PCRE benchmarks exist as a comparison point to a mature regular expression
-library. In general, this regex library compares favorably (there are even a
-few benchmarks that PCRE simply runs too slowly on or outright can't execute at
-all). I would love to add other regular expression library benchmarks
-(especially RE2), but PCRE is the only one with reasonable bindings.
+The PCRE and Oniguruma benchmarks exist as a comparison point to a mature
+regular expression library. In general, this regex library compares favorably
+(there are even a few benchmarks that PCRE simply runs too slowly on or
+outright can't execute at all). I would love to add other regular expression
+library benchmarks (especially RE2).
 
 If you're hacking on one of the matching engines and just want to see
 benchmarks, then all you need to run is:
 
-    $ cargo bench --bench dynamic
+    $ ./run-bench rust
 
 If you want to compare your results with older benchmarks, then try:
 
-    $ cargo bench --bench dynamic | tee old
+    $ ./run-bench rust | tee old
     $ ... make it faster
-    $ cargo bench --bench dynamic | tee new
+    $ ./run-bench rust | tee new
     $ cargo-benchcmp old new --improvements
 
 The `cargo-benchcmp` utility is available here:
 https://github.com/BurntSushi/cargo-benchcmp
 
-To run the same benchmarks on PCRE, you'll need to use the sub-crate in
-`regex-pcre-benchmark` like so:
-
-    $ cargo bench --manifest-path regex-pcre-benchmark/Cargo.toml
-
-The PCRE benchmarks are separated from the main regex crate so that its
-dependency doesn't break builds in environments without PCRE.
+The `run-bench` utility can run benchmarks for PCRE and Oniguruma too. See
+`./run-bench --help`.

benches/Cargo.toml

@@ -14,7 +14,6 @@ enum-set = "0.0.6"
 lazy_static = "0.1"
 onig = { version = "0.4", optional = true }
 pcre = { version = "0.2", optional = true }
-rand = "0.3"
 regex = { version = "0.1", path = ".." }
 regex_macros = { version = "0.1", path = "../regex_macros", optional = true }
 regex-syntax = { version = "0.3", path = "../regex-syntax" }

benches/src/bench_onig.rs

@@ -12,7 +12,6 @@
 
 #[macro_use] extern crate lazy_static;
 extern crate onig;
-extern crate rand;
 extern crate test;
 
 use std::ops::Deref;

benches/src/bench_pcre.rs

@@ -24,7 +24,6 @@
 extern crate enum_set;
 #[macro_use] extern crate lazy_static;
 extern crate pcre;
-extern crate rand;
 extern crate test;
 
 /// A nominal wrapper around pcre::Pcre to expose an interface similar to

benches/src/bench_rust.rs

@@ -11,7 +11,6 @@
 #![feature(test)]
 
 #[macro_use] extern crate lazy_static;
-extern crate rand;
 extern crate regex;
 extern crate regex_syntax;
 extern crate test;

benches/src/bench_rust_bytes.rs

@@ -11,7 +11,6 @@
 #![feature(test)]
 
 #[macro_use] extern crate lazy_static;
-extern crate rand;
 extern crate regex;
 extern crate regex_syntax;
 extern crate test;

benches/src/bench_rust_plugin.rs

@@ -12,7 +12,6 @@
 #![plugin(regex_macros)]
 
 #[macro_use] extern crate lazy_static;
-extern crate rand;
 extern crate regex;
 extern crate regex_syntax;
 extern crate test;

benches/src/misc.rs

@@ -130,6 +130,14 @@ bench_match!(one_pass_long_prefix_not, regex!("^.bcdefghijklmnopqrstuvwxyz.*$"),
     "abcdefghijklmnopqrstuvwxyz".to_owned()
 });
 
+bench_match!(long_needle1, regex!("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaab"), {
+    repeat("a").take(100_000).collect::<String>() + "b"
+});
+
+bench_match!(long_needle2, regex!("bbbbbbbbbbbbbbbbbbbbbbbbbbbbbba"), {
+    repeat("b").take(100_000).collect::<String>() + "a"
+});
+
 #[cfg(feature = "re-rust")]
 #[bench]
 fn replace_all(b: &mut Bencher) {

benches/src/rust_compile.rs

@@ -29,6 +29,13 @@ fn compile_simple_bytes(b: &mut Bencher) {
     });
 }
 
+#[bench]
+fn compile_simple_full(b: &mut Bencher) {
+    b.iter(|| {
+        regex!(r"^bc(d|e)*$")
+    });
+}
+
 #[bench]
 fn compile_small(b: &mut Bencher) {
     b.iter(|| {
@@ -45,6 +52,13 @@ fn compile_small_bytes(b: &mut Bencher) {
     });
 }
 
+#[bench]
+fn compile_small_full(b: &mut Bencher) {
+    b.iter(|| {
+        regex!(r"\p{L}|\p{N}|\s|.|\d")
+    });
+}
+
 #[bench]
 fn compile_huge(b: &mut Bencher) {
     b.iter(|| {
@@ -60,3 +74,10 @@ fn compile_huge_bytes(b: &mut Bencher) {
         Compiler::new().bytes(true).compile(&[re]).unwrap()
     });
 }
+
+#[bench]
+fn compile_huge_full(b: &mut Bencher) {
+    b.iter(|| {
+        regex!(r"\p{L}{100}")
+    });
+}

regex-debug/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+publish = false
+name = "regex-debug"
+version = "0.1.0"
+authors = ["The Rust Project Developers"]
+license = "MIT/Apache-2.0"
+repository = "https://github.com/rust-lang/regex"
+documentation = "http://doc.rust-lang.org/regex"
+homepage = "https://github.com/rust-lang/regex"
+description = "A tool useful for debugging regular expressions."
+
+[dependencies]
+docopt = "0.6"
+regex = { version = "0.1", path = ".." }
+regex-syntax = { version = "0.3", path = "../regex-syntax" }
+rustc-serialize = "0.3"
+
+[profile.release]
+debug = true