Filtersets API

Filtersets are a novel way of querying spans in ENTRACE, inspired by
nixpkgs's Filesets.
Filtersets are a generalization of the old en_filter function.

A Filterset is a description of a set of span ids. For example, a
Primitive filterset is a list of ids or a range of ids; a Filter
filterset is the set of ids that match the filter predicate.

These description compose via relations such as AND, OR, NOT, etc, this allows filtersets to
have an expressive SQL-like API with a minimal backend implementation.
In a way, filtersets encode dataflow itself.

The other defining feature of a Filterset is that it can be
*materialized* (turned into a Vec<id>) freely at any time, which aids
debugging, and makes it trivial to switch between Filtersets and
"traditional" lua when desired.

The performance speedup (1.3-2x) of filtersets primarily comes from the fact that
via filtersets, many allocations in the Lua evaluator can be avoided;
and that the Filterset evaluator includes a "normalization" step, which
tries to rewrite the filterset to the fastest possible query.

Filtersets have a long way to go still. The filterset evaluator itself
is very generic on purpose; it assumes nothing about ENTRACE itself.
In the future, it can (and should) be extracted from this project, as it
may be useful in other software as well.

There is still a long way to go with filtersets, many possible
optmizations (e.g. introducing a DNF type in the evaluator to reduce
scans in queries having OR), a BlackBox type that materializes its
source without rewriting; but they are already powerful enough to speed
up queries; and to provide a nicer syntax.

Author: algorithmiker

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "bench"
-version = "0.1.0"
+version = "0.1.2"
 edition = "2024"
 
 [dependencies]

bench/Cargo.toml

@@ -0,0 +1,54 @@
+THE FILTERSET CALCULUS
+---
+
+### TYPES
+
+```rust
+type FiltersetId = usize;
+struct Predicate {
+    attr: String,
+    rel: Ordering,
+    con: EnValue,
+}
+#[derive(Debug)]
+enum Filterset {
+    Dead,
+    Primitive(Roaring),
+    Rel(Predicate, FiltersetId),
+    RelIntersect(Vec<Predicate>, FiltersetId),
+    RelUnion(Vec<Predicate>, FiltersetId),
+    And(Vec<FiltersetId>),
+    Or(Vec<FiltersetId>),
+    Not(FiltersetId),
+}
+```
+
+### REWRITE RULES
+
+**Flattening**
+
+And([... And(children) ...]) -> And(flattened)
+Or([... Or(children) ...])  -> Or(flattened)
+Not(Not(X)) -> X
+
+**Eliminating trivial ops (unimplemeneted)**
+And([A]) -> A
+Or([A]) -> A
+
+**REL composition**
+
+I.   Rel(p1, Rel(p2, A)) -> RelIntersect([p1, p2], A)
+II.  Rel(p, RelIntersect(ps, A)) -> RelIntersect([p] + ps, A)
+III. RelIntersect(ps1, RelIntersect(ps2, A)) -> RelIntersect(ps1 + ps2, A)
+IV.  RelIntersect(ps, Rel(p, A)) -> RelIntersect(ps + [p], A)
+V.   RelUnion(ps1, RelUnion(ps2, A)) -> RelUnion(ps1 + ps2, A)
+
+
+**Flattening on the same level (unimplemeneted)**
+And([RelIntersect(ps, A), RelIntersect(ps2, A), RelIntersect(_, B) ...])
+  -> And([RelIntersect(ps1+ps2, A), RelIntersect(_, B)])
+    - This could be hard to detect, instead there could be just:
+      And([RelIntersect(ps_1, A), .. RelIntersect(ps_n, A)]) -> RelIntersect(ps_1+..+ps_n, A)
+Or([RelUnion(ps, A), RelUnion(ps2, A), RelUnion(_, B) ...])
+    -> Or([RelUnion(ps1+ps2, A), RelIntersect(_, B)])
+    - same applies here.

docs/filtersets.md

@@ -1,6 +1,6 @@
 [package]
 name = "entrace_query"
-version = "0.1.1"
+version = "0.1.2"
 edition = "2024"
 
 [dependencies]
@@ -9,3 +9,8 @@ entrace_core = { version = "0.1.1", path = "../entrace_core/" }
 anyhow = "1.0.100"
 memchr = "2.7.6"
 thiserror = "2.0.17"
+roaring = "0.11.2"
+
+[[bin]]
+name="entrace-query-test"
+path="src/main.rs"