Add WatchSet for watching a dynamic set of channels

[joamaki]

When writing a controller that needs to react to specific queries changing it's useful to be able to watch a dynamic set of channels. Add the [WatchSet] utility for this.

Example use:

  var names []string = ...
  ws := statedb.NewWatchSet()
  for {
    for _, name := range names {
      things, watch := myTable.ListWatch(txn, ThingsByName(name))
      ws.Add(watch)
      processThings(things)
    }
    // Wait for things that have a name in 'names' to change.
    if err := ws.Wait(ctx); err != nil {
      // Context cancelled
      break
    }
  }

[github-actions]

$ make test
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading golang.org/x/text v0.14.0
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
ok  	github.com/cilium/statedb	26.423s	coverage: 82.6% of statements
ok  	github.com/cilium/statedb/index	0.005s	coverage: 28.7% of statements
ok  	github.com/cilium/statedb/internal	0.012s	coverage: 46.7% of statements
ok  	github.com/cilium/statedb/part	2.511s	coverage: 83.2% of statements
ok  	github.com/cilium/statedb/reconciler	0.253s	coverage: 89.0% of statements
-----
$ make bench
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                    	  349190	      2977 ns/op	    335910 objects/sec	    2816 B/op	      32 allocs/op
BenchmarkDB_WriteTxn_10-4                   	 1000000	      1002 ns/op	    998180 objects/sec	     743 B/op	      10 allocs/op
BenchmarkDB_WriteTxn_100-4                  	 1513856	       788.8 ns/op	   1267736 objects/sec	     593 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_1000-4                 	 1458288	       836.0 ns/op	   1196111 objects/sec	     549 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4   	  480303	      2397 ns/op	    417266 objects/sec	    1504 B/op	      37 allocs/op
BenchmarkDB_Modify-4                        	    1286	    955636 ns/op	   1046425 objects/sec	  769196 B/op	    8456 allocs/op
BenchmarkDB_GetInsert-4                     	    1191	   1031009 ns/op	    969925 objects/sec	  752423 B/op	    8457 allocs/op
BenchmarkDB_RandomInsert-4                  	    2373	    528976 ns/op	   1890447 objects/sec	  402314 B/op	    7098 allocs/op
BenchmarkDB_RandomReplace-4                 	     309	   3821099 ns/op	    261705 objects/sec	 2356602 B/op	   48575 allocs/op
BenchmarkDB_SequentialInsert-4              	    1464	    830236 ns/op	   1204476 objects/sec	  551041 B/op	    7287 allocs/op
BenchmarkDB_Changes_Baseline-4              	    1224	    964242 ns/op	   1037086 objects/sec	  553119 B/op	   10252 allocs/op
BenchmarkDB_Changes-4                       	     682	   1742756 ns/op	    573804 objects/sec	  993620 B/op	   14494 allocs/op
BenchmarkDB_RandomLookup-4                  	   22428	     53497 ns/op	  18692558 objects/sec	     160 B/op	       1 allocs/op
BenchmarkDB_SequentialLookup-4              	   27344	     44619 ns/op	  22411783 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4         	    7155	    156031 ns/op	   6408984 objects/sec	   93906 B/op	    1044 allocs/op
BenchmarkDB_FullIteration_All-4             	     920	   1455121 ns/op	  68722895 objects/sec	     480 B/op	      12 allocs/op
BenchmarkDB_FullIteration_Get-4             	     219	   5444885 ns/op	  18365891 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4              	  502544	      2395 ns/op	        20.00 50th_µs	        25.00 90th_µs	        70.00 99th_µs	    1590 B/op	      24 allocs/op
PASS
ok  	github.com/cilium/statedb	27.756s
PASS
ok  	github.com/cilium/statedb/index	0.003s
PASS
ok  	github.com/cilium/statedb/internal	0.003s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Insert_RootOnlyWatch-4    	    8642	    135276 ns/op	   7392280 objects/sec	  104171 B/op	    2041 allocs/op
Benchmark_Insert-4                  	    6624	    181648 ns/op	   5505139 objects/sec	  219057 B/op	    3064 allocs/op
Benchmark_Modify-4                  	    7585	    161158 ns/op	   6205104 objects/sec	  212700 B/op	    1206 allocs/op
Benchmark_GetInsert-4               	    6259	    185355 ns/op	   5395072 objects/sec	  212606 B/op	    1205 allocs/op
Benchmark_Replace-4                 	26639566	        45.24 ns/op	  22102239 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Replace_RootOnlyWatch-4   	26793568	        45.09 ns/op	  22177977 objects/sec	       0 B/op	       0 allocs/op
Benchmark_txn_1-4                   	 2801019	       425.6 ns/op	   2349898 objects/sec	     448 B/op	       7 allocs/op
Benchmark_txn_10-4                  	 7190023	       166.3 ns/op	   6012482 objects/sec	     154 B/op	       2 allocs/op
Benchmark_txn_100-4                 	 7843916	       149.6 ns/op	   6683801 objects/sec	     224 B/op	       2 allocs/op
Benchmark_txn_1000-4                	 7095576	       167.0 ns/op	   5987508 objects/sec	     216 B/op	       2 allocs/op
Benchmark_txn_delete_1-4            	 3148944	       377.3 ns/op	   2650233 objects/sec	     856 B/op	       6 allocs/op
Benchmark_txn_delete_10-4           	 8880606	       135.0 ns/op	   7408372 objects/sec	     132 B/op	       1 allocs/op
Benchmark_txn_delete_100-4          	10996095	       107.5 ns/op	   9305312 objects/sec	      60 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4         	11818016	       101.5 ns/op	   9851500 objects/sec	      26 B/op	       1 allocs/op
Benchmark_Get-4                     	   38145	     31770 ns/op	  31476536 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterate-4                 	  174421	      7096 ns/op	 140929277 objects/sec	      80 B/op	       3 allocs/op
Benchmark_Hashmap_Insert-4          	   16046	     74937 ns/op	  13344503 objects/sec	   86562 B/op	      64 allocs/op
Benchmark_Hashmap_Get_Uint64-4      	  159448	      7649 ns/op	 130732085 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4       	  152857	      7776 ns/op	 128595255 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Uint64Map_Random-4        	    1342	    892171 ns/op	   1120863 items/sec	 2700608 B/op	    9038 allocs/op
Benchmark_Uint64Map_Sequential-4    	    1383	    862480 ns/op	   1159449 items/sec	 2492405 B/op	    9749 allocs/op
PASS
ok  	github.com/cilium/statedb/part	28.492s
PASS
ok  	github.com/cilium/statedb/reconciler	0.004s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 3.01 seconds (batch size 1000)
Throughput 332448.22 objects per second
Allocated 6011313 objects, 424799kB bytes, 524320kB bytes still in use

[pippolo84]

Love the idea of chunking and unrolling to boost efficiency, nice! 💯