add roaring_bitmap_rank_many(): get `rank()` values in Bulk #527

longqimin · 2023-11-29T01:57:45Z

this pr provide an api implementation for fast getting rank of many elements in bulk mode.
as well as test/benchmark.

$ microbenchmarks/bench --benchmark_filter="RankMany*"
2023-11-29T09:55:50+08:00
Running microbenchmarks/bench
Run on (36 X 4600 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x18)
  L1 Instruction 32 KiB (x18)
  L2 Unified 1024 KiB (x18)
  L3 Unified 25344 KiB (x1)
Load Average: 0.57, 0.73, 0.61
AVX-2 hardware: yes
AVX-512: supported by compiler
AVX-512 hardware: no
In RAM volume in MiB (estimated): 1.802828
benchmarking other files: You may pass is a data directory as a parameter.
data source: CRoaring/benchmarks/realdata/census1881
number of bitmaps: 200
x64: detected
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-----------------------------------------------------------------------
Benchmark             Time             CPU   Iterations UserCounters...
-----------------------------------------------------------------------
RankManySlow      14070 ns        14030 ns        49497 GHz=4.60848 cycles=61.537k instructions=145.324k
RankMany           5616 ns         5601 ns       125617 GHz=4.42231 cycles=24.734k instructions=75.405k

longqimin · 2023-11-30T05:18:16Z

@lemire Would you take a look ?

lemire · 2023-11-30T05:50:11Z

Of course I will review.

lemire · 2023-11-30T16:54:31Z

include/roaring/roaring.h

+ * it puts rank value of each element in `[begin .. end)` to `ans[]`
+ *
+ * the values in `[begin .. end)` must be sorted in Ascending order;
+ * the `ans` must have enough size.


May be we can be more precise and state what is meant by size. It is size_t length = (end-begin). Correct?

ok. i borrowed a doc from roaring_bitmap_to_uint32_array().

* ans = malloc((end-begin) * sizeof(uint64_t));

lemire · 2023-11-30T16:55:32Z

@longqimin This looks very good to me. Can you check my comment above?

lemire · 2023-11-30T23:17:56Z

include/roaring/containers/array.h

+        uint16_t xhigh = x >> 16;
+        if(xhigh != high) return iter - begin;// stop at next container
+
+        const int32_t idx = binarySearch(arr->array+pos, arr->cardinality-pos, x&0xFFFF);


Suggested change

const int32_t idx = binarySearch(arr->array+pos, arr->cardinality-pos, x&0xFFFF);

const int32_t idx = binarySearch(arr->array+pos, arr->cardinality-pos, (uint16_t)x);

lemire · 2023-11-30T23:18:52Z

include/roaring/containers/array.h

@@ -412,6 +412,29 @@ inline int array_container_rank(const array_container_t *arr, uint16_t x) {
    }
 }

+// bulk version of array_container_rank(); return number of consumed elements
+inline uint32_t array_container_rank_many(const array_container_t *arr, uint64_t start_rank, const uint32_t* begin, const uint32_t* end, uint64_t* ans){
+    const uint16_t high = (*begin) >> 16;


Suggested change

const uint16_t high = (*begin) >> 16;

const uint16_t high = (uint16_t)((*begin) >> 16);

lemire · 2023-11-30T23:19:08Z

include/roaring/containers/array.h

+    const uint32_t* iter = begin;
+    for(; iter != end; iter++) {
+        uint32_t x = *iter;
+        uint16_t xhigh = x >> 16;


Suggested change

uint16_t xhigh = x >> 16;

uint16_t xhigh = (uint16_t)(x >> 16);

lemire · 2023-11-30T23:20:06Z

src/containers/bitset.c

@@ -1232,6 +1232,29 @@ int bitset_container_rank(const bitset_container_t *container, uint16_t x) {
  return sum;
 }

+uint32_t bitset_container_rank_many(const bitset_container_t *container, uint64_t start_rank, const uint32_t* begin, const uint32_t* end, uint64_t* ans){
+  const uint16_t high = (*begin) >> 16;


Suggested change

const uint16_t high = (*begin) >> 16;

const uint16_t high = (uint16_t)((*begin) >> 16);

lemire · 2023-11-30T23:20:20Z

src/containers/bitset.c

+  const uint32_t* iter = begin;
+  for(; iter != end; iter++) {
+      uint32_t x = *iter;
+      uint16_t xhigh = x >> 16;


Suggested change

uint16_t xhigh = x >> 16;

uint16_t xhigh = (uint16_t)(x >> 16);

lemire · 2023-11-30T23:20:41Z

src/containers/bitset.c

+      uint16_t xhigh = x >> 16;
+      if(xhigh != high) return iter - begin; // stop at next container
+
+      uint16_t xlow = x & 0xFFFF;


Suggested change

uint16_t xlow = x & 0xFFFF;

uint16_t xlow = (uint16_t)x;

lemire · 2023-11-30T23:22:35Z

src/containers/run.c

+    int i = 0;
+    for(;iter != end; iter++) {
+        uint32_t x = *iter;
+        uint16_t xhigh = x >> 16;


Suggested change

uint16_t xhigh = x >> 16;

uint16_t xhigh = (uint16_t)(x >> 16);

…ny(begin,end,ans)`

lemire · 2023-12-01T02:15:44Z

Merged. Will be part of the next release. Thanks.

Dr-Emann · 2023-12-01T03:15:36Z

the values in [begin .. end) must be sorted in Ascending order

It might be worth documenting which of the following can occur if this is not upheld:

Incorrect values
Infinite looping
Full Undefined Behavior

roaring_bitmap_rank_many(): get rank() values in Bulk

7180926

longqimin force-pushed the rank-many branch from 06670f7 to 7180926 Compare November 29, 2023 02:03

add C++ Wrapper of roaring_bitmap_rank_many()

8c51a45

longqimin changed the title ~~roaring_bitmap_rank_many(): get rank() values in Bulk~~ add roaring_bitmap_rank_many(): get rank() values in Bulk Nov 30, 2023

lemire reviewed Nov 30, 2023

View reviewed changes

longqimin force-pushed the rank-many branch from 3a6971e to b65d275 Compare December 1, 2023 01:44

explicit cast uint32 to uint16; add more precise document on `rank_ma…

c863be8

…ny(begin,end,ans)`

longqimin force-pushed the rank-many branch from b65d275 to c863be8 Compare December 1, 2023 01:46

lemire merged commit 370199f into RoaringBitmap:master Dec 1, 2023
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add roaring_bitmap_rank_many(): get `rank()` values in Bulk #527

add roaring_bitmap_rank_many(): get `rank()` values in Bulk #527

longqimin commented Nov 29, 2023 •

edited

Loading

longqimin commented Nov 30, 2023

lemire commented Nov 30, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire commented Nov 30, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire Nov 30, 2023

longqimin Dec 1, 2023

lemire commented Dec 1, 2023

Dr-Emann commented Dec 1, 2023

	const int32_t idx = binarySearch(arr->array+pos, arr->cardinality-pos, x&0xFFFF);
	const int32_t idx = binarySearch(arr->array+pos, arr->cardinality-pos, (uint16_t)x);

	const uint16_t high = (*begin) >> 16;
	const uint16_t high = (uint16_t)((*begin) >> 16);

	uint16_t xhigh = x >> 16;
	uint16_t xhigh = (uint16_t)(x >> 16);

add roaring_bitmap_rank_many(): get rank() values in Bulk #527

add roaring_bitmap_rank_many(): get rank() values in Bulk #527

Conversation

longqimin commented Nov 29, 2023 • edited Loading

longqimin commented Nov 30, 2023

lemire commented Nov 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lemire commented Nov 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lemire commented Dec 1, 2023

Dr-Emann commented Dec 1, 2023

add roaring_bitmap_rank_many(): get `rank()` values in Bulk #527

add roaring_bitmap_rank_many(): get `rank()` values in Bulk #527

longqimin commented Nov 29, 2023 •

edited

Loading