Update data-structure-on-rocksdb.md for Hyperloglog #207
Conversation
| ``` | ||
| - `size` records the number of registers, which is a constant, even no element is added. | ||
|
|
||
| #### hyperloglog sub keys-values |
There was a problem hiding this comment.
Would you mind try Bitmap structure? "bitfield" command might help in this case?
There was a problem hiding this comment.
I had changed my previous implementation of the storage format, currently, the format is the same as the bitmap's, and each HLL subkey represents a segment with 1024 registers.
Also, I tried to use bitfield, but finally, I found that it requires byte alignment because of 'memcpy' (it should only be able to operate bits within a byte). The implementation of Redis only uses 6 bits, not aligned, to save more space, so it seems not suitable.
However, another problem arises. When the register is set to 6-bit, unit test runs fail on a few OS releases in CI, but there is no problem when it is set to 8-bit, maybe it has something to do with byte alignment, but I don’t know the reason yet. Therefore, the current implementation still maintains 8 bits without using bitfield.
If I understand something wrong, correct me, thanks.
|
|
||
| ```text | ||
| +---------------+ | ||
| key|version|register_index => | 0s count | |
There was a problem hiding this comment.
we can denote the format of count. In current implementation, the count is an count-to-string ?
|
Thanks! I'll take a careful round these two days |
| ``` | ||
| The register index is calculated using the first 14 bits of the user element's hash value (64 bits), which is why the register array length is 16384. | ||
| The length of consecutive zeros is calculated using the last 50 digits of the hash value of the user key. | ||
| Inspired by the bitmap implementation, hyperloglog divides the register array into 16 segments, each with 1024 registers. |
There was a problem hiding this comment.
So it's now not actually register_index? But something like segment index.
There was a problem hiding this comment.
Yes, it is register_index, the same as the bitmap segment in which the 1st segment subkey is 'subkey0', the 2nd is 'subkey1024', and so on.
No description provided.