Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data-structure-on-rocksdb.md for Hyperloglog #207

Merged
merged 5 commits into from
Jul 31, 2024
+40 −0
Merged

Update data-structure-on-rocksdb.md for Hyperloglog #207

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5af4eec
Update data-structure-on-rocksdb.md for Hyperloglog
tutububug Mar 8, 2024
53c56f9
Update data-structure-on-rocksdb.md for hyperloglog
tutububug Mar 8, 2024
2748009
HLL: update storage format
tutububug Apr 2, 2024
100a75a
Merge branch 'main' into patch-1
mapleFU Jul 20, 2024
e99e701
update protocol
mapleFU Jul 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions community/data-structure-on-rocksdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ The values encoded for other data types in flags can be found in the table below
| Stream | 8 |
| BloomFilter| 9 |
| JSON | 10 |
| Hyperloglog| 12 |

In the encoding version `0`, `expire` is stored in seconds and as a 4byte field (32bit integer), `size` is stored as also a 4byte field (32bit integer);
while in the encoding version `1`, `expire` is stored in milliseconds and as a 8byte field (64bit integer), `size` is stored as also a 8byte field (64bit integer).
Expand Down Expand Up @@ -314,3 +315,28 @@ where the `payload` is a string encoded in the corresponding `format`:
| CBOR | 1 |

Also, if we decide to add a more IO-friendly format to avoid reading all payload to the memory before searching an element via JSONPath or seperate a relatively large JSON to multiple key-values, we can take advantage of the `format` field.

## Hyperloglog

Redis hyperloglog can be thought of as a static array with a length of 16384. The array elements are called registers, which are used to store the maximum count of consecutive 0s. This register array is the input parameter for the hyperloglog algorithm.
In Kvrocks, the hyperloglog data structure is stored in following two parts:

#### hyperloglog metadata

```text
+----------+------------+-----------+-----------+
key => | flags | expire | version | size |
| (1byte) | (Ebyte) | (8byte) | (Sbyte) |
+----------+------------+-----------+-----------+
```
- `size` records the number of registers, which is a constant, even no element is added.

#### hyperloglog sub keys-values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind try Bitmap structure? "bitfield" command might help in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had changed my previous implementation of the storage format, currently, the format is the same as the bitmap's, and each HLL subkey represents a segment with 1024 registers.
Also, I tried to use bitfield, but finally, I found that it requires byte alignment because of 'memcpy' (it should only be able to operate bits within a byte). The implementation of Redis only uses 6 bits, not aligned, to save more space, so it seems not suitable.
However, another problem arises. When the register is set to 6-bit, unit test runs fail on a few OS releases in CI, but there is no problem when it is set to 8-bit, maybe it has something to do with byte alignment, but I don’t know the reason yet. Therefore, the current implementation still maintains 8 bits without using bitfield.
If I understand something wrong, correct me, thanks.


```text
+---------------+
key|version|register_index => | 0s count |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can denote the format of count. In current implementation, the count is an count-to-string ?

+---------------+
```
The register index is calculated using the first 14 bits of the user key's hash value (64 bits), which is why the register array length is 16384.
The length of consecutive zeros is calculated using the last 50 digits of the hash value of the user key.