Skip to content

Conversation

mobinasri
Copy link
Contributor

@mobinasri mobinasri commented Dec 10, 2020

This PR is created to solve the issue #12 . Most of the changes are inspired by what has been done for Kyoto Tycoon to support splitting large records.
Again the assumption is that no two processes will attempt to write the same split key at the same time. Since the max record size for redis (512M) is more than Kyoto Tycoon (10M) we should expect less splitting for Redis. Using Redis Lists and Sets enabled a more straightforward implementation of splitting in sonLib. The main notes regarding this implementation are as below:

  • A unique Set _SPLIT_RECORDS is used for storing the keys with split (Listed) values. So by searching this SET in constant time we can check if the key is split or not. When a split record is being removed its key should also be removed from this Set.

  • In order to keep the benefits of pipelining multiple commands, which is mainly used in 3 bulk functions (SET, GET and REMOVE), there are two implementations for checking if a record is split or not. One is recordIsSplitDB() that checks the split-ness by asking the database directly. The other one is recordIsSplitCache() that searchesdb->splitRecords (a stSet copy of all split keys) instead of asking the database directly. The second one does not interfere with pipelining other commands so can be used in bulk functions. db->splitRecords can be filled using the function fillSplitRecordsCache() so it should be called at the beginning of each of the 3 bulk functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant