Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KVStore support #1389

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

KVStore support #1389

wants to merge 5 commits into from

Conversation

hbs
Copy link
Collaborator

@hbs hbs commented Jan 9, 2025

This PR adds a Key/Value store feature to Warp 10, enabling the storage of arbitrary Key/Value pairs for the following storage engines:

  • LevelDB
  • in memory
  • FoundationDB for both standalone+ and distributed versions

Additionally, stored K/V pairs can be replicated via the Datalog mechanism.

WarpLib functions KVSTORE and KVLOAD allow to store and retrieved stored K/V pairs.

Retrieval is either range or key based, allowing to scan the key space or access specific keys explicitely.

Tenant prefixes are supported for FoundationDB based versions.

Optional custom prefix can be added to keys prior to storage to allow for user segregation by key spaces.

@hbs hbs requested review from stggn and pi-r-p January 9, 2025 07:30
@pi-r-p
Copy link
Contributor

pi-r-p commented Jan 30, 2025

Is it possible to support cast of key to string in KVLOAD ?

Usage :

{  'key' $bytes } $wt KVSTORE
{
  'token' $rt
  'keys' [ 'key' ]
  'keycast' true           // default to false ?
} KVLOAD 'key' GET 

@pi-r-p
Copy link
Contributor

pi-r-p commented Jan 30, 2025

BYTES as MAP keys are not usable (at all). Here is a test:

{
  'drink' 'brawndo'
} $wt KVSTORE 


{
  'token' $rt
  'keys' [  'drink' ]
}
KVLOAD 'r' STORE

$r
<% 
  [ 'k' 'v' ] STORE 
  $k 'ascii' BYTES->
  $v 'ascii' BYTES->
%> F MMAP    /// ok, great, see the drink key

$r 'drink' 'ascii' ->BYTES  GET   // null


@hbs
Copy link
Collaborator Author

hbs commented Jan 31, 2025

PTAL

@pi-r-p
Copy link
Contributor

pi-r-p commented Jan 31, 2025

About new configurations not in the conf templates..

missing: datalog.manager.logkvstores , 'warp.kvstore.maxk' 'warp.kvstore.maxv'

What is the link between this PR and egress.fdb.retrylimit ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants