Deno KV - support data structures

[ClaudiuCeia]

I think it goes without saying that support for native data structures in a KV store is compelling, as evidenced by an abundance of Redis-compatible data stores (Vercel KV, MemoryDB, Dragonfly, KeyDB, Tidis and probably more).

Deno.KV is already a fast, durable store. If it allowed for more complex storage patterns it could be the primary data storage for a lot of applications. I've successfully used Redis in the past as a primary data store using an hexastore pattern and for now I'm generally looking at MemoryDB for doing this, considering you get durability out of the box. Even support for just hashmaps would allow building an entity-like system, but without lists or sets of some type, secondary indexing for them would be rather difficult.

I think for Deno, Redis compatibility doesn't have to include the RESP protocol since KV is built-in, but support for data structures would be amazing. I think this can probably be implemented in user-land to some degree, but it would entail maintaining a pretty complex key allocation schema and I imagine it would result in subpar performance. Since Redis is already quite well-established, I think taking a similar approach to what was done for the standard library (following Go), could be done here as well.

Right now the only thing stopping me from using KV in any serious matter is the fact that I'm not sure where it fits. As a main data store, it doesn't feel powerful enough and I think there's a lot to consider in terms of data layout. For caching, I rather use Redis (or some flavor of Redis) - I find myself using sorted sets a lot so it's maybe out of commodity. All there's left are simple side projects, in my case at least.

[bartlomieju]

CC @losfair @lucacasonato

[itsfuad]

@ClaudiuCeia Thank you

Deno supports Javascript object storage which is good. But when it comes to update a single field on an object then it get messy. I really miss hash from redis where I can just update a single field.

And KV is still under development. I don't know when It will even release officially.

[skybrian]

KV has basic support for maps by storing values as separate records under the same key prefix. You will need to choose a different prefix for each primary key or index. You can look up individual records by key or list all of them, in either forward or reverse order, in whatever sort orders your indexes allow.

Although it’s low-level compared to SQL, it doesn’t seem hard once you get used to it. You do need to update multiple records (the primary record and its index entries) using a transaction, which requires a loop for retries.

I don’t recommend putting a lot of data under a single key due to the 64k limit on values (on Deno Deploy) which seems quite low. Also, there are limits on how many records you can update in a transaction, so you wouldn’t want to create a lot of indexes for the same table.

The built-in operations are pretty primitive. I’ve seen a few attempts to make higher-level libraries on top of KV, but none seem popular yet. Maybe someone will build something with Redis-like operations?