You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The purpose of this document is to suggest a high level design to implement transaction support for all vector search commands in all languages.
Prerequisites:
Design for implementing Transactions in GO Lang. #2791: This document gives an overview of transaction design for GO lang. You can also find the high level design for most of the GO and Python code in the prerequisites section of this document. This will give you enough context to understand how everything is structured in our code.
Understanding how JSON module transactions are implemented. Explained in brief in the next 3 sections of this document.
For implementing transactions for any module, the basic idea is to reuse the transaction class for all languages and pass it as an input to the already defined exec() function for the respective language.
Python design for JSON module transaction support:
The following diagram gives an overview of how transactions are implemented for JSON commands in the Python language:
The transaction object used in normal commands is also used for JSON commands as an input to the command.
Inside the definition of each JSON transaction command, the customCommand function(defined inside the transaction object) is called to append the custom command for the respective JSON transaction command to the commands list.
The JSON command then returns this transaction object that contains the updated commands list.
We can then pass this transaction object to the exec() function defined in the clients to run the entire transaction.
Node JS design for JSON module transaction support:
In case of Node JS, we use the idea as Python.
We pass the transaction object as an input to the JSON transaction command.
Inside the JSON command definition we return the updated transaction object containing the updated commands list.
The difference between the design of Python and Node:
Python JSON transaction commands are at the top level of the file, not inside any class.
Node JSON transaction commands are inside a separate GlideMultiJson class.
Note: Difference could be because of the way the modules are imported in the 2 languages. So, it is more of a language specific design decision.
Note: For JSON module implementation for GO lang, use the file and class structure that is the best for Go lang. Don't copy directly from Node or Python. Each language has a different way of importing and therefore, we might need to design according to what is best for the Go lang user/customer.
Design for adding transaction support for Vector search (VSS) commands in all languages:
Implementation wise the design is similar to JSON module. There are few structure improvement suggestions mentioned in the above diagram for NodeJS and Python.
QUEUED
FT.CREATE hash_idx1 ON HASH PREFIX 1 hash: SCHEMA vec AS VEC VECTOR HNSW 6 DIM 2 TYPE FLOAT32
QUEUED
DISTANCE_METRIC L2
QUEUED
FT.CREATE json_idx1 ON JSON PREFIX 1 json: SCHEMA $.vec AS VEC VECTOR HNSW 6 DIM 6 TYPE FLOAT32
QUEUED
DISTANCE_METRIC L2
QUEUED
HSET hash:0 vec "\x00\x00\x00\x00\x00\x00\x00\x00"
QUEUED
HSET hash:1 vec "\x00\x00\x00\x00\x00\x00\x80\xbf"
QUEUED
JSON.SET json:0 . '{"vec":[1,2,3,4,5,6]}'
QUEUED
JSON.SET json:1 . '{"vec":[10,20,30,40,50,60]}'
QUEUED
JSON.SET json:2 . '{"vec":[1.1,1.2,1.3,1.4,1.5,1.6]}'
QUEUED
FT.DROPINDEX json_idx1
QUEUED
FT.CREATE json_idx1 ON JSON PREFIX 1 json: SCHEMA $.vec AS VEC VECTOR FLAT 6 DIM 6 TYPE FLOAT32
QUEUED
DISTANCE_METRIC L2
QUEUED
FT.SEARCH json_idx1 "*=>[KNN 100 @vec $query_vec]" PARAMS 2 query_vec "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" DIALECT 2
QUEUED
EXEC
(error) Index already exists
(error) Index already exists
(integer) 0
(integer) 0
OK
OK
OK
OK
OK
[Important] Possible issues with VSS transaction implementation.
Special cases:
FT commands are not fully transactional. Atomicity for standard commands is guaranteed but it not for FT commands.
Index creation command might be running in the background while the rest of the transaction is being executed. This is one of the special case that must be accounted for while creating transactions using such commands. For example: The index might not be immediately available for search operations and can lead to inconsistency in case of such commands in the transaction, like calling FT.SEARCH immediately after FT.CREATE in a transaction.
Similarly, like index creation, removal of index might is also cause issues as mentioned in point 2.
You can clearly see that json_idx1 is not updated for searching and therefore below FT.SEARCH gives inconsistent search result compared to the FT.SEARCH in the transaction. Sometimes the server can also throw an error saying that the index is under construction.
FT.SEARCH json_idx1 "*"
(integer) 8
"json:4"
"$"
"{"vec":"1"}"
"json:7"
"$"
"{"vec":"1"}"
"json:5"
"$"
"{"vec":"1"}"
"json:2"
"$"
"{"vec":"1"}"
"json:6"
"$"
"{"vec":"1"}"
"json:3"
"$"
"{"vec":"1"}"
"json:1"
"$"
"{"vec":"1"}"
"json:0"
"$"
"{"vec":"1"}"
How to handle such cases:
Avoid using index creation/removal in the same MULTI block as other commands to avoid an inconsistency.
Manual checks. For example, check for existence of indexes before running a set of transactions on the index.
Known issues:
It is only possible to create an index in a MULTI/EXEC transaction if the database is not clustered[https://redis.io/kb/doc/16ekjs4rja/is-it-possible-to-create-an-index-in-a-multi-exec-transaction]
Checklist
Task item 1
Additional Notes
FAQ:
Consistency in code structure and naming of classes/files?
Add a separate class called JsonTransaction, instead of a standalone class for modules?
No response
The text was updated successfully, but these errors were encountered:
prateek-kumar-improving
changed the title
Design for transaction support for Vector search commands
Design for transaction support for Vector Search(VSS) commands
Jan 2, 2025
prateek-kumar-improving
changed the title
Design for transaction support for Vector Search(VSS) commands
Design for transaction support for Vector Search(VSS) commands
Jan 2, 2025
Problem Statement:
The purpose of this document is to suggest a high level design to implement transaction support for all vector search commands in all languages.
Prerequisites:
For implementing transactions for any module, the basic idea is to reuse the transaction class for all languages and pass it as an input to the already defined exec() function for the respective language.
Python design for JSON module transaction support:
The following diagram gives an overview of how transactions are implemented for JSON commands in the Python language:
Code reference: #2684
Node JS design for JSON module transaction support:
The difference between the design of Python and Node:
Note: Difference could be because of the way the modules are imported in the 2 languages. So, it is more of a language specific design decision.
Code reference: #2862
JAVA JSON module transaction support design:
Code reference:
#2691
GO JSON module transaction support design:
Note: For JSON module implementation for GO lang, use the file and class structure that is the best for Go lang. Don't copy directly from Node or Python. Each language has a different way of importing and therefore, we might need to design according to what is best for the Go lang user/customer.
Design for adding transaction support for Vector search (VSS) commands in all languages:
Implementation wise the design is similar to JSON module. There are few structure improvement suggestions mentioned in the above diagram for NodeJS and Python.
Examples:
Using the examples on vector search FT.CREATE documentation(https://docs.aws.amazon.com/memorydb/latest/devguide/vector-search-commands-ft.create.html) to test the commands in MULTI/EXEC transaction block from cli:
[Important] Possible issues with VSS transaction implementation.
Special cases:
Example for this is as follows:
You can clearly see that json_idx1 is not updated for searching and therefore below FT.SEARCH gives inconsistent search result compared to the FT.SEARCH in the transaction. Sometimes the server can also throw an error saying that the index is under construction.
How to handle such cases:
Known issues:
Checklist
Additional Notes
FAQ:
No response
The text was updated successfully, but these errors were encountered: