Atomicity Between WATCH and EXEC

[suxb201]

How can we ensure the atomicity between WATCH and EXEC?

WATCH k1
MULTI
SET k2 v2
EXEC

When k1 is modified, the current conn context's watched_dirty flag is set to true. If watched_dirty is false, EXEC will begin execution. However, how can we ensure that the check of this flag and the execution of the transaction itself are atomic?

In other words, there is a potential issue where k1 could be modified during the EXEC execution, or k1 could be modified just before EXEC is executed and after the flag is checked.

[romange]

You are right, the implementation of this feature is indeed lacks the atomicity guarantees. The right way to do it would be to check if something is modified during the schedule phase of MULTI transaction, when run locally on the shard thread.

[romange]

Please open a bug

[romange]

@suxb201 if you would like to work at DragonflyDB - you just passed the interview.

[suxb201]

We have a similar multi-threaded database, but its implementation is completely different from DragonflyDB. We are almost "shared-everything" and use Two-Phase Locking (2PL) to handle conflicts. The advantage is that we maintain 100% compatibility with Redis, but we lose further scalability in performance. I have not deeply studied DragonflyDB and have a few curiosities:

1. Is optimistic concurrency really better than pessimistic concurrency? I feel that the cost of rollback operations is high, and comparing transaction IDs can easily trigger rollbacks. For example:
   a. txn1
   b. txn2 txn1
   c. txn2
   This might be a legal sequence, but txn1 could roll back because the transaction IDs are not incrementing.
2. In my testing, the primary node showed significant performance loss when setting up master-slave replication, which is unacceptable for operations.

[romange]

1. I have not seen significant problems with our TX scheduling in our tests and real world use-cases.
2. We do not observe this. Of course the replication requires additional CPU but we did not observe "significant performance loss". Can you please describe your test setup?

[suxb201]

2. I didn't explain it clearly. A long time ago, I did some tests where I wrote a large amount of data to the master node and kept the high write traffic going. At the moment when master-slave replication is set up, the master node will block some requests intermittently for about 10 seconds. I guess this is because the master node needs to perform some COW snapshots. I didn't specifically measure the impact, but it seems to be worse than a regular fork().

[romange]

If you kept the testing code by any chance I would love to try and reproduce it. We actually do not do much COW, but if there are huge sets, hashmaps, lists etc - we do serialize them atomically, which may create such effect. But with regular medium to large values I am not aware of such effect.

[suxb201]

If I test DragonflyDB again in the future, I will try to reproduce this scenario.