SA: Improve concurrency robustness of CRL leasing transactions #8030
+32
−19
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a few places within the SA, we use explicit transactions to wrap read-then-update style operations. Because we set the transaction isolation level on a per-session basis, these transactions do not in fact change their isolation level, and therefore generally remain at the default isolation level of REPEATABLE READ.
Unfortunately, we cannot resolve this simply by converting the SELECT statements into SELECT...FOR UPDATE statements: although this would fix the issue by making those queries into locking statements, it also triggers what appears to be an InnoDB bug when many transactions all attempt to select-then-insert into a table with both a primary key and a separate unique key, as the crlShards table has. This causes the integration tests in GitHub Actions, which run with an empty database and therefore use the needToInsert codepath instead of the update codepath, to consistently flake.
Instead, resolve the issue by having the UPDATE statements specify that the value of the leasedUntil column is still the same as was read by the initial SELECT. Although two crl-updaters may still attempt these transactions concurrently, the UPDATE statements will still be fully sequenced, and the latter one will fail.
Part of #8031