Add shard connection backoff policy

[dkropachev]

Introduce ShardReconnectionPolicy and its implementations:

• NoDelayShardConnectionBackoffPolicy: no delay or concurrency limit, ensures at most one pending connection per host+shard.
• LimitedConcurrencyShardConnectionBackoffPolicy: limits pending concurrent connections to max_concurrent per host with backoff between shard connections.

The idea of this PR is to shift responsibility of scheduling HostConnection._open_connection_to_missing_shard from HostConnection to ShardConnectionBackoffPolicy, that gives ShardConnectionBackoffPolicy control over process of opening connections.

This feature enables finer control over process of creating per shard connections, helping to prevent connections storms.

Fixes: #483

Solutions tested and rejected

Naive delay

Description

Policy would introduce a delay instead of executing connection creation request right away.
Policy would remember last time when connection creation was scheduled to and when it tries to schedule next request it would make sure that there is time between old and new request execution is equal or more than delay it is configured with.

Results

It worked fine when cluster operates in a normal way.

However, during testing with artificial delays, it became clear that this approach breaks down when the time to establish a
connection exceeds the configured delay.
In such cases, connections begin to pile up - the greater the connection initialization time relative to the delay, the faster they accumulate.

This becomes especially problematic during connection storms.
As the cluster becomes overloaded and connection initialization slows down, the delay-based throttling loses its effectiveness. In other words, the more the cluster suffers, the less effective the policy becomes.

Solution

The solution was to give the policy direct control over the connection initialization process.
This allows the policy to track how many connections are currently pending and apply delays after connections are created, rather than before.
That change ensures the policy remains effective even under heavy load.

This behavior is exactly what has been implemented in this PR.

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce.
• I have adjusted the documentation in ./docs/source/.
• I added appropriate Fixes: annotations to PR description.

[mykaul]

Shouldn't we have some warning / info level log when backoff is taking place?

[dkropachev]

Shouldn't we have some warning / info level log when backoff is taking place?

I would rather not do it, it is not useful and can potentially pollute the log

[Lorak-mmk]

Do you know what caused the test failure?

  =================================== FAILURES ===================================
  ___________________________ TypeTests.test_datetype ____________________________
  
  self = <tests.unit.test_types.TypeTests testMethod=test_datetype>
  
      def test_datetype(self):
          now_time_seconds = time.time()
          now_datetime = datetime.datetime.fromtimestamp(now_time_seconds, tz=datetime.timezone.utc)
      
          # Cassandra timestamps in millis
          now_timestamp = now_time_seconds * 1e3
      
          # same results serialized
  >       self.assertEqual(DateType.serialize(now_datetime, 0), DateType.serialize(now_timestamp, 0))
  E       AssertionError: b'\x00\x00\x01\x97<\x17\xda\xf9' != b'\x00\x00\x01\x97<\x17\xda\xf8'

it is a unit test that at the first glance should be fully deterministic. Failure is unexpected.
From the assertion it looks like some off-by-one error.

[dkropachev]

Do you know what caused the test failure?

  =================================== FAILURES ===================================
  ___________________________ TypeTests.test_datetype ____________________________
  
  self = <tests.unit.test_types.TypeTests testMethod=test_datetype>
  
      def test_datetype(self):
          now_time_seconds = time.time()
          now_datetime = datetime.datetime.fromtimestamp(now_time_seconds, tz=datetime.timezone.utc)
      
          # Cassandra timestamps in millis
          now_timestamp = now_time_seconds * 1e3
      
          # same results serialized
  >       self.assertEqual(DateType.serialize(now_datetime, 0), DateType.serialize(now_timestamp, 0))
  E       AssertionError: b'\x00\x00\x01\x97<\x17\xda\xf9' != b'\x00\x00\x01\x97<\x17\xda\xf8'

it is a unit test that at the first glance should be fully deterministic. Failure is unexpected. From the assertion it looks like some off-by-one error.

It is known issue, conversion goes wrong somewhere

[Lorak-mmk]

General comment: integration tests for new policies are definitely needed here.

[mykaul]

The patchset lacks documentation, which would have helped to understand the feature and when/how to use it. Is documentation a separate repo / commit?

[Copilot]

Pull Request Overview

This PR adds shard‐aware reconnection policies with support for scheduling constraints. Key changes include new policy implementations and schedulers in cassandra/policies.py, modifications to connection management in cassandra/pool.py and cassandra/cluster.py, and comprehensive tests in both unit and integration suites to validate the new behavior.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/unit/test_shard_aware.py	Adds tests for both immediate and delayed reconnection behavior using new policies.
tests/unit/test_policies.py	Introduces extensive tests for scope bucket and scheduler behavior.
tests/unit/test_host_connection_pool.py	Updates HostConnectionPool tests to integrate the new scheduler.
tests/integration/long/test_policies.py	Validates backoff policies and correct connection formation across shards.
tests/integration/init.py	Adds a marker for tests designed for Scylla-specific behavior.
cassandra/pool.py	Refactors connection replacements to use the new scheduler instead of direct submission.
cassandra/policies.py	Implements new scheduler classes and backoff policies for shard connections.
cassandra/cluster.py	Exposes a new property and uses the scheduler for initializing shard connections.

[dkropachev]

The patchset lacks documentation, which would have helped to understand the feature and when/how to use it. Is documentation a separate repo / commit?

I have added documentation to all classes.
Way it is done here in repo, small features are documented at the docstring of the classes, big ones get .rst in the docs/.
I personally think it is a small one, so it has no separate file in the docs/, let me know if you want to see one.

[mykaul]

The patchset lacks documentation, which would have helped to understand the feature and when/how to use it. Is documentation a separate repo / commit?

I have added documentation to all classes. Way it is done here in repo, small features are documented at the docstring of the classes, big ones get .rst in the docs/. I personally think it is a small one, so it has no separate file in the docs/, let me know if you want to see one.

I don't think it's such a small feature, and I think details might be missing. I did skim briefly over the code - so I might have missed it - where's the random jitter discussed, so multiple clients when do a concurrent backoff? (again - may have missed it!)

[dkropachev]

The patchset lacks documentation, which would have helped to understand the feature and when/how to use it. Is documentation a separate repo / commit?

I have added documentation to all classes. Way it is done here in repo, small features are documented at the docstring of the classes, big ones get .rst in the docs/. I personally think it is a small one, so it has no separate file in the docs/, let me know if you want to see one.

I don't think it's such a small feature, and I think details might be missing. I did skim briefly over the code - so I might have missed it - where's the random jitter discussed, so multiple clients when do a concurrent backoff? (again - may have missed it!)

ok, I will add it, jitter comes from ExponentialReconnectionPolicy or from ConstantShardConnectionBackoffSchedule

[Lorak-mmk]

Can we somehow write this test without sleeps?
Python Driver tests are already unbearably slow, I don't want to make them any slower unless there is absolutely now way to avoid it.

[dkropachev]

@Lorak-mmk , done, all comments addressed please take a look