Add SubnetAddressTranslator to translate Cassandra node IPs from private network based on its subnet mask #2013

jahstreet · 2025-02-10T11:22:28Z

When running Cassandra in a private network and accessing it from outside of that private network via some kind of proxy, we have an option to use FixedHostNameAddressTranslator. But when we want to set it up in a HA way and have more control over latencies in multi-datacenter deployments, that is not enough.

This PR proposes a SubnetAddressTranslator, which translates Cassandra node IP addresses based on the match to the configured subnet IP range (CIDR notation). The assumption is that each Cassandra datacenter nodes belong to different subnets not having intersecting IP ranges, which is the usual configuration for multi-DC Kubernetes and K8ssandra, for example.

jahstreet

Is there any additional documentation I should update with this change?

core/pom.xml

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

core/pom.xml

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

core/src/main/resources/reference.conf

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

core/src/main/resources/reference.conf

manual/core/address_resolution/README.md

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

tolbertam

Excellent work @jahstreet, thank you! Have some suggestions, but I'm +1 either way.

tolbertam · 2025-02-14T22:31:13Z

core/src/main/resources/reference.conf

+    # default-address = "cassandra.datacenter1.com:9042"
+    # Whether to resolve the addresses once on initialization (if true) or on each node (re-)connection (if false).
+    # If not configured, defaults to false.
+    # resolve-addresses = false


👍 while resolve-contact-points defaults to true, I was thinking about whether we should do the same here, but I think you probably will generally use a DNS name that might change its backing IPs from time to time, so makes sense for this to be false. Was that what you were thinking as well?

100%, I expect proxy to have at least 2 replicas eligible for "periodic" restarts, that was the intuition to decide here.

...va/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslatorTest.java

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

jahstreet · 2025-02-15T17:37:33Z

Excellent work @jahstreet, thank you! Have some suggestions, but I'm +1 either way.

Thanks! I will push a commit with annotations till Monday morning.

manual/core/address_resolution/README.md

core/src/main/java/com/datastax/oss/driver/internal/core/util/AddressUtils.java

core/src/main/java/com/datastax/oss/driver/internal/core/ContactPoints.java

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

absurdfarce

I like the overall idea, just a few things I think need to be tweaked here. Moving the DriverOptions around shouldn't be too bad but I am a bit concerned about adding a new dependency just for this. I'm also not sure I love the additional exception handling code that's been added now... but I can be convinced on either point.

...n/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslator.java

absurdfarce · 2025-03-17T21:15:18Z

core/pom.xml

+      <groupId>com.github.seancfoley</groupId>
+      <artifactId>ipaddress</artifactId>
+      <optional>true</optional>
+    </dependency>


I really don't love the inclusion of another dependency here, even if it's an optional one. It's only used in one class (near as I can tell)... is there really no way to get the functionality we need without adding this in?

First thing that comes to mind is implementing it ourselves (or in other words copying it over from the library). Lemme evaluate how much of the util code is needed.

We need at least the following functionality to work with subnets here:

Validate subnet string is in a prefix block format

Check if subnet contains IP address

All for IPv4 and IPv6

The library is quite big, so copying over its parts is an overkill.
Then the alternative is to implement these functions ourselves.
Looking into it.

A bit of vibe-coding and we can have it with around 100 lines of code. Will work on integrating a change.

Submitted the change and dropped the dependency, PTAL 🙏 .

absurdfarce · 2025-03-17T22:52:16Z

core/src/main/java/com/datastax/oss/driver/internal/core/ContactPoints.java

+        addresses = AddressUtils.extract(spec, resolve);
+      } catch (RuntimeException e) {
+        LOG.warn("Ignoring invalid contact point {} ({})", spec, e.getMessage(), e);
+      }


Could just continue here to next iteration of the outer for loop. You know addresses is the empty set at this point so there's no point iterating over it below.

As I look at this now it feels like we had to make this a bit more complicated because AddressUtils.extract() now throws exceptions in most cases rather than just logging errors and returning an empty set. Was there a particular reason for this change? It's not immediately clear the exceptions buy you much here.

Previously, #extract code was used only in this class and we logged errors together with reasons of these errors. Now the info about reasons is moved to util method, which is called from multiple places. In this class, I aimed to keep logging (as well as other functionality) as close to the origin as seemed possible to avoid opinionated refactoring, so I needed a way to get reasons of errors from the utility #extract to log them together with the context logs.
Happy to agree on the way it should look like and change accordingly.

absurdfarce · 2025-03-17T22:52:33Z

core/src/main/java/com/datastax/oss/driver/internal/core/ContactPoints.java

+            "Contact point {} resolves to multiple addresses, will use them all ({})",
+            spec,
+            addresses);
+      }


Does this log message offer us much useful information?

Same as above, it was there so I kept it as is.
As for me, this log is a good additional info when debugging failed to connect issues. Like, one could be surprised to see the client failed to connect logs where contact points do not match the configured ones.
What is your opinion on the need of it?

absurdfarce · 2025-04-01T22:31:50Z

Apologies, this is on my list but I haven't made it back to reconsider the updated comments in this review. I appreciate your patience @jahstreet!

FWIW I have added this to the 4.19.1 release planning doc under the working assumption that we'll almost certainly get this in in some form we can all agree on.

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddress.java

...rc/test/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTest.java

core/src/test/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetTest.java

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java

absurdfarce

Nice work @jahstreet! I very much like the idea of bringing the subnet management logic into code in this PR; I appreciate the efforts to avoid the introduction of an extra external dependency.

I want to run this through a full set of unit and integration tests but it looks like there's a few steps that need to happen before we can get this to build with Maven. The changes I'm asking for in this PR should get you to a spot where we can build and run tests... well, these changes plus a "mvn com.coveo:fmt-maven-plugin:format" but you get my point. :)

jahstreet · 2025-06-07T11:15:31Z

Deal, thx for the feedback.
Will address it asap.

UPD: done.

jahstreet · 2025-06-10T19:36:11Z

Suggested commit message:

"Add SubnetAddressTranslator to translate Cassandra node IPs from private network based on its subnet mask"

absurdfarce · 2025-06-10T19:55:58Z

Confirmed that the build looks good locally with the recent changes from @jahstreet . Kicking off a DataStax Jenkins run now to confirm that we haven't had any unexpected regressions.

absurdfarce · 2025-06-10T23:09:21Z

Bah, Jenkins failed with complaints like the following:

[2025-06-10T22:25:35.767Z] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ java-driver-core ---
[2025-06-10T22:25:35.767Z] [INFO] Compiling 796 source files to /home/jenkins/workspace/drivers_java_oss_PR-2013/core/target/classes
[2025-06-10T22:25:38.559Z] [INFO] -------------------------------------------------------------
[2025-06-10T22:25:38.559Z] [ERROR] COMPILATION ERROR : 
[2025-06-10T22:25:38.559Z] [INFO] -------------------------------------------------------------
[2025-06-10T22:25:38.559Z] [ERROR] /home/jenkins/workspace/drivers_java_oss_PR-2013/core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java:[21,30] package com.google.common.base does not exist
[2025-06-10T22:25:38.559Z] [INFO] 1 error
[2025-06-10T22:25:38.559Z] [INFO] -------------------------------------------------------------

Weird thing was that local builds were just fine. This made absolutely no sense to me; I spent the afternoon trying various combinations of Maven versions and POM settings to see what might account for the difference.

Naturally the answer was pretty much right in front of me the whole time. This PR was branched off from a point in 4.x before this commit went in. And that commit fixes precisely this behaviour. Once I included this change in my local checkout of the PR branch I was able to easily reproduce the failure above in my local build.

@jahstreet can you merge 4.x into your PR branch so that we can get this fix on your branch as well? I think such a merge + the other work you've already done should enable us to run a full build on our Jenkins server.

Thanks!

absurdfarce · 2025-06-10T23:13:12Z

Oh, almost forgot... the reason the build now fails with that change in place is because you're using the normal Guava packages here (and potentially other places in your PR... I admit I haven't checked yet). This should be changed to use the shaded packages for Guava along the lines of the VisibleForTesting import just above your addition.

jahstreet · 2025-06-11T06:06:44Z

Sorry to make you spend much time time on it. Rebase is obviously a good thing to have always. On it.

absurdfarce · 2025-06-11T06:16:42Z

No worries @jahstreet ... I'm just happy we figured it out and have a clear path now to get this in!

...va/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslatorTest.java

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java

jahstreet · 2025-06-11T07:43:36Z

Did the visual check and corrected guava references. Hope now it is in a good shape 🤞 .

@absurdfarce how (with which maven args) do you build/compile locally? I wasn't able to reproduce the dependency errors with basic mvn clean package -DskipTests

absurdfarce · 2025-06-11T19:31:17Z

These changes look good @jahstreet , thanks for jumping on this so fast! A local test build seemed to succeed without a problem so I've kicked off a Jenkins build to (a) confirm that we're good and (b) throw all the unit and integration tests at it. I'm cautiously optimistic we've got it this time!

As for what I've been running... most of the time I've been using the same command used by the DataStax Jenkinsfile (mvn -B -V install -DskipTests -Dmaven.javadoc.skip=true) but I've been able to reproduce it with the even simpler mvn clean install -DskipTests=true. Note that I did not see this failure in my local test builds until I manually added the change to make guava-shaded an optional dependency; prior to that change the only place I saw it was on Jenkins. And I'm pretty sure that's because Jenkins was merging the changes in the PR into the current state of 4.x... which meant the commit making the guava-shaded dependency optional was brought into play. That's just a hypothesis but it seems to fit all the facts.

absurdfarce · 2025-06-11T20:21:22Z

Hey @jahstreet, the Jenkins run is still ongoing but I'm seeing multiple test failures for SubnetTest and SubnetAddressTest. Haven't looked into it too deeply but most (all?) of the failures look to be from an empty string being passed to the Integer.parseInt() call in Subnet.parse(). I was able to repro immediately by running the test locally in my IDE.

absurdfarce · 2025-06-11T20:24:12Z

Oh, I see... fortunately this one's an easy fix:

diff --git a/core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java b/core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java
index ec83626c5..7c25e94e2 100644
--- a/core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java
+++ b/core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java
@@ -45,7 +45,7 @@ class Subnet {
   }
 
   static Subnet parse(String subnetCIDR) throws UnknownHostException {
-    List<String> parts = Splitter.on("/").splitToList("/");
+    List<String> parts = Splitter.on("/").splitToList(subnetCIDR);
     if (parts.size() != 2) {
       throw new IllegalArgumentException("Invalid subnet: " + subnetCIDR);
     }

…ate network based on its subnet mask

jahstreet · 2025-06-11T22:46:17Z

Thx for the hints @absurdfarce 🦅 👁️ . Indeed, the fast refactoring outputted the tiny bug. Thx for checking.
Double checked the tests locally ✅ .
🤞 for another CI run. Will check it in my CEST morning.

jahstreet commented Feb 10, 2025

View reviewed changes

core/pom.xml Outdated Show resolved Hide resolved

tolbertam reviewed Feb 10, 2025

View reviewed changes

tolbertam reviewed Feb 12, 2025

View reviewed changes

tolbertam approved these changes Feb 14, 2025

View reviewed changes

aratno reviewed Feb 17, 2025

View reviewed changes

aratno approved these changes Feb 18, 2025

View reviewed changes

absurdfarce requested changes Mar 17, 2025

View reviewed changes

jahstreet force-pushed the add-subnet-address-translator branch from cf06929 to 85d5931 Compare March 23, 2025 11:55

jahstreet requested a review from absurdfarce May 16, 2025 11:19

tolbertam self-requested a review June 2, 2025 17:40

aratno self-requested a review June 2, 2025 17:41

absurdfarce reviewed Jun 4, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java Show resolved Hide resolved

absurdfarce reviewed Jun 4, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddress.java Show resolved Hide resolved

absurdfarce reviewed Jun 4, 2025

View reviewed changes

...rc/test/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTest.java Show resolved Hide resolved

absurdfarce reviewed Jun 4, 2025

View reviewed changes

core/src/test/java/com/datastax/oss/driver/internal/core/addresstranslation/SubnetTest.java Show resolved Hide resolved

absurdfarce reviewed Jun 4, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java Outdated Show resolved Hide resolved

absurdfarce reviewed Jun 4, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java Show resolved Hide resolved

absurdfarce requested changes Jun 4, 2025

View reviewed changes

jahstreet force-pushed the add-subnet-address-translator branch from 49a2ebf to 0f00215 Compare June 10, 2025 19:37

jahstreet changed the title ~~Add SubnetAddressTranslator~~ Add SubnetAddressTranslator to translate Cassandra node IPs from private network based on its subnet mask Jun 11, 2025

jahstreet commented Jun 11, 2025

View reviewed changes

...va/com/datastax/oss/driver/internal/core/addresstranslation/SubnetAddressTranslatorTest.java Outdated Show resolved Hide resolved

jahstreet commented Jun 11, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/internal/core/addresstranslation/Subnet.java Outdated Show resolved Hide resolved

jahstreet force-pushed the add-subnet-address-translator branch from 0f00215 to c72413e Compare June 11, 2025 07:49

Add SubnetAddressTranslator to translate Cassandra node IPs from priv…

9bc06ca

…ate network based on its subnet mask

jahstreet force-pushed the add-subnet-address-translator branch from c72413e to 9bc06ca Compare June 11, 2025 22:33

Add SubnetAddressTranslator to translate Cassandra node IPs from private network based on its subnet mask #2013

Are you sure you want to change the base?

Add SubnetAddressTranslator to translate Cassandra node IPs from private network based on its subnet mask #2013

Conversation

jahstreet commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jahstreet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tolbertam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jahstreet commented Feb 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

absurdfarce left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jahstreet Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

absurdfarce commented Apr 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

absurdfarce left a comment

Choose a reason for hiding this comment

jahstreet commented Feb 10, 2025 •

edited

Loading

jahstreet Mar 18, 2025 •

edited

Loading

jahstreet commented Jun 7, 2025 •

edited

Loading

jahstreet commented Jun 11, 2025 •

edited

Loading