connection error failed to send request

[itboy87]

What happened?
Connection error on k3s using Quarkus:

 WARN  [com.dat.oss.dri.int.cor.con.ControlConnection] (vert.x-eventloop-thread-5) [s0] Error connecting to Node(endPoint=/127.0.0.1:9042, hostId=null, hashCode=65944760), trying next node (ConnectionInitException: [s0|control|connecting...] Protocol initialization request, step 1 (OPTIONS): failed to send request (io.netty.channel.StacklessClosedChannelException))

Also when i try to port-forward service I'm getting error:

kubectl port-forward svc/cassandra-cluster-dc1-contact-points-service -n k8ssandra-operator 9042:9042
error: cannot attach to *v1.Service: invalid service 'cassandra-cluster-dc1-contact-points-service': Service is defined without a selector

How to reproduce it (as minimally and precisely as possible):

1- install operator

helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace

2- deploy custler:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: k8ssandra-cluster
  namespace: k8ssandra-operator
spec:
  cassandra:
    serverVersion: "5.0.2"
    clusterName: "cassandra-cluster"
    datacenters:
      - metadata:
          name: dc1
        size: 3
        storageConfig:
          cassandraDataVolumeClaimSpec:
            storageClassName: ceph-block
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 25Gi
        config:
          jvmOptions:
            heapSize: 2000Mi
    mgmtAPIHeap: 512Mi
  medusa:
    storageProperties:
      storageProvider: s3_compatible
      storageSecretRef:
        name: prod-k8ssandra-medusa-key
      bucketName: fleeksoft-backup
      maxBackupCount: 30

3- Follow Quarkus cassandra client guide:

quarkus.cassandra.contact-points=cassandra-cluster-dc1-contact-points-service.k8ssandra-operator.svc.cluster.local
quarkus.cassandra.local-datacenter=dc1
quarkus.cassandra.keyspace=k1

4- In code when try to access Cassandra keep getting above mentioned error

Environment

• K8ssandra Operator version:

  cr.k8ssandra.io/k8ssandra/k8ssandra-operator:v1.20.3

• Kubernetes version information:

  kubectl version
  Client Version: v1.32.0
  Kustomize Version: v5.5.0
  Server Version: v1.31.4+k3s1

• Kubernetes cluster kind:

  I have installed k3s on my 3 dedicated servers.

• K8ssandra Operator Logs:

2025-01-11T08:12:26.896Z	INFO	All DCs reconciled	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.896Z	INFO	Reconciling Stargate and Reaper for dc dc1	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.896Z	INFO	reconciling telemetry	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.896Z	INFO	merged TelemetrySpec constructed	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "mergedSpec": null, "cluster": "k8ssandra-cluster"}
2025-01-11T08:12:26.898Z	INFO	Stargate not present	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "Stargate": {"name":"cassandra-cluster-dc1-stargate","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.898Z	INFO	Reaper not present for DC dc1	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "Reaper": {"name":"cassandra-cluster-dc1-reaper","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.919Z	INFO	Finished reconciling the k8ssandracluster	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}}
2025-01-11T08:12:26.940Z	INFO	updated k8ssandracluster status	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "k8ssandra-cluster", "reconcileID": "0bc765b4-1826-4137-98bb-19effc938ea2", "K8ssandraCluster": {"name":"k8ssandra-cluster","namespace":"k8ssandra-operator"}}

K8ssandra all deployments:

kubectl get all -n k8ssandra-operator
NAME                                                    READY   STATUS      RESTARTS        AGE
pod/cassandra-cluster-dc1-default-sts-0                 3/3     Running     5 (6h9m ago)    4d21h
pod/cassandra-cluster-dc1-default-sts-1                 3/3     Running     5 (7h21m ago)   4d21h
pod/cassandra-cluster-dc1-default-sts-2                 3/3     Running     5 (8h ago)      4d21h
pod/cassandra-cluster-dc1-medusa-purge-28939680-gc2k2   0/1     Completed   0               2d9h
pod/cassandra-cluster-dc1-medusa-purge-28941120-fljmq   0/1     Completed   0               33h
pod/cassandra-cluster-dc1-medusa-purge-28942560-q9mgw   0/1     Completed   0               9h
pod/k8ssandra-operator-7c9b89b489-vfg7j                 1/1     Running     0               4d23h
pod/k8ssandra-operator-cass-operator-74db5bd4db-pzlnq   1/1     Running     0               4d23h

NAME                                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
service/cassandra-cluster-dc1-additional-seed-service      ClusterIP   None            <none>        <none>                                         4d21h
service/cassandra-cluster-dc1-all-pods-service             ClusterIP   None            <none>        9042/TCP,8080/TCP,9103/TCP,9000/TCP            4d21h
service/cassandra-cluster-dc1-contact-points-service       ClusterIP   None            <none>        9000/TCP,9042/TCP,8080/TCP,9103/TCP            4d21h
service/cassandra-cluster-dc1-service                      ClusterIP   None            <none>        9042/TCP,9142/TCP,8080/TCP,9103/TCP,9000/TCP   4d21h
service/cassandra-cluster-seed-service                     ClusterIP   None            <none>        <none>                                         4d21h
service/k8ssandra-operator-cass-operator-webhook-service   ClusterIP   10.43.137.113   <none>        443/TCP                                        4d23h
service/k8ssandra-operator-webhook-service                 ClusterIP   10.43.79.66     <none>        443/TCP                                        4d23h

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/k8ssandra-operator                 1/1     1            1           4d23h
deployment.apps/k8ssandra-operator-cass-operator   1/1     1            1           4d23h

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/k8ssandra-operator-7c9b89b489                 1         1         1       4d23h
replicaset.apps/k8ssandra-operator-cass-operator-74db5bd4db   1         1         1       4d23h

NAME                                                 READY   AGE
statefulset.apps/cassandra-cluster-dc1-default-sts   3/3     4d21h

NAME                                               SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/cassandra-cluster-dc1-medusa-purge   0 0 * * *   <none>     False     0        9h              4d21h

NAME                                                    STATUS     COMPLETIONS   DURATION   AGE
job.batch/cassandra-cluster-dc1-medusa-purge-28939680   Complete   1/1           13s        2d9h
job.batch/cassandra-cluster-dc1-medusa-purge-28941120   Complete   1/1           3s         33h
job.batch/cassandra-cluster-dc1-medusa-purge-28942560   Complete   1/1           3s         9h

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: K8OP-304