Kubernetes Operator for Nessie #7967

adutra · 2024-01-19T14:22:40Z

No description provided.

gradle/libs.versions.toml

operator/README.md

operator/src/main/kubernetes/nessie.svg

operator/src/main/java/org/projectnessie/operator/resource/NessieSpec.java

operator/src/main/java/org/projectnessie/operator/resource/NessieImage.java

operator/src/main/java/org/projectnessie/operator/resource/NessieSpec.java

operator/PROJECT

operator/src/main/java/org/projectnessie/operator/resource/Nessie.java

operator/src/main/java/org/projectnessie/operator/resource/NessieAwsSecret.java

operator/Makefile

operator/src/main/java/org/projectnessie/operator/resource/NessieAuthentication.java

.../main/java/org/projectnessie/operator/dependent/HorizontalPodAutoscalerV2Beta1Dependent.java

operator/src/main/java/org/projectnessie/operator/helper/EventsHelper.java

operator/src/main/java/org/projectnessie/operator/resource/NessieVersionStore.java

operator/src/main/java/org/projectnessie/operator/resource/NessieAwsCredentials.java

operator/src/main/java/org/projectnessie/operator/resource/NessieImage.java

snazy

Not really reviewing yet - just adding comments "randomly".

dorsegal · 2024-07-19T13:24:59Z

I went through the CRDs, reviewed some of the code and this is a good start for an operator. The operator framework is a decent fit and cuts down on the amount of logic to write and lifecycle to manage.

Note, I have not used it in anger, so I don't have any comments on the usability aspect beyond being able to say the CRDs are well formed and logica, in otherwords, I know what I'm configuring just by reading them.

Also, I think you may find that you will ultimately get a request for an operator UI as what you see here https://github.com/zalando/postgres-operator/blob/master/docs/operator-ui.md

Final comment, generally speaking, it does seem helm charts still remain very popular, so you will have to maintain both this and a helm chart, if there are not resources and you have to pick one I would pick the helm chart, but you will have users that prefer the operator, or even demand it.

Note about the helm chart, I feel we (devops) want to use operators if we can but want to do it using regular helm charts. There are a lot of operators support installation + configuration using helm chart.

adutra · 2024-07-19T13:48:00Z

@dorsegal our plan is to distribute the operator using Helm charts initially. Operator Hub would be the ultimate goal, but that requires a lot more work.

snazy · 2024-07-19T15:05:44Z

operator/build.gradle.kts

+  intTestImplementation(project(":nessie-keycloak-testcontainer"))
+  intTestImplementation(project(":nessie-container-spec-helper"))
+
+  intTestCompileOnly(libs.microprofile.openapi)


Suggested change

intTestCompileOnly(libs.microprofile.openapi)

intTestCompileOnly(libs.microprofile.openapi)

intTestCompileOnly(libs.immutables.value.annotations)

snazy · 2024-07-19T15:13:05Z

operator/src/intTest/resources/org/projectnessie/operator/testinfra/Dockerfile-bigtable-version

+# Version is managed by Renovate - do not edit.
+# See https://cloud.google.com/sdk/docs/downloads-docker#docker_image_options
+# Use debian_component_based because it supports linux/arm
+FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:483.0.0-debian_component_based


Suggested change

FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:483.0.0-debian_component_based

FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:484.0.0-debian_component_based

snazy · 2024-07-19T15:14:32Z

operator/src/intTest/resources/org/projectnessie/operator/testinfra/Dockerfile-k3s-version

@@ -0,0 +1,3 @@
+# Dockerfile to provide the image name and tag to a test.
+# Version is managed by Renovate - do not edit.
+FROM rancher/k3s:v1.30.2-k3s2


Suggested change

FROM rancher/k3s:v1.30.2-k3s2

FROM docker.io/rancher/k3s:v1.30.2-k3s2

snazy · 2024-07-19T15:14:38Z

operator/src/intTest/resources/org/projectnessie/operator/testinfra/Dockerfile-mongo-version

@@ -0,0 +1,3 @@
+# Dockerfile to provide the image name and tag to a test.
+# Version is managed by Renovate - do not edit.
+FROM mongo:7.0.12


Suggested change

FROM mongo:7.0.12

FROM docker.io/mongo:7.0.12

snazy · 2024-07-19T15:14:44Z

operator/src/intTest/resources/org/projectnessie/operator/testinfra/Dockerfile-postgres-version

@@ -0,0 +1,3 @@
+# Dockerfile to provide the image name and tag to a test.
+# Version is managed by Renovate - do not edit.
+FROM postgres:16.3


Suggested change

FROM postgres:16.3

FROM docker.io/postgres:16.3

snazy · 2024-07-19T15:37:21Z

.github/workflows/ci.yml

@@ -446,6 +456,61 @@ jobs:
          job-name: 'int-test-quarkus'
          java-version: ${{ matrix.java-version }}

+  int-test-operator:


Just a heads-up - when it's merged.

snazy · 2024-07-19T15:38:01Z

...jectnessie/operator/reconciler/nessie/dependent/HorizontalPodAutoscalerV2Beta1Dependent.java

+import org.projectnessie.operator.reconciler.nessie.resource.options.AutoscalingOptions;
+
+@KubernetesDependent(labelSelector = NessieReconciler.DEPENDENT_RESOURCES_SELECTOR)
+public class HorizontalPodAutoscalerV2Beta1Dependent


...ator/src/intTest/java/org/projectnessie/operator/testinfra/K3sContainerLifecycleManager.java

snazy · 2024-07-22T07:25:58Z

...src/intTest/java/org/projectnessie/operator/testinfra/AbstractContainerLifecycleManager.java

+    container = createContainer();
+    container
+        .withNetwork(Network.SHARED)
+        .withLogConsumer(new Slf4jLogConsumer(logger))


Suggested change

.withLogConsumer(new Slf4jLogConsumer(logger))

.withLogConsumer(new Slf4jLogConsumer(logger).withPrefix(container.getDockerImageName()))

snazy · 2024-07-22T08:03:29Z

...src/intTest/java/org/projectnessie/operator/testinfra/AbstractContainerLifecycleManager.java

+    implements QuarkusTestResourceLifecycleManager {
+
+  protected C container;
+  protected String inDockerIpAddress;


Suggested change

protected String inDockerIpAddress;

private String inDockerIpAddress;

protected String inDockerIpAddress() {

if (inDockerIpAddress == null) {

inDockerIpAddress = getInDockerIpAddress();

}

return inDockerIpAddress;

}

snazy · 2024-07-22T08:03:49Z

...src/intTest/java/org/projectnessie/operator/testinfra/AbstractContainerLifecycleManager.java

+    inDockerIpAddress =
+        Objects.requireNonNull(
+            getInDockerIpAddress(), "could not determine container's in-docker IP address");


Suggested change

inDockerIpAddress =

Objects.requireNonNull(

getInDockerIpAddress(), "could not determine container's in-docker IP address");

snazy · 2024-07-22T08:04:10Z

...src/intTest/java/org/projectnessie/operator/testinfra/AbstractContainerLifecycleManager.java

+
+  @Override
+  public Map<String, String> start() {
+    Logger logger = LoggerFactory.getLogger(getClass());


Suggested change

Logger logger = LoggerFactory.getLogger(getClass());

inDockerIpAddress = null;

Logger logger = LoggerFactory.getLogger(getClass());

snazy · 2024-07-22T08:07:37Z

Still can't get the integration tests to work. K3s doesn't work properly in rootless. There seem to be some ways to get that working - but currently hitting this error from k3s.

expected sysctl value \"net.ipv4.ip_forward\" to be \"1\", got \"0\"; try adding \"net.ipv4.ip_forward=1\" to /etc/sysctl.conf and running `sudo sysctl --system`

adutra · 2024-07-22T08:11:58Z

Still can't get the integration tests to work. K3s doesn't work properly in rootless.

Yeah I'm not sure running a Kubernetes cluster inside a rootless container is even possible.

snazy · 2024-07-22T08:20:27Z

...ator/src/intTest/java/org/projectnessie/operator/testinfra/K3sContainerLifecycleManager.java

+    // Mitigate eviction issues in CI by setting eviction thresholds for nodefs very low
+    commandParts.add("--kubelet-arg=eviction-hard=nodefs.available<1%,nodefs.inodesFree<1%");
+    // Enable rootless containers
+    commandParts.add("--kubelet-arg=feature-gates=KubeletInUserNamespace=true");


Suggested change

commandParts.add("--kubelet-arg=feature-gates=KubeletInUserNamespace=true");

commandParts.add("--kubelet-arg=feature-gates=KubeletInUserNamespace=true");

commandParts.add("--rootless");

commandParts.add("--snapshotter=fuse-overlayfs");

K3s refuses to start for me:

2024-07-22 10:21:14 time="2024-07-22T08:21:14Z" level=warning msg="Running RootlessKit as the root user is unsupported." 2024-07-22 10:21:14 time="2024-07-22T08:21:14Z" level=warning msg="The host root filesystem is mounted as \"master:34\". Setting child propagation to \"\" is not supported." 2024-07-22 10:21:14 time="2024-07-22T08:21:14Z" level=fatal msg="failed to setup UID/GID map: failed to compute uid/gid map: open /etc/subuid: no such file or directory"

dorsegal · 2024-09-04T10:25:22Z

Do you plan to have GC implemented as part of the operator? I think it can be useful to have CRD for it

adutra · 2024-09-04T11:51:32Z

Do you plan to have GC implemented as part of the operator? I think it can be useful to have CRD for it

See here: #9415

So it's planned yes, but no ETA at the moment due to lack of interest from the community.

adutra force-pushed the kubernetes-operator branch from c528161 to d6d439d Compare January 19, 2024 14:24

snazy reviewed Jan 19, 2024

View reviewed changes

dimas-b reviewed Jan 20, 2024

View reviewed changes

operator/PROJECT Show resolved Hide resolved

operator/src/main/java/org/projectnessie/operator/resource/Nessie.java Outdated Show resolved Hide resolved

operator/src/main/java/org/projectnessie/operator/resource/NessieAwsSecret.java Outdated Show resolved Hide resolved

adutra commented Jan 22, 2024

View reviewed changes

operator/Makefile Outdated Show resolved Hide resolved

adutra commented Jan 22, 2024

View reviewed changes

operator/src/main/java/org/projectnessie/operator/resource/NessieAuthentication.java Outdated Show resolved Hide resolved

adutra force-pushed the kubernetes-operator branch 10 times, most recently from bd00481 to 02e994a Compare January 25, 2024 20:21

adutra marked this pull request as ready for review January 25, 2024 20:24

adutra changed the title ~~WIP: Kubernetes Operator for Nessie~~ Kubernetes Operator for Nessie Jan 25, 2024

snazy reviewed Jan 26, 2024

View reviewed changes

adutra force-pushed the kubernetes-operator branch 6 times, most recently from de7990f to 68051e7 Compare January 30, 2024 17:45

adutra force-pushed the kubernetes-operator branch 5 times, most recently from 4a2d6f7 to 544c574 Compare February 12, 2024 16:56

adutra force-pushed the kubernetes-operator branch 4 times, most recently from 0c8f675 to 6d5185a Compare June 29, 2024 12:53

adutra force-pushed the kubernetes-operator branch 8 times, most recently from 4f472c3 to 50869d6 Compare July 15, 2024 16:18

adutra force-pushed the kubernetes-operator branch from 50869d6 to 3432a15 Compare July 19, 2024 13:28

snazy reviewed Jul 19, 2024

View reviewed changes

...ator/src/intTest/java/org/projectnessie/operator/testinfra/K3sContainerLifecycleManager.java Show resolved Hide resolved

adutra force-pushed the kubernetes-operator branch from 3432a15 to 170f350 Compare July 22, 2024 07:02

snazy reviewed Jul 22, 2024

View reviewed changes

adutra force-pushed the kubernetes-operator branch from 170f350 to b724582 Compare July 22, 2024 07:48

snazy reviewed Jul 22, 2024

View reviewed changes

adutra force-pushed the kubernetes-operator branch from b724582 to 35e7213 Compare July 22, 2024 08:15

snazy reviewed Jul 22, 2024

View reviewed changes

adutra force-pushed the kubernetes-operator branch from 35e7213 to 99edd79 Compare August 27, 2024 09:13

Kubernetes Operator for Nessie

934c2ce

adutra force-pushed the kubernetes-operator branch from 99edd79 to 934c2ce Compare August 27, 2024 09:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes Operator for Nessie #7967

Kubernetes Operator for Nessie #7967

adutra commented Jan 19, 2024

snazy left a comment

dorsegal commented Jul 19, 2024

adutra commented Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 19, 2024

snazy Jul 22, 2024

snazy Jul 22, 2024

snazy Jul 22, 2024

snazy Jul 22, 2024

snazy commented Jul 22, 2024

adutra commented Jul 22, 2024

snazy Jul 22, 2024

adutra Jul 22, 2024

dorsegal commented Sep 4, 2024

adutra commented Sep 4, 2024

	intTestCompileOnly(libs.microprofile.openapi)
	intTestCompileOnly(libs.microprofile.openapi)
	intTestCompileOnly(libs.immutables.value.annotations)

	FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:483.0.0-debian_component_based
	FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:484.0.0-debian_component_based

	FROM rancher/k3s:v1.30.2-k3s2
	FROM docker.io/rancher/k3s:v1.30.2-k3s2

	.withLogConsumer(new Slf4jLogConsumer(logger))
	.withLogConsumer(new Slf4jLogConsumer(logger).withPrefix(container.getDockerImageName()))

-  protected String inDockerIpAddress;
+  private String inDockerIpAddress;
+  protected String inDockerIpAddress() {
+    if (inDockerIpAddress == null) {
+      inDockerIpAddress = getInDockerIpAddress();
+    }
+    return inDockerIpAddress;
+  }

	inDockerIpAddress =
	Objects.requireNonNull(
	getInDockerIpAddress(), "could not determine container's in-docker IP address");

	Logger logger = LoggerFactory.getLogger(getClass());
	inDockerIpAddress = null;
	Logger logger = LoggerFactory.getLogger(getClass());

Kubernetes Operator for Nessie #7967

Are you sure you want to change the base?

Kubernetes Operator for Nessie #7967

Conversation

adutra commented Jan 19, 2024

snazy left a comment

Choose a reason for hiding this comment

dorsegal commented Jul 19, 2024

adutra commented Jul 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snazy commented Jul 22, 2024

adutra commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dorsegal commented Sep 4, 2024

adutra commented Sep 4, 2024