VMDistributedCluster CR: orchestrate VMCluster upgrades #1556

vrutkovs · 2025-10-21T07:27:53Z

Add a new CR - VMDistributedCluster -so that multiple VMClusters could be upgraded in orchestrated fashion, ensuring the read VMAuth is disabled before upgrade and VMAgent (if available) doesn't have pending bytes to send.

See #1515 (comment) for agreed limitations for v1alpha1 version.

Fixes #1515

TODO:

Add changelog entry
Description-less CRD should be applied for development only. Rephrase descriptions in existing parts to make it fit for production
Fix flaking tests
Squash commits
Keeping original commits for review as its useful to show how the feature was developed
Allow objects from other namespaces?
During development I realized VMAuth / VMClusters may be in different namespaces. The initial version requires all objects to be in the same namespace as VMDistributedCluster. Do we want to keep it that way for API simplicity or worth having the initial version with cross-namespace support?
Watch VMClusters and run CreateOrUpdate to apply spec overrides accordingly
Set ownerRefs to managed VMClusters
Other improvements:
- Multiple VMAgents per VMCluster - use label selector instead of name?
- Label selector for VMClusters? Not sure how to select VMAgents then

config/rbac/role.yaml

config/rbac/operator_vmdistributedcluster_editor_role.yaml

AndrewChubatiuk · 2025-10-21T07:49:28Z

initially thought distributed CR is needed for full distributed setup management, but looks like it only performs version upgrade. In this case just curious why we need different CRs for VM, VT and VL?

vrutkovs · 2025-10-21T09:05:09Z

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

In this case just curious why we need different CRs for VM, VT and VL?

VL and VT don't have agents (yet) so their specs would be different. However we can reuse the same approach and probably even some helper functions

Haleygo

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.
And since vmclusterSpec.ClusterVersion is optional, users could specify component versions inside vmclusterSpec which overrides the vmclusterSpec.ClusterVersion.

And currently, it seems VMDistributedCluster only covers a limited scenario where resources like vmcluster, vmuser, vmauth are defined and configured as needed.
Could you please provide an example of how to config them to achieve similar topology described in victoria-metrics-distributed chart? I expect VMDistributedCluster to be supported there when released.

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go

vrutkovs · 2025-10-24T06:09:09Z

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.

Yup, setting generic overrideParams would be more flexible and, along with upgrades, would cover other maintenance tasks, i.e., adding replicas or setting flags

ginkgo1.txt

AndrewChubatiuk · 2025-11-03T12:53:44Z

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go

+		}
+	}
+
+	// Fetch global vmagent


are we going to have a single VMAgent? are there any requirements to amount of replicas it should have according to zones count?

Single VMAgent iiuc, however, later we may want to support zone-specific agents

api/operator/v1alpha1/vmdistributedcluster_types.go

Haleygo · 2025-11-04T13:54:55Z

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go

+
+	// Loop until VMAgent metrics show that disk buffer is empty
+	err := wait.PollUntilContextTimeout(ctx, 1*time.Second, deadline, true, func(ctx context.Context) (done bool, err error) {
+		metricValue, err := fetchVMAgentDiskBufferMetric(ctx, httpClient, vmAgentPath)


The vmAgentPath uses vmagent service address, what if the vmagent deployment has multiple replicas?

Is it a common configuration? I don't see the value in running several identical replicas of vmagent, usually (redundancy?).

TODO: fetch endpoints to be sure

I don't see the value in running several identical replicas of vmagent, usually (redundancy?).

It's common for large setups, for HA or increase cluster capacity.

Haleygo · 2025-11-04T13:58:06Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+type VMDistributedClusterSpec struct {
+	// ParsingError contents error with context if operator was failed to parse json object from kubernetes api server
+	ParsingError string `json:"-" yaml:"-"`
+	// VMAgent points to the VMAgent object for collecting metrics from multiple VMClusters


The main role of this vmagent is the global write entrypoint(which buffers data if one of the vmcluster is down). It might not be used to scrape targets(including vmclusters), but only to proxy data.
What about renaming it to something like GlobalWriteEntryVMAgent?

I'd prefer to keep the name short - there is no need to write a long one as there is no other VMAgent to confuse this setting with.

I agree however this should have a better description.

Haleygo · 2025-11-04T14:02:35Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+	// VMUsers is a list of VMUser objects controlling traffic distribution between multiple VMClusters
+	VMUsers []corev1.LocalObjectReference `json:"vmUsers,omitempty"`
+	// Zones is a list of VMCluster instances to update. Each VMCluster in the list represents a "zone" within the distributed cluster.
+	Zones []VMClusterRefOrSpec `json:"zones,omitempty"`


zones is ambiguous without contexts, and since it only contains vmcluster resources, I think we can call it something like vmclusterList to be more clear.

Haleygo · 2025-11-04T14:06:24Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.updateStatus",description="current status of update rollout"
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
+// VMDistributedClusterSpec is progressively rolling out updates to multiple VMClusters.
+type VMDistributedCluster struct {


The feature isn’t limited to vmcluster, it can also be applied to vmsingle.
Later, we can support vmsingle with the same CRD by replacing vmcluster objects under VMDistributedClusterSpec with vmsingle.
What about calling it VMDistributed?

Yes, I think it's fair to extend this to VMSingle too. I think it would be trivial to extend this to VMSingles - by adding VMType to Zones property. We didn't get a request to extend this to VMSingle though, so I'd prefer to focus on VMClusters.

Not quite sure about VMDistributed - its fair to treat multiple zones as a cluster (so VMCluster would be a fantastic name :) ), while VMDistributed doesn't really pinpoint it.

Haleygo · 2025-11-04T14:14:10Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+	// VMAgent points to the VMAgent object for collecting metrics from multiple VMClusters
+	VMAgent corev1.LocalObjectReference `json:"vmAgent,omitempty"`
+	// VMUsers is a list of VMUser objects controlling traffic distribution between multiple VMClusters
+	VMUsers []corev1.LocalObjectReference `json:"vmUsers,omitempty"`


Please name the field more specifically.
As I understand, we don't need VMUser objects controlling traffic distribution between multiple VMClusters, but only the vmuser which proxies query requests between two vmcluster.
For example, the vmuser for vmauth-read-proxy in below diagram(without the vmauth-read-balancer since it changes the target for vmauth-read-proxy).

Later, We should also support vmauth's unauthorized_user.

Agree on documentation update, but it would be convenient to keep the property name short (there are no other VMUsers specified there too)

Haleygo · 2025-11-04T14:34:41Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+	// Name specifies the static name to be used for the VMCluster when Spec is provided.
+	// If Ref is used, this field is ignored.
+	// +optional
+	Name string `json:"name,omitempty"`
+	// Ref points to the VMCluster object.
+	// +optional
+	Ref *corev1.LocalObjectReference `json:"ref,omitempty"`
+	// Spec defines the desired state of a new VMCluster.
+	// +optional
+	Spec *vmv1beta1.VMClusterSpec `json:"spec,omitempty"`
+	// OverrideSpec specifies an override to the VMClusterSpec of the referenced object.
+	// This field is ignored if `Spec` is specified.
+	// This override is applied to the referenced object if `Ref` is specified.
+	// +kubebuilder:validation:Type=object
+	// +kubebuilder:validation:XPreserveUnknownFields
+	// +optional
+	OverrideSpec *apiextensionsv1.JSON `json:"overrideSpec,omitempty"`


So Name and Spec define a new vmcluster, while Ref and OverrideSpec go to an existing one. Please put related fields closer for readability:

// Name creates a new VMCluster with the below spec. // Only one of Ref or Name can be used. // +optional Name string `json:"name,omitempty"` // Spec defines the spec with a new vmcluster. // +optional Spec *vmv1beta1.VMClusterSpec `json:"spec,omitempty"` // Ref points to an existing VMCluster object. // Only one of Ref or Name can be used. // +optional Ref *corev1.LocalObjectReference `json:"ref,omitempty"` // OverrideSpec specifies spec overrides for the existing vmcluster. // +optional OverrideSpec *apiextensionsv1.JSON `json:"overrideSpec,omitempty"`

Also the comments are conflicted, it says:

// If Ref is used, this field is ignored.
Name string json:"name,omitempty"

which means Ref takes precedence. But

// This field is ignored if Spec is specified.
OverrideSpec *apiextensionsv1.JSON json:"overrideSpec,omitempty"

means Spec(Name) takes precedence.
Since we don't allow to have both Name and Ref, let's just highlight this in the doc.

Please put related fields closer for readability

Good idea. Rearranged and added better description in 95bc654

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go

Haleygo · 2025-11-04T15:02:37Z

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go

+		if !needsToBeCreated && !modified {
+			continue
+		}


If the vmcluster was created by VMDistributedCluster, then user changed its spec, the change won't be detected by the check, and the vmcluster won't be updated.

Good catch, we'll need to set up an informer for VMClusters in this controller - do you know if there is some nice example on how to do that?

Haleygo · 2025-11-04T15:19:20Z

api/operator/v1alpha1/vmdistributedcluster_types.go

+}
+
+// +k8s:openapi-gen=true
+// VMClusterRefOrSpec is either a reference to existing VMCluster or a specification of a new VMCluster.


It would be nice to tag the newly created vmcluster as managed by VMDistributed, and not allowed to modify the spec in vmcluster reconcile.

If we setup watchers for VMClusters the users are allowed to modify them (although results may be unexpected).

It is worth setting ownerRef for affected clusters, I agree

… uses server-side apply This prevents kubectl from creating an annotation which is too large and failing with `The CustomResourceDefinition "vmdistributedclusters.operator.victoriametrics.com" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes` error

…" e2e test

…are set

vrutkovs requested review from AndrewChubatiuk, Haleygo and f41gh7 as code owners October 21, 2025 07:27

f41gh7 self-assigned this Oct 21, 2025

AndrewChubatiuk reviewed Oct 21, 2025

View reviewed changes

config/rbac/role.yaml Show resolved Hide resolved

AndrewChubatiuk reviewed Oct 21, 2025

View reviewed changes

config/rbac/operator_vmdistributedcluster_editor_role.yaml Outdated Show resolved Hide resolved

vrutkovs force-pushed the vmdistributed-cluster branch from eaeacd4 to c02b24c Compare October 21, 2025 09:01

Haleygo reviewed Oct 23, 2025

View reviewed changes

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go Outdated Show resolved Hide resolved

internal/controller/operator/factory/vmdistributedcluster/vmdistributedcluster.go Show resolved Hide resolved

vrutkovs force-pushed the vmdistributed-cluster branch 2 times, most recently from 280b2e6 to 04b44f9 Compare October 30, 2025 08:56

vrutkovs requested review from AndrewChubatiuk and Haleygo October 30, 2025 14:49

vrutkovs force-pushed the vmdistributed-cluster branch from 04b44f9 to 1336f73 Compare November 3, 2025 12:43

AndrewChubatiuk reviewed Nov 3, 2025

View reviewed changes

ginkgo1.txt Outdated Show resolved Hide resolved

vrutkovs force-pushed the vmdistributed-cluster branch from 1336f73 to c3b3e24 Compare November 3, 2025 12:49

AndrewChubatiuk reviewed Nov 3, 2025

View reviewed changes

Haleygo reviewed Nov 4, 2025

View reviewed changes

vrutkovs added 9 commits November 5, 2025 08:51

VMDistributedCluster: initial commit

70af829

Controller update

6959c69

Generate v1alpha1 types

b42afae

.golangci.yml: add alias for v1alpha1

87bedc3

Regenerate manifests

c781223

Register v1alpha1 type in manager and e2e tests

e19fc52

Initial operator implementation

185739d

Add tests for vmdistributedcluster controller

0a4ffe8

vrutkovs added 26 commits November 5, 2025 08:51

Use vmagent which doesn't take in-memory data into account

2930ccd

test fixes

8f55964

Fix "should handle rolling updates with VMAgent configuration changes…

b460a79

…" e2e test

More e2e test fixes

33f757d

Add unit tests for TestGetReferencedVMCluster

580c5af

Add unit tests for reconcileInlineVMCluster

9fc3d8d

Add tests for findVMUserReadRuleForVMCluster

ed726a2

Add unit test for waitForVMClusterReady

c673afe

Fix some e2e tests

0da3c18

Add reconciliation e2e

959c7a0

test fixes

59e6bed

Fixes

1a34c1f

Update vmdistributedcluster_test.go

6275cee

fix paused reconciliation test

49874de

Fixes

63c659d

Fix updateVMUserTargetRefs

95330fd

More test fixes

19af1ed

Fix waiting test

c5027f9

gci

4909c34

fetchVMClusters: rewrite as switch

39bc0b5

Fix lint errors

6344323

tests: use real statuses for mockClientWithPollingResponse

ea20c88

Makefile: remove descriptions from CRDs

25b3c1b

Add Changelog entry

e67f285

vmAgent / vmUsers: use lowercased names

e2240eb

Rearrange fields in VMClusterRefOrSpec for clarity

95bc654

vrutkovs force-pushed the vmdistributed-cluster branch from c3b3e24 to 95bc654 Compare November 5, 2025 08:17

vrutkovs added 2 commits November 5, 2025 09:53

validateVMClusterRefOrSpec: fix error message when both ref and spec …

129d0b6

…are set

Set ownerRef to managed VMClusters

42b277d

vrutkovs force-pushed the vmdistributed-cluster branch from 9919c2b to 42b277d Compare November 5, 2025 11:28

VMDistributedCluster CR: orchestrate VMCluster upgrades #1556

Are you sure you want to change the base?

VMDistributedCluster CR: orchestrate VMCluster upgrades #1556

Conversation

vrutkovs commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndrewChubatiuk commented Oct 21, 2025

Uh oh!

vrutkovs commented Oct 21, 2025

Uh oh!

Haleygo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vrutkovs commented Oct 24, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vrutkovs Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vrutkovs commented Oct 21, 2025 •

edited

Loading

vrutkovs Nov 3, 2025 •

edited

Loading