[WIP] Implement support for volume group mirroring #4739

Madhu-1 · 2024-07-29T13:02:24Z

This PR does the required changes so that the volume group can also implement the Mirror interface and we do need to make many changes in the replication code to work differently for the mirroring of an image or a mirroring of a group.

Note:- The last commit to add use go-ceph is commented out, please don't review it, Will remove that code and make changes to the mirror implementation of the group. except that other commits can be reviewed.

Depends-on: ceph/go-ceph#1010

internal/rbd/manager.go

internal/csi-addons/rbd/volumegroup.go

internal/csi-addons/rbd/replication.go

internal/rbd/group/volume_group.go

nixpanic · 2024-07-30T15:33:06Z

internal/rbd/group/volume_group.go

+
+// GroupStatus is a wrapper around librbd.MirrorGroupInfo that contains the
+// group mirror info.
+type GroupStatus struct {


why not just do it like

type GroupStatus *librbd.MirrorGroupInfo

Methods defined on the original type are not available to the new type unless you explicitly defined

if we create a type i need to implement duplication methods of MirrorGroupInfo here which is again not much useful, lets keep it this way which allow us to use methods of librbd struct without any problem

mergify · 2024-07-30T19:43:34Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

Madhu-1 · 2024-08-01T05:27:05Z

@nixpanic This is ready for one more round of review :)

mergify · 2024-08-01T08:31:55Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

nixpanic · 2024-08-01T14:56:48Z

internal/rbd/types/manager.go

+
+	// GetMirrorSource returns the source of the mirror for the given volume or group.
+	GetMirrorSource(ctx context.Context, volumeID string,
+		rep *replication.ReplicationSource) ([]Volume, Mirror, error)


it is not clear to me why []Volume is returned. Should that not be either a single Volume or a VolumeGroup?

would it make sense to have a Mirror.GetSource() function so that the API is easier to understand?

Let's put everything related to processing volumes within functions in mirror type ?

would it make sense to have a Mirror.GetSource() function so that the API is easier to understand?

@nixpanic I put it with the Manager because that is the one who is generating the volume and group from an ID, moving this to Mirroring

it is not clear to me why []Volume is returned. Should that not be either a single Volume or a VolumeGroup?

This is required because we need to work on all the volumes if its a group, i am trying to avoid type casting and doing the ListVolumes and generating the volumes again that contains the connections, let me know if you have any suggestions

Let's put everything related to processing volumes within functions in mirror type ?

I mostly added all the mirroring specific things to mirror type but there are again generic one as well that the reason i left it in volume type as it makes more sense.

👍
We can check on volumes and refactor it later on.
Everything LGTM
Only concern now is ceph's API stability and go-ceph pr going in.

ceph/go-ceph#1010 (comment)
ceph/ceph#53793 (comment)

nixpanic · 2024-08-01T15:01:43Z

internal/rbd/group/volume_group.go

@@ -469,3 +474,296 @@ func (vg *volumeGroup) ListVolumes(ctx context.Context) ([]types.Volume, error)
 func (vg *volumeGroup) ToMirror() (types.Mirror, error) {
 	return vg, nil
 }
+
+func (vg *volumeGroup) EnableMirroring(ctx context.Context, mode librbd.ImageMirrorMode) error {


I wonder if it is worth to simplify things a little by placing this in a separate file group_mirror.go?

type volumeGroupMirror *volumeGroup // or make it a struct with extra members? func (vgm *volumeGroupMirror) EnableMirroring(ctx context.Context, mode librbd.ImageMirrorMode) error { ...

This would keep the VolumeGroup interface smal(ler). I am considering a similar approach for VolumeGroupSnapshot.

nixpanic · 2024-08-01T15:05:52Z

internal/rbd/group/volume_group.go

+		return nil, fmt.Errorf("failed to get volume group mirroring info %q: %w", vg, err)
+	}
+
+	return GroupInfo{MirrorGroupInfo: info}, nil


you could also do

type GroupInfo *librbd.MirrorGroupInfo

and cast it here

return GroupInfo(info)

That looks a little simpler to me.

Yes but again this will have problem like #4739 (comment) and adding new field to GroupInfo need more refractoring, IMO this should be okay but the struct name should not be exported(taken care of it)

nixpanic · 2024-08-01T15:06:14Z

internal/rbd/group/volume_group.go

+}
+
+func (status GlobalMirrorGroupStatus) GetState() string {
+	return status.GlobalMirrorGroupStatus.Info.State.String()


could any of the members be nil?

Not possible as the members are not pointers

nixpanic · 2024-08-01T15:08:11Z

internal/rbd/group/volume_group.go

+
+// GroupInfo is a wrapper around librbd.MirrorGroupInfo that contains the
+// group mirror info.
+type GroupInfo struct {


If you want to use a struct, consider exposing an interface instead of the struct. That encourages a cleaner API towards consumers.

Rakshith-R · 2024-08-02T07:04:48Z

internal/csi-addons/rbd/replication.go

@@ -280,15 +280,12 @@ func (rs *ReplicationServer) EnableVolumeReplication(ctx context.Context,
 	mgr := rbd.NewManager(rs.csiID, req.GetParameters(), req.GetSecrets())
 	defer mgr.Destroy(ctx)

-	rbdVol, err := mgr.GetVolumeByID(ctx, volumeID)
+	volumes, mirror, err := mgr.GetMirrorSource(ctx, volumeID, req.GetReplicationSource())


let's call volumeID something else like reqID or replicationID.
It can then be either referring to a single volume or group (or later on snapshot).

This adds the required functionality to call the go-ceph API's for the rbd volume group. Signed-off-by: Madhu Rajanna <[email protected]>

internal/rbd/manager.go

implementing GetMirrorSource in manager to return volume or the volumegroup based on the replication source, if replication source is nil return the volume details for backward compatibility. Signed-off-by: Madhu Rajanna <[email protected]>

This is temporary test only commit and good suitable for review. Signed-off-by: Madhu Rajanna <[email protected]>

mergify · 2024-08-09T10:07:31Z

The PR description contains the unsupported [skip ci] command, please update the description and mark the PR ready for review again.

mergify · 2024-08-15T13:23:44Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

github-actions · 2024-09-14T21:01:19Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions · 2024-10-16T21:01:45Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

Madhu-1 force-pushed the implement-rbd-vg-mirror branch 7 times, most recently from bd794c1 to 4a5761c Compare July 30, 2024 11:53

nixpanic reviewed Jul 30, 2024

View reviewed changes

Madhu-1 mentioned this pull request Jul 30, 2024

RBD: improvments to existing code to better align with volumegroup #4743

Merged

Madhu-1 force-pushed the implement-rbd-vg-mirror branch 3 times, most recently from 3337a23 to add2d0b Compare August 1, 2024 05:00

Madhu-1 requested review from nixpanic and Rakshith-R August 1, 2024 05:26

Madhu-1 force-pushed the implement-rbd-vg-mirror branch from add2d0b to 3808a65 Compare August 1, 2024 09:55

nixpanic reviewed Aug 1, 2024

View reviewed changes

Rakshith-R reviewed Aug 2, 2024

View reviewed changes

Nikhil-Ladha mentioned this pull request Aug 2, 2024

Add volume group replication controller code csi-addons/kubernetes-csi-addons#610

Open

rbd: implement volume group using go-ceph

5ddcbfb

This adds the required functionality to call the go-ceph API's for the rbd volume group. Signed-off-by: Madhu Rajanna <[email protected]>

Madhu-1 force-pushed the implement-rbd-vg-mirror branch from 3808a65 to 20f0600 Compare August 6, 2024 08:47

Rakshith-R reviewed Aug 6, 2024

View reviewed changes

internal/rbd/manager.go Outdated Show resolved Hide resolved

Madhu-1 force-pushed the implement-rbd-vg-mirror branch from 20f0600 to d0c541f Compare August 6, 2024 10:12

Madhu-1 added 2 commits August 6, 2024 12:14

rbd: implement GetMirrorSource in manager

b4545b7

implementing GetMirrorSource in manager to return volume or the volumegroup based on the replication source, if replication source is nil return the volume details for backward compatibility. Signed-off-by: Madhu Rajanna <[email protected]>

ci: remove this commit later

ea1f176

This is temporary test only commit and good suitable for review. Signed-off-by: Madhu Rajanna <[email protected]>

Madhu-1 force-pushed the implement-rbd-vg-mirror branch from d0c541f to ea1f176 Compare August 6, 2024 10:14

github-actions bot added the stale label Sep 14, 2024

Madhu-1 removed the stale label Sep 16, 2024

github-actions bot added the stale label Oct 16, 2024

Madhu-1 added keepalive This label can be used to disable stale bot activiity in the repo and removed stale labels Oct 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Implement support for volume group mirroring #4739

[WIP] Implement support for volume group mirroring #4739

Madhu-1 commented Jul 29, 2024 •

edited by nixpanic

Loading

nixpanic Jul 30, 2024

Madhu-1 Jul 30, 2024

mergify bot commented Jul 30, 2024

Madhu-1 commented Aug 1, 2024

mergify bot commented Aug 1, 2024

nixpanic Aug 1, 2024

nixpanic Aug 1, 2024

Rakshith-R Aug 2, 2024

Madhu-1 Aug 6, 2024

Rakshith-R Aug 6, 2024

nixpanic Aug 1, 2024

nixpanic Aug 1, 2024

Madhu-1 Aug 6, 2024

nixpanic Aug 1, 2024

Madhu-1 Aug 6, 2024

nixpanic Aug 1, 2024

Rakshith-R Aug 2, 2024

mergify bot commented Aug 9, 2024

mergify bot commented Aug 15, 2024

github-actions bot commented Sep 14, 2024

github-actions bot commented Oct 16, 2024

[WIP] Implement support for volume group mirroring #4739

Are you sure you want to change the base?

[WIP] Implement support for volume group mirroring #4739

Conversation

Madhu-1 commented Jul 29, 2024 • edited by nixpanic Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Jul 30, 2024

Madhu-1 commented Aug 1, 2024

mergify bot commented Aug 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Aug 9, 2024

mergify bot commented Aug 15, 2024

github-actions bot commented Sep 14, 2024

github-actions bot commented Oct 16, 2024

Madhu-1 commented Jul 29, 2024 •

edited by nixpanic

Loading