Do not use beta API for hyperdisk in multi-writer mode.

[karkunpavan]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Hyper disk multi-writer support is now in compute/v1 but the persistent disk csi driver still calls v0.beta which leads a error while creating a hyperdisk with multi-writer mode. Error:

Warning ProvisioningFailed 31m (x15 over 57m) pd.csi.storage.gke.io_wduan-1030a-g-vdnqh-master-0_dc46e9f6-1725-4f15-922c-99f0e281e102 failed to provision volume with StorageClass
"balanced-storage": rpc error: code = InvalidArgument desc = CreateVolume failed: rpc error: code = InvalidArgument desc = CreateVolume failed to create single zonal disk pvc-0b65d680-636b-46c6-876d-d3a6c412c3ef: failed to insert zonal disk: unknown Insert disk error:
googleapi: Error 400: Invalid value for field 'resource.multiWriter': 'true'.
Cannot specify the multi writer field for 'hyperdisk-balanced' disks,
please use access mode instead., invalid%!v(MISSING)

To fix this we will call the v1 API which accepts accessMode = ReadWriteMany.

Which issue(s) this PR fixes:

Fixes #1863

Special notes for your reviewer:

• Discussed this issue and the fix with Matthew
• Making this change only for disk types hyperdisk*, multi-writer support for other disk types is not the focus of this fix.

Does this PR introduce a user-facing change?:

NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: k8s-ci-robot / name: Kubernetes Prow Robot (d8c777e, c29a4ee, bc9302c, 486a336, f872f15, 13bf655, e85e9f8, 5675f13, a06d250, 2b5cadc, e457c73)
• ✅ login: pwschuurman (0862fd6, a1ed4e4, 50f8af2)
• ✅ login: saikat-royc / name: Saikat Roychowdhury (524895b, 96338ca, 2e9c71c)
• ✅ login: karkunpavan / name: Pavan Karkun (35d3d55, 4168542, 37e19fd, 8081b51, ce02f42)
• ✅ login: travisyx / name: Travis Xiang (51fb1d0, 1a7eeeb, 6039d2b)
• ✅ login: Fricounet / name: Baptiste Girard-Carrabin (7407af0)
• ✅ login: mattcary / name: Matt Cary (a28f8d3, 0f74882, 4249ee3)
• ✅ login: amacaskill / name: Alexis MacAskill (4f2fd63)
• ✅ login: Sneha-at / name: Sneha-at (6449ad1)

[k8s-ci-robot]

Welcome @karkunpavan!

It looks like this is your first PR to kubernetes-sigs/gcp-compute-persistent-disk-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gcp-compute-persistent-disk-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @karkunpavan. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mattcary]

/ok-to-test

[karkunpavan]

Detailed code changes as requested. Updated the PR with all of these code changes.

1. In insertZonalDisk() PD CSI driver currently calls beta APIs when multi-writer is set. We need to add a switch based on disk type. For hyperdisk use v1 API and for persistent disks use beta APIs.

2. Similar changes needed in insertRegionalDisk()

3. AccessMode flag is already set here and no changes needed to explicitly set it for hyperdisks

4. GetMultiWriter() currently thinks that v1 APIs do not support multi-writer and this needs to be fixed. We should return a True when AccessMode == READ_WRITE_MANY

5. Remove comments about hyperdisk not supporting multi-writer

[karkunpavan]

/retest-required

The failures are SSH timeouts which seem to be unrelated to my changes.

[karkunpavan]

/retest - seems like a flaky test which timed out.

[k8s-ci-robot]

@karkunpavan: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test pull-gcp-compute-persistent-disk-csi-driver-e2e
• /test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration
• /test pull-gcp-compute-persistent-disk-csi-driver-sanity
• /test pull-gcp-compute-persistent-disk-csi-driver-unit
• /test pull-gcp-compute-persistent-disk-csi-driver-verify

The following commands are available to trigger optional jobs:

• /test pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2019

Use /test all to run the following jobs that were automatically triggered:

• pull-gcp-compute-persistent-disk-csi-driver-e2e
• pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration
• pull-gcp-compute-persistent-disk-csi-driver-sanity
• pull-gcp-compute-persistent-disk-csi-driver-unit
• pull-gcp-compute-persistent-disk-csi-driver-verify

In response to this:

/retest - seems like a flaky test which timed out.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[karkunpavan]

/retest

[gtomitsuka]

@mattcary @leiyiz any potential follow-up on why the test could be failing?

this is pretty important to my company, would let us move from standard-rwx with 1TB min capacity to something much more fine-grained. happy to support with anything

[mattcary]

@mattcary @leiyiz any potential follow-up on why the test could be failing?

this is pretty important to my company, would let us move from standard-rwx with 1TB min capacity to something much more fine-grained. happy to support with anything

This is not a replacement for standard-rwx. It enables multiwriter on a block device, but this does not mean you have a multiwriter filesystem. A distributed filesystem is hard to make, and multiwriter devices is only one piece of the puzzle. ext4 and xfs are not distributed.

[gtomitsuka]

@mattcary I understand, my case specifically is that we have Performance pods (= one pod per node) running ML models on GKE, and we'd like to enable rolling updates for it while keeping the models persistent to reduce overhead.

Right now, this rules us out from either doing rolling updates or using hyperdisk, due to a "true" distributed FS requirement, which has very high overhead. Making it read-only would prohibit us from using some Python libraries we use like stanza.

Our understanding is that this is the k8s equivalent to using a cannon to catch a fish and in practice a much simpler solution would suffice, since writes are extremely rare and no more than one pod would be writing at any given time (we can also guarantee this using external locks if needed).

Am I understanding this incorrectly?

[mattcary]

@gtomitsuka The kernel filesystem modules try very hard to do things like cache in memory. So even if writes are rare, the in-memory structures on different machines are going to be out of date and desynchronized.

It seems possible to have some ROX with a single writer, which unmounts all readers, updates the single writer, and then remounts, while keeping all the disks attached. But that will require a new csi driver anyway.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: karkunpavan
Once this PR has been reviewed and has the lgtm label, please assign mattcary for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[karkunpavan]

Closing this pull request, will raise a new one with planned changes

[k8s-ci-robot]

@karkunpavan: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gcp-compute-persistent-disk-csi-driver-unit	ce02f42	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-unit
pull-gcp-compute-persistent-disk-csi-driver-e2e	ce02f42	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-e2e
pull-gcp-compute-persistent-disk-csi-driver-verify	ce02f42	link	true	/test pull-gcp-compute-persistent-disk-csi-driver-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.