Add creation of kueue resources in e2e tests

[ChughShilpa]

Issue link

RHOAIENG-11047

What changes have been made

Since we are adjusting e2e tests to run with AMD gpu, thus

• Adding creation of kueue resources as part of test to make it compatible with gpuresources (which is covered in next PR)
• Removing creation of kueue resources from setup script
• Adjust methods with updated Ray cluster client

Verification steps

Run test manually
go test -timeout 30m -v ./test/e2e -run TestMnistPyTorchAppWrapperCpu

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign christianzaccaria for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[astefanutti]

Should it be removed if numberOfGpus is 0?

[ChughShilpa]

IIRC specifying the resource name is mandatory even if nominalquota is 0

[ChughShilpa]

https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2-latest/html/working_with_distributed_workloads/configuring-distributed-workloads_distributed-workloads#configuring-quota-management-for-distributed-workloads_distributed-workloads:~:text=Specify%20the%20resource%20name%20and%20nominalQuota%20in%20the%20flavors.resources%20section%2C%20even%20if%20the%20nominalQuota%20value%20is%200.

[sutaakar]

The cleanest solution could be to move creation of ResourceFlavor and ClusterQueue (in the end also RayCluster) into Test<something>Gpu/Cpu functions. Each test function would then set properties relevant just to the test function.
That way we would get rid of conditions in test, reducing its reading complexity. Though that would mean a large rewrite.
I am ok with current solution too, just thinking out loud.

[sutaakar]

IIRC specifying the resource name is mandatory even if nominalquota is 0

I think it is (was?) mandatory in RHOAI due to SDK passing GPU values. Not sure if it is still needed.
Anyway in real use cases for finetuning users need GPU.

[astefanutti]

Right, I'm thinking the same, it's an issue at the SDK level right? It should not request 0 GPU?

[ChughShilpa]

Created a jira issue for SDK to address this - https://issues.redhat.com/browse/RHOAIENG-13618
@astefanutti @sutaakar

[sutaakar]

/lgtm