Convert MNIST PyTorch and MNIST Ray test cases to go

[sutaakar]

Converting

distributed-workloads/tests/basictests/distributed-workloads.sh

Lines 33 to 90 in e9ce413

	function test_mcad_torchx_functionality() {
	header "Testing MCAD TorchX Functionality"
	########### Clean Cluster should be free of these resources ############
	# Get appwrapper name
	AW=$(oc get appwrapper -n ${ODHPROJECT} | grep mnistjob | cut -d ' ' -f 1) || true
	# Clean up resources
	if [[ -n $AW ]]; then
	os::cmd::expect_success "oc delete appwrapper $AW -n ${ODHPROJECT} || true"
	fi
	os::cmd::expect_success "oc delete notebook jupyter-nb-kube-3aadmin -n ${ODHPROJECT} || true"
	os::cmd::expect_success "oc delete cm notebooks-mcad -n ${ODHPROJECT} || true"
	os::cmd::expect_success "oc delete pvc jupyterhub-nb-kube-3aadmin-pvc -n ${ODHPROJECT} || true"
	##############################################################################
	# Wait for the notebook controller ready
	os::cmd::try_until_text "oc get deployment odh-notebook-controller-manager -n ${ODHPROJECT} --no-headers=true | awk '{print \$2}'" "1/1" $odhdefaulttimeout $odhdefaultinterval
	# Create a mnist_ray_mini.ipynb as a configMap
	os::cmd::expect_success "oc create configmap notebooks-mcad -n ${ODHPROJECT} --from-file=${RESOURCEDIR}/mnist_mcad_mini.ipynb"
	# Get Token
	local TESTUSER_BEARER_TOKEN="$(curl -skiL -u $TEST_USER:$TEST_PASS -H 'X-CSRF-Token: xxx' "$OPENSHIFT_OAUTH_ENDPOINT/oauth/authorize?response_type=token&client_id=openshift-challenging-client" | grep -oE 'access_token=[^&]*'| sed 's/access_token=//')"
	os::cmd::expect_success "cat ${RESOURCEDIR}/custom-nb-small.yaml \
	| sed s/%INGRESS%/$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})/g \
	| sed s/%OCPSERVER%/$(oc whoami --show-server=true|cut -f3 -d "/")/g \
	| sed s/%OCPTOKEN%/${TESTUSER_BEARER_TOKEN}/g \
	| sed s/%NAMESPACE%/${ODHPROJECT}/g \
	| sed s/%JOBTYPE%/mcad/g | oc apply -n ${ODHPROJECT} -f -"
	# Wait for the notebook-server to be ready
	os::cmd::try_until_text "oc get pod -n ${ODHPROJECT} | grep "jupyter-nb-kube-3aadmin" | awk '{print \$2}'" "2/2" $odhdefaulttimeout $odhdefaultinterval
	# Wait for appwrapper to exist
	os::cmd::try_until_text "oc get appwrapper -n ${ODHPROJECT} | grep mnistjob" "mnistjob-*" $odhdefaulttimeout $odhdefaultinterval
	# Get appwrapper name
	AW=$(oc get appwrapper -n ${ODHPROJECT} | grep mnistjob | cut -d ' ' -f 1)
	# Wait for the mnisttest appwrapper state to become running
	os::cmd::try_until_text "oc get appwrapper $AW -n ${ODHPROJECT} -ojsonpath='{.status.state}'" "Running" $odhdefaulttimeout $odhdefaultinterval
	# Wait for workload to succeed and clean up
	os::cmd::try_until_text "oc get appwrapper $AW -n ${ODHPROJECT}" ".*NotFound.*" $odhdefaulttimeout $odhdefaultinterval
	# Test clean up resources
	os::cmd::expect_success "oc delete notebook jupyter-nb-kube-3aadmin -n ${ODHPROJECT}"
	os::cmd::expect_failure "oc get notebook jupyter-nb-kube-3aadmin -n ${ODHPROJECT}"
	os::cmd::expect_success "oc delete cm notebooks-mcad -n ${ODHPROJECT} || true"
	os::cmd::expect_failure "oc get cm notebooks-mcad -n ${ODHPROJECT}"
	os::cmd::expect_success "oc delete appwrapper $AW -n ${ODHPROJECT} || true"
	os::cmd::expect_failure "oc get appwrapper $AW -n ${ODHPROJECT}"
	os::cmd::expect_success "oc delete pvc jupyterhub-nb-kube-3aadmin-pvc -n ${ODHPROJECT} || true"
	os::cmd::expect_failure "oc get pvc jupyterhub-nb-kube-3aadmin-pvc -n ${ODHPROJECT}"
	}

into go lang integration tests.

Description

As part of #63 this PR converts MNIST PyTorch test case into go lang.

How Has This Been Tested?

Tested by executing locally against OCP.

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[anishasthana]

This doesn't seem right

[anishasthana]

Ah, i see the pr is marked wip. Sorry!

[sutaakar]

@astefanutti adjusted PR:

• added a test for MCAD Ray
• replaced oc CLI commands with dynamic client
• minor adjustments here and there

TODO:

• change AppWrapper group - waiting for a new SDK release

There may be potential issue with MCAD Ray test for OpenShift CI as Ray head requires a lot of resources. Maybe there is a need for another node (or for a possibility to modify head node requirements). Will see once TODO is addressed.

[astefanutti]

Awesome. I've left a couple of small comments. Otherwise I'll LTGM it once ready.

[sutaakar]

Changes applied.

[astefanutti]

/lgtm

[astefanutti]

/lgtm

[sutaakar]

Failing because the ODH core KfDef/DSC is not installed in OpenShift CI.
Should be fixed either by openshift/release#43247 or once I migrate the ODH DW to use DSC.

[sutaakar]

/retest

[sutaakar]

/retest

[sutaakar]

/retest

[sutaakar]

/retest

[sutaakar]

/retest

[sutaakar]

TestMnistPyTorchMCAD passed, TestMCADRay failed, supposedly on lack of resources to spin up the Ray head node

[sutaakar]

Skipping unstable/currently failing tests.

[sutaakar]

/retest

[sutaakar]

@astefanutti Can you please take a final look/provide approval?

[astefanutti]

/lgtm

[astefanutti]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sutaakar]

/retest