examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HyperParameter tunning using  CMA-ES\n",
    "\n",
    "In this example you will deploy 3 Katib Experiments with Covariance Matrix Adaptation Evolution Strategy (CMA-ES) using Jupyter Notebook and Katib SDK. These Experiments have various resume policies.\n",
    "\n",
    "Reference documentation:\n",
    "- https://www.kubeflow.org/docs/components/katib/experiment/#cmaes\n",
    "- https://www.kubeflow.org/docs/components/katib/resume-experiment/\n",
    "\n",
    "The notebook shows how to create, get, check status and delete an Experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Katib SDK\n",
    "\n",
    "You need to install Katib SDK to run this Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO (andreyvelich): Change to release version when SDK with the new APIs is published.\n",
    "!pip install git+https://github.com/kubeflow/katib.git#subdirectory=sdk/python/v1beta1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "\n",
    "from kubeflow.katib import KatibClient\n",
    "from kubernetes.client import V1ObjectMeta\n",
    "from kubeflow.katib import V1beta1Experiment\n",
    "from kubeflow.katib import V1beta1AlgorithmSpec\n",
    "from kubeflow.katib import V1beta1ObjectiveSpec\n",
    "from kubeflow.katib import V1beta1FeasibleSpace\n",
    "from kubeflow.katib import V1beta1ExperimentSpec\n",
    "from kubeflow.katib import V1beta1ObjectiveSpec\n",
    "from kubeflow.katib import V1beta1ParameterSpec\n",
    "from kubeflow.katib import V1beta1TrialTemplate\n",
    "from kubeflow.katib import V1beta1TrialParameterSpec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define your Experiment\n",
    "\n",
    "You have to create your Experiment object before deploying it. This Experiment is similar to [this](https://github.com/kubeflow/katib/blob/master/examples/v1beta1/hp-tuning/cma-es.yaml) example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment name and namespace.\n",
    "namespace = \"kubeflow-user-example-com\"\n",
    "experiment_name = \"cmaes-example\"\n",
    "\n",
    "metadata = V1ObjectMeta(\n",
    "    name=experiment_name,\n",
    "    namespace=namespace\n",
    ")\n",
    "\n",
    "# Algorithm specification.\n",
    "algorithm_spec=V1beta1AlgorithmSpec(\n",
    "    algorithm_name=\"cmaes\"\n",
    ")\n",
    "\n",
    "# Objective specification.\n",
    "objective_spec=V1beta1ObjectiveSpec(\n",
    "    type=\"maximize\",\n",
    "    goal= 0.99,\n",
    "    objective_metric_name=\"Validation-accuracy\",\n",
    "    additional_metric_names=[\"Train-accuracy\"]\n",
    ")\n",
    "\n",
    "# Experiment search space. In this example we tune learning rate, number of layer and optimizer.\n",
    "parameters=[\n",
    "    V1beta1ParameterSpec(\n",
    "        name=\"lr\",\n",
    "        parameter_type=\"double\",\n",
    "        feasible_space=V1beta1FeasibleSpace(\n",
    "            min=\"0.01\",\n",
    "            max=\"0.06\"\n",
    "        ),\n",
    "    ),\n",
    "    V1beta1ParameterSpec(\n",
    "        name=\"num-layers\",\n",
    "        parameter_type=\"int\",\n",
    "        feasible_space=V1beta1FeasibleSpace(\n",
    "            min=\"2\",\n",
    "            max=\"5\"\n",
    "        ),\n",
    "    ),\n",
    "    V1beta1ParameterSpec(\n",
    "        name=\"optimizer\",\n",
    "        parameter_type=\"categorical\",\n",
    "        feasible_space=V1beta1FeasibleSpace(\n",
    "            list=[\"sgd\", \"adam\", \"ftrl\"]\n",
    "        ),\n",
    "    ),\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "# JSON template specification for the Trial's Worker Kubernetes Job.\n",
    "trial_spec={\n",
    "    \"apiVersion\": \"batch/v1\",\n",
    "    \"kind\": \"Job\",\n",
    "    \"spec\": {\n",
    "        \"template\": {\n",
    "            \"metadata\": {\n",
    "                \"annotations\": {\n",
    "                    \"sidecar.istio.io/inject\": \"false\"\n",
    "                }\n",
    "            },\n",
    "            \"spec\": {\n",
    "                \"containers\": [\n",
    "                    {\n",
    "                        \"name\": \"training-container\",\n",
    "                        \"image\": \"docker.io/kubeflowkatib/mxnet-mnist:v0.14.0\",\n",
    "                        \"command\": [\n",
    "                            \"python3\",\n",
    "                            \"/opt/mxnet-mnist/mnist.py\",\n",
    "                            \"--batch-size=64\",\n",
    "                            \"--num-epochs=1\",\n",
    "                            \"--lr=${trialParameters.learningRate}\",\n",
    "                            \"--num-layers=${trialParameters.numberLayers}\",\n",
    "                            \"--optimizer=${trialParameters.optimizer}\"\n",
    "                        ]\n",
    "                    }\n",
    "                ],\n",
    "                \"restartPolicy\": \"Never\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "# Configure parameters for the Trial template.\n",
    "trial_template=V1beta1TrialTemplate(\n",
    "    primary_container_name=\"training-container\",\n",
    "    trial_parameters=[\n",
    "        V1beta1TrialParameterSpec(\n",
    "            name=\"learningRate\",\n",
    "            description=\"Learning rate for the training model\",\n",
    "            reference=\"lr\"\n",
    "        ),\n",
    "        V1beta1TrialParameterSpec(\n",
    "            name=\"numberLayers\",\n",
    "            description=\"Number of training model layers\",\n",
    "            reference=\"num-layers\"\n",
    "        ),\n",
    "        V1beta1TrialParameterSpec(\n",
    "            name=\"optimizer\",\n",
    "            description=\"Training model optimizer (sdg, adam or ftrl)\",\n",
    "            reference=\"optimizer\"\n",
    "        ),\n",
    "    ],\n",
    "    trial_spec=trial_spec\n",
    ")\n",
    "\n",
    "\n",
    "# Experiment object.\n",
    "experiment = V1beta1Experiment(\n",
    "    api_version=\"kubeflow.org/v1beta1\",\n",
    "    kind=\"Experiment\",\n",
    "    metadata=metadata,\n",
    "    spec=V1beta1ExperimentSpec(\n",
    "        max_trial_count=3,\n",
    "        parallel_trial_count=2,\n",
    "        max_failed_trial_count=1,\n",
    "        algorithm=algorithm_spec,\n",
    "        objective=objective_spec,\n",
    "        parameters=parameters,\n",
    "        trial_template=trial_template,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Experiments with resume policy\n",
    "\n",
    "We will define another 2 Experiments with ResumePolicy = Never and ResumePolicy = FromVolume.\n",
    "\n",
    "Experiment with _Never_ resume policy can't be resumed, the Suggestion resources will be deleted.\n",
    "\n",
    "Experiment with _FromVolume_ resume policy can be resumed, volume is attached to the Suggestion. Suggestion's PVC be created for the Suggestion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_never_resume_name = \"never-resume-cmaes\"\n",
    "experiment_from_volume_resume_name = \"from-volume-resume-cmaes\"\n",
    "\n",
    "# Create new Experiments from the previous Experiment info.\n",
    "# Define Experiment with Never resume.\n",
    "experiment_never_resume = copy.deepcopy(experiment)\n",
    "experiment_never_resume.metadata.name = experiment_never_resume_name\n",
    "experiment_never_resume.spec.resume_policy = \"Never\"\n",
    "experiment_never_resume.spec.max_trial_count = 4\n",
    "\n",
    "# Define Experiment with FromVolume resume.\n",
    "experiment_from_volume_resume = copy.deepcopy(experiment)\n",
    "experiment_from_volume_resume.metadata.name = experiment_from_volume_resume_name\n",
    "experiment_from_volume_resume.spec.resume_policy = \"FromVolume\"\n",
    "experiment_from_volume_resume.spec.max_trial_count = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can print the Experiment's info to verify it before submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cmaes-example\n",
      "cmaes\n",
      "-----------------\n",
      "never-resume-cmaes\n",
      "Never\n",
      "-----------------\n",
      "from-volume-resume-cmaes\n",
      "FromVolume\n"
     ]
    }
   ],
   "source": [
    "print(experiment.metadata.name)\n",
    "print(experiment.spec.algorithm.algorithm_name)\n",
    "print(\"-----------------\")\n",
    "print(experiment_never_resume.metadata.name)\n",
    "print(experiment_never_resume.spec.resume_policy)\n",
    "print(\"-----------------\")\n",
    "print(experiment_from_volume_resume.metadata.name)\n",
    "print(experiment_from_volume_resume.spec.resume_policy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create your Experiment\n",
    "\n",
    "You have to create Katib client to use the SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment kubeflow-user-example-com/cmaes-example has been created\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Katib Experiment cmaes-example link <a href=\"/_/katib/#/katib/hp_monitor/kubeflow-user-example-com/cmaes-example\" target=\"_blank\">here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create Katib client.\n",
    "kclient = KatibClient()\n",
    "\n",
    "# Create your Experiment.\n",
    "kclient.create_experiment(experiment,namespace=namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create other Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment kubeflow-user-example-com/never-resume-cmaes has been created\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Katib Experiment never-resume-cmaes link <a href=\"/_/katib/#/katib/hp_monitor/kubeflow-user-example-com/never-resume-cmaes\" target=\"_blank\">here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment kubeflow-user-example-com/from-volume-resume-cmaes has been created\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Katib Experiment from-volume-resume-cmaes link <a href=\"/_/katib/#/katib/hp_monitor/kubeflow-user-example-com/from-volume-resume-cmaes\" target=\"_blank\">here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create Experiment with never resume.\n",
    "kclient.create_experiment(experiment_never_resume,namespace=namespace)\n",
    "# Create Experiment with from volume resume.\n",
    "kclient.create_experiment(experiment_from_volume_resume,namespace=namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get your Experiment\n",
    "\n",
    "You can get your Experiment by name and receive required data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'api_version': 'kubeflow.org/v1beta1',\n",
      " 'kind': 'Experiment',\n",
      " 'metadata': {'annotations': None,\n",
      "              'creation_timestamp': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
      "              'deletion_grace_period_seconds': None,\n",
      "              'deletion_timestamp': None,\n",
      "              'finalizers': ['update-prometheus-metrics'],\n",
      "              'generate_name': None,\n",
      "              'generation': 1,\n",
      "              'labels': None,\n",
      "              'managed_fields': [{'api_version': 'kubeflow.org/v1beta1',\n",
      "                                  'fields_type': 'FieldsV1',\n",
      "                                  'fields_v1': {'f:spec': {'.': {},\n",
      "                                                           'f:algorithm': {'.': {},\n",
      "                                                                           'f:algorithmName': {}},\n",
      "                                                           'f:maxFailedTrialCount': {},\n",
      "                                                           'f:maxTrialCount': {},\n",
      "                                                           'f:objective': {'.': {},\n",
      "                                                                           'f:additionalMetricNames': {},\n",
      "                                                                           'f:goal': {},\n",
      "                                                                           'f:objectiveMetricName': {},\n",
      "                                                                           'f:type': {}},\n",
      "                                                           'f:parallelTrialCount': {},\n",
      "                                                           'f:parameters': {},\n",
      "                                                           'f:trialTemplate': {'.': {},\n",
      "                                                                               'f:primaryContainerName': {},\n",
      "                                                                               'f:trialParameters': {},\n",
      "                                                                               'f:trialSpec': {'.': {},\n",
      "                                                                                               'f:apiVersion': {},\n",
      "                                                                                               'f:kind': {},\n",
      "                                                                                               'f:spec': {'.': {},\n",
      "                                                                                                          'f:template': {'.': {},\n",
      "                                                                                                                         'f:metadata': {'.': {},\n",
      "                                                                                                                                        'f:annotations': {'.': {},\n",
      "                                                                                                                                                          'f:sidecar.istio.io/inject': {}}},\n",
      "                                                                                                                         'f:spec': {'.': {},\n",
      "                                                                                                                                    'f:containers': {},\n",
      "                                                                                                                                    'f:restartPolicy': {}}}}}}}},\n",
      "                                  'manager': 'OpenAPI-Generator',\n",
      "                                  'operation': 'Update',\n",
      "                                  'subresource': None,\n",
      "                                  'time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal())},\n",
      "                                 {'api_version': 'kubeflow.org/v1beta1',\n",
      "                                  'fields_type': 'FieldsV1',\n",
      "                                  'fields_v1': {'f:metadata': {'f:finalizers': {'.': {},\n",
      "                                                                                'v:\"update-prometheus-metrics\"': {}}}},\n",
      "                                  'manager': 'katib-controller',\n",
      "                                  'operation': 'Update',\n",
      "                                  'subresource': None,\n",
      "                                  'time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal())},\n",
      "                                 {'api_version': 'kubeflow.org/v1beta1',\n",
      "                                  'fields_type': 'FieldsV1',\n",
      "                                  'fields_v1': {'f:status': {'.': {},\n",
      "                                                             'f:conditions': {},\n",
      "                                                             'f:currentOptimalTrial': {'.': {},\n",
      "                                                                                       'f:observation': {}},\n",
      "                                                             'f:runningTrialList': {},\n",
      "                                                             'f:startTime': {},\n",
      "                                                             'f:trials': {},\n",
      "                                                             'f:trialsRunning': {}}},\n",
      "                                  'manager': 'katib-controller',\n",
      "                                  'operation': 'Update',\n",
      "                                  'subresource': 'status',\n",
      "                                  'time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal())}],\n",
      "              'name': 'cmaes-example',\n",
      "              'namespace': 'kubeflow-user-example-com',\n",
      "              'owner_references': None,\n",
      "              'resource_version': '26516',\n",
      "              'self_link': None,\n",
      "              'uid': '1d59819e-4e5f-4adc-90cc-62c2ee867f72'},\n",
      " 'spec': {'algorithm': {'algorithm_name': 'cmaes', 'algorithm_settings': None},\n",
      "          'early_stopping': None,\n",
      "          'max_failed_trial_count': 1,\n",
      "          'max_trial_count': 3,\n",
      "          'metrics_collector_spec': {'collector': {'custom_collector': None,\n",
      "                                                   'kind': 'StdOut'},\n",
      "                                     'source': None},\n",
      "          'nas_config': None,\n",
      "          'objective': {'additional_metric_names': ['Train-accuracy'],\n",
      "                        'goal': 0.99,\n",
      "                        'metric_strategies': [{'name': 'Validation-accuracy',\n",
      "                                               'value': 'max'},\n",
      "                                              {'name': 'Train-accuracy',\n",
      "                                               'value': 'max'}],\n",
      "                        'objective_metric_name': 'Validation-accuracy',\n",
      "                        'type': 'maximize'},\n",
      "          'parallel_trial_count': 2,\n",
      "          'parameters': [{'feasible_space': {'list': None,\n",
      "                                             'max': '0.06',\n",
      "                                             'min': '0.01',\n",
      "                                             'step': None},\n",
      "                          'name': 'lr',\n",
      "                          'parameter_type': 'double'},\n",
      "                         {'feasible_space': {'list': None,\n",
      "                                             'max': '5',\n",
      "                                             'min': '2',\n",
      "                                             'step': None},\n",
      "                          'name': 'num-layers',\n",
      "                          'parameter_type': 'int'},\n",
      "                         {'feasible_space': {'list': ['sgd', 'adam', 'ftrl'],\n",
      "                                             'max': None,\n",
      "                                             'min': None,\n",
      "                                             'step': None},\n",
      "                          'name': 'optimizer',\n",
      "                          'parameter_type': 'categorical'}],\n",
      "          'resume_policy': 'LongRunning',\n",
      "          'trial_template': {'config_map': None,\n",
      "                             'failure_condition': 'status.conditions.#(type==\"Failed\")#|#(status==\"True\")#',\n",
      "                             'primary_container_name': 'training-container',\n",
      "                             'primary_pod_labels': None,\n",
      "                             'retain': None,\n",
      "                             'success_condition': 'status.conditions.#(type==\"Complete\")#|#(status==\"True\")#',\n",
      "                             'trial_parameters': [{'description': 'Learning '\n",
      "                                                                  'rate for '\n",
      "                                                                  'the '\n",
      "                                                                  'training '\n",
      "                                                                  'model',\n",
      "                                                   'name': 'learningRate',\n",
      "                                                   'reference': 'lr'},\n",
      "                                                  {'description': 'Number of '\n",
      "                                                                  'training '\n",
      "                                                                  'model '\n",
      "                                                                  'layers',\n",
      "                                                   'name': 'numberLayers',\n",
      "                                                   'reference': 'num-layers'},\n",
      "                                                  {'description': 'Training '\n",
      "                                                                  'model '\n",
      "                                                                  'optimizer '\n",
      "                                                                  '(sdg, adam '\n",
      "                                                                  'or ftrl)',\n",
      "                                                   'name': 'optimizer',\n",
      "                                                   'reference': 'optimizer'}],\n",
      "                             'trial_spec': {'apiVersion': 'batch/v1',\n",
      "                                            'kind': 'Job',\n",
      "                                            'spec': {'template': {'metadata': {'annotations': {'sidecar.istio.io/inject': 'false'}},\n",
      "                                                                  'spec': {'containers': [{'command': ['python3',\n",
      "                                                                                                       '/opt/mxnet-mnist/mnist.py',\n",
      "                                                                                                       '--batch-size=64',\n",
      "                                                                                                       '--num-epochs=1',\n",
      "                                                                                                       '--lr=${trialParameters.learningRate}',\n",
      "                                                                                                       '--num-layers=${trialParameters.numberLayers}',\n",
      "                                                                                                       '--optimizer=${trialParameters.optimizer}'],\n",
      "                                                                                           'image': 'docker.io/kubeflowkatib/mxnet-mnist:v0.14.0',\n",
      "                                                                                           'name': 'training-container'}],\n",
      "                                                                           'restartPolicy': 'Never'}}}}}},\n",
      " 'status': {'completion_time': None,\n",
      "            'conditions': [{'last_transition_time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
      "                            'last_update_time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
      "                            'message': 'Experiment is created',\n",
      "                            'reason': 'ExperimentCreated',\n",
      "                            'status': 'True',\n",
      "                            'type': 'Created'},\n",
      "                           {'last_transition_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
      "                            'last_update_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
      "                            'message': 'Experiment is running',\n",
      "                            'reason': 'ExperimentRunning',\n",
      "                            'status': 'True',\n",
      "                            'type': 'Running'}],\n",
      "            'current_optimal_trial': {'best_trial_name': None,\n",
      "                                      'observation': {'metrics': None},\n",
      "                                      'parameter_assignments': None},\n",
      "            'early_stopped_trial_list': None,\n",
      "            'failed_trial_list': None,\n",
      "            'killed_trial_list': None,\n",
      "            'last_reconcile_time': None,\n",
      "            'metrics_unavailable_trial_list': None,\n",
      "            'pending_trial_list': None,\n",
      "            'running_trial_list': ['cmaes-example-f64n8vb5',\n",
      "                                   'cmaes-example-l6zkx5jx'],\n",
      "            'start_time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
      "            'succeeded_trial_list': None,\n",
      "            'trial_metrics_unavailable': None,\n",
      "            'trials': 2,\n",
      "            'trials_early_stopped': None,\n",
      "            'trials_failed': None,\n",
      "            'trials_killed': None,\n",
      "            'trials_pending': None,\n",
      "            'trials_running': 2,\n",
      "            'trials_succeeded': None}}\n",
      "-----------------\n",
      "\n",
      "3\n",
      "{'last_transition_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
      " 'last_update_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
      " 'message': 'Experiment is running',\n",
      " 'reason': 'ExperimentRunning',\n",
      " 'status': 'True',\n",
      " 'type': 'Running'}\n"
     ]
    }
   ],
   "source": [
    "exp = kclient.get_experiment(name=experiment_name, namespace=namespace)\n",
    "print(exp)\n",
    "print(\"-----------------\\n\")\n",
    "\n",
    "# Get the max trial count and latest status.\n",
    "print(exp.spec.max_trial_count)\n",
    "print(exp.status.conditions[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get all Experiments\n",
    "\n",
    "You can get list of the current Experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cmaes-example\n",
      "from-volume-resume-cmaes\n",
      "never-resume-cmaes\n"
     ]
    }
   ],
   "source": [
    "# Get names from the running Experiments.\n",
    "exp_list = kclient.list_experiments(namespace=namespace)\n",
    "\n",
    "for exp in exp_list:\n",
    "    print(exp.metadata.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the current Experiment conditions\n",
    "\n",
    "You can check the current Experiment conditions and check if Experiment is Succeeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'last_transition_time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
       "  'last_update_time': datetime.datetime(2023, 1, 6, 14, 28, 28, tzinfo=tzlocal()),\n",
       "  'message': 'Experiment is created',\n",
       "  'reason': 'ExperimentCreated',\n",
       "  'status': 'True',\n",
       "  'type': 'Created'},\n",
       " {'last_transition_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
       "  'last_update_time': datetime.datetime(2023, 1, 6, 14, 28, 52, tzinfo=tzlocal()),\n",
       "  'message': 'Experiment is running',\n",
       "  'reason': 'ExperimentRunning',\n",
       "  'status': 'True',\n",
       "  'type': 'Running'}]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kclient.get_experiment_conditions(name=experiment_name, namespace=namespace)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kclient.is_experiment_succeeded(name=experiment_name, namespace=namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List of the current Trials\n",
    "\n",
    "You can get list of the current Trials with the latest status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial Name: cmaes-example-dd4x6tsh\n",
      "Trial Status: {'last_transition_time': datetime.datetime(2023, 1, 6, 14, 30, 43, tzinfo=tzlocal()),\n",
      " 'last_update_time': datetime.datetime(2023, 1, 6, 14, 30, 43, tzinfo=tzlocal()),\n",
      " 'message': 'Trial is running',\n",
      " 'reason': 'TrialRunning',\n",
      " 'status': 'True',\n",
      " 'type': 'Running'}\n",
      "\n",
      "Trial Name: cmaes-example-f64n8vb5\n",
      "Trial Status: {'last_transition_time': datetime.datetime(2023, 1, 6, 14, 30, 43, tzinfo=tzlocal()),\n",
      " 'last_update_time': datetime.datetime(2023, 1, 6, 14, 30, 43, tzinfo=tzlocal()),\n",
      " 'message': 'Trial has succeeded',\n",
      " 'reason': 'TrialSucceeded',\n",
      " 'status': 'True',\n",
      " 'type': 'Succeeded'}\n",
      "\n",
      "Trial Name: cmaes-example-l6zkx5jx\n",
      "Trial Status: {'last_transition_time': datetime.datetime(2023, 1, 6, 14, 30, 45, tzinfo=tzlocal()),\n",
      " 'last_update_time': datetime.datetime(2023, 1, 6, 14, 30, 45, tzinfo=tzlocal()),\n",
      " 'message': 'Trial has succeeded',\n",
      " 'reason': 'TrialSucceeded',\n",
      " 'status': 'True',\n",
      " 'type': 'Succeeded'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Trial list.\n",
    "trial_list = kclient.list_trials(experiment_name=experiment_name, namespace=namespace)\n",
    "for trial in trial_list:\n",
    "    print(f\"Trial Name: {trial.metadata.name}\")\n",
    "    print(f\"Trial Status: {trial.status.conditions[-1]}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the optimal HyperParameters\n",
    "\n",
    "You can get the current optimal Trial from your Experiment. For the each metric you can see the max, min and latest value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'best_trial_name': 'cmaes-example-l6zkx5jx',\n",
       " 'observation': {'metrics': [{'latest': '0.955613',\n",
       "                              'max': '0.955613',\n",
       "                              'min': '0.955613',\n",
       "                              'name': 'Validation-accuracy'},\n",
       "                             {'latest': '0.922775',\n",
       "                              'max': '0.922775',\n",
       "                              'min': '0.922775',\n",
       "                              'name': 'Train-accuracy'}]},\n",
       " 'parameter_assignments': [{'name': 'lr', 'value': '0.04511033252270099'},\n",
       "                           {'name': 'num-layers', 'value': '3'},\n",
       "                           {'name': 'optimizer', 'value': 'sgd'}]}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Optimal HPs.\n",
    "kclient.get_optimal_hyperparameters(name=experiment_name, namespace=namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Status for the Suggestion objects\n",
    "\n",
    "Once Experiment is Succeeded, you can check the Suggestion object status for more information about resume status.\n",
    "\n",
    "For Experiment with FromVolume you should be able to check created PVC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Suggestion is succeeded, can't be restarted\n",
      "-----------------\n",
      "Suggestion is succeeded, suggestion volume is not deleted, can be restarted\n"
     ]
    }
   ],
   "source": [
    "# Get the current Suggestion status for the never resume Experiment.\n",
    "suggestion = kclient.get_suggestion(name=experiment_never_resume_name, namespace=namespace)\n",
    "\n",
    "print(suggestion.status.conditions[-1].message)\n",
    "print(\"-----------------\")\n",
    "\n",
    "# Get the current Suggestion status for the from volume Experiment.\n",
    "suggestion = kclient.get_suggestion(name=experiment_from_volume_resume_name, namespace=namespace)\n",
    "\n",
    "print(suggestion.status.conditions[-1].message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete your Experiments\n",
    "\n",
    "You can delete your Experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment kubeflow-user-example-com/cmaes-example has been deleted\n",
      "Experiment kubeflow-user-example-com/never-resume-cmaes has been deleted\n",
      "Experiment kubeflow-user-example-com/from-volume-resume-cmaes has been deleted\n"
     ]
    }
   ],
   "source": [
    "kclient.delete_experiment(name=experiment_name, namespace=namespace)\n",
    "kclient.delete_experiment(name=experiment_never_resume_name, namespace=namespace)\n",
    "kclient.delete_experiment(name=experiment_from_volume_resume_name, namespace=namespace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}