block2/Using tapispy to Execute Actor Containers on the Abaco Cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using tapispy to Execute Actor Containers on the Abaco Cloud\n",
    "\n",
    "In this notebook, we explore the `tapispy` library's support for executing actor containers on the TACC Abaco Cloud. This tutorial assumes the user has a TACC account and access to the TACC Cloud APIs via a set of OAuth API keys. Full details on getting the required accounts can be found on Abaco's Getting Started Guide here: https://abaco.readthedocs.io/en/latest/getting-started/index.html#account-creation-and-software-installation\n",
    "\n",
    "## Installation\n",
    "\n",
    "The `tapispy` library is a Python library for interacting with TACC's Tapis API platform. It can be installed directly from PyPI using pip - `pip install tapispy`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing and Instantiating the Tapis object\n",
    "\n",
    "The primary functionality is accessed through a Tapis client, a Python object containing methods for accessing the primary APIs within the Tapis framework. The first step is to import and instantiate one of these objects.\n",
    "\n",
    "When instantiating the `Tapis` object, we must provide the `api_server` to interact with. This is because Tapis is a multi-tenant platform and serves different projects at different base URLs. To interact with the primary \"TACC\" tenant, use `api_server=https://api.tacc.utexas.edu`.\n",
    "\n",
    "We also need to supply the `Tapis` constructor with some authentication credentials. Here we have several options, the simplest of which is to just provide an OAuth access token via the `token` parameter. You can also provide a TACC `username` and `password`, together with an `api_key` and `api_secret` and use the `Tapis` object to generate an access token. For now, we'll just assume we have an access token already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tapispy.tapis import Tapis\n",
    "tp = Tapis(api_server='https://api.tacc.utexas.edu', token='<insert_access_token>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing Actors\n",
    "\n",
    "The `tp` object has object members representing the various Tapis APIs. For instance, there is an `actors` object for interacting with the TACC Abaco APIs. The `actors` object has methods for interacting with specific endpoints within the Abaco API. For example, we can list actors using the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'_links': {'executions': 'https://api.tacc.utexas.edu/actors/v2/YmDqmbXyRAj5D/executions',\n",
       "   'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "   'self': 'https://api.tacc.utexas.edu/actors/v2/YmDqmbXyRAj5D'},\n",
       "  'createTime': '2019-06-18 16:35:39.817858',\n",
       "  'defaultEnvironment': {},\n",
       "  'description': '',\n",
       "  'gid': 811324,\n",
       "  'id': 'YmDqmbXyRAj5D',\n",
       "  'image': 'abacosamples/test',\n",
       "  'lastUpdateTime': '2019-06-18 16:35:39.817858',\n",
       "  'mounts': [],\n",
       "  'owner': 'jstubbs',\n",
       "  'privileged': False,\n",
       "  'queue': 'default',\n",
       "  'state': {},\n",
       "  'stateless': True,\n",
       "  'status': 'READY',\n",
       "  'statusMessage': ' ',\n",
       "  'tasdir': '01837/jstubbs',\n",
       "  'type': 'none',\n",
       "  'uid': 811324,\n",
       "  'useContainerUid': False},\n",
       " {'_links': {'executions': 'https://api.tacc.utexas.edu/actors/v2/6PlMbDLa4zlON/executions',\n",
       "   'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "   'self': 'https://api.tacc.utexas.edu/actors/v2/6PlMbDLa4zlON'},\n",
       "  'createTime': '2018-07-03 23:14:55.433342',\n",
       "  'defaultEnvironment': {},\n",
       "  'description': 'Actor that counts words.',\n",
       "  'id': '6PlMbDLa4zlON',\n",
       "  'image': 'abacosamples/wc',\n",
       "  'lastUpdateTime': '2018-07-03 23:14:55.433342',\n",
       "  'mounts': [],\n",
       "  'name': 'word_counter',\n",
       "  'owner': 'jstubbs',\n",
       "  'privileged': False,\n",
       "  'queue': 'default',\n",
       "  'state': {},\n",
       "  'stateless': False,\n",
       "  'status': 'READY',\n",
       "  'statusMessage': ' ',\n",
       "  'type': 'none',\n",
       "  'useContainerUid': False},\n",
       " {'_links': {'executions': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7/executions',\n",
       "   'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "   'self': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7'},\n",
       "  'createTime': '2019-07-14 06:10:01.774301',\n",
       "  'defaultEnvironment': {},\n",
       "  'description': '',\n",
       "  'gid': 811324,\n",
       "  'id': 'pm0vLq1aEJW7',\n",
       "  'image': 'taccsciapps/abaco_classifier',\n",
       "  'lastUpdateTime': '2019-07-14 06:10:01.774301',\n",
       "  'mounts': [],\n",
       "  'owner': 'jstubbs',\n",
       "  'privileged': False,\n",
       "  'queue': 'default',\n",
       "  'state': {},\n",
       "  'stateless': True,\n",
       "  'status': 'READY',\n",
       "  'statusMessage': ' ',\n",
       "  'tasdir': '01837/jstubbs',\n",
       "  'type': 'none',\n",
       "  'uid': 811324,\n",
       "  'useContainerUid': False}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tp.actors.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should see our `taccsciapps/abaco_classifier` actor in the list above. We can take note of the actor ID in a variable for use later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# update this cell with your actor ID:\n",
    "actor_id = 'pm0vLq1aEJW7'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing the Classifier Actor\n",
    "\n",
    "We are now ready to execute our actor. We do this by sending the actor a message which can be done using the `tp.actors.sendMessage()` method. We pass the actor id using the `actorId` parameter and the message data using the `body` parameter, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_links': {'messages': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7/messages',\n",
       "  'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "  'self': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7/executions/YyOJBgy10lWw8'},\n",
       " 'executionId': 'YyOJBgy10lWw8',\n",
       " 'msg': 'https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/12231410/Labrador-Retriever-On-White-01.jpg'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_url = 'https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/12231410/Labrador-Retriever-On-White-01.jpg'\n",
    "tp.actors.sendMessage(actorId=actor_id, body={'message': image_url})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving the Execution Details and Logs\n",
    "\n",
    "Abaco responds with the `executionId` associated with our execution. We can use this to retrieve the execution details (including its status) and the logs associated with the execution. We use the `tp.actors.getExecution()` and `tp.actors.getExecutionLogs()` methods, respectively, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_links': {'logs': 'https://api.tacc.utexas.edu/actors/v2/TACC-PROD_pm0vLq1aEJW7/executions/YyOJBgy10lWw8/logs',\n",
       "  'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "  'self': 'https://api.tacc.utexas.edu/actors/v2/TACC-PROD_pm0vLq1aEJW7/executions/YyOJBgy10lWw8'},\n",
       " 'actorId': 'pm0vLq1aEJW7',\n",
       " 'apiServer': 'https://api.tacc.utexas.edu',\n",
       " 'cpu': 25684504485,\n",
       " 'executor': 'jstubbs',\n",
       " 'exitCode': 0,\n",
       " 'finalState': {'Dead': False,\n",
       "  'Error': '',\n",
       "  'ExitCode': 0,\n",
       "  'FinishedAt': '2019-07-14T20:28:57.870510748Z',\n",
       "  'OOMKilled': False,\n",
       "  'Paused': False,\n",
       "  'Pid': 0,\n",
       "  'Restarting': False,\n",
       "  'Running': False,\n",
       "  'StartedAt': '2019-07-14T20:28:49.891162864Z',\n",
       "  'Status': 'exited'},\n",
       " 'id': 'YyOJBgy10lWw8',\n",
       " 'io': 446918670,\n",
       " 'messageReceivedTime': '2019-07-14 20:28:42.531652',\n",
       " 'runtime': 8,\n",
       " 'startTime': '2019-07-14 20:28:49.228643',\n",
       " 'status': 'COMPLETE',\n",
       " 'workerId': 'raoPzaXMV7mwA'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# update with your execution id:\n",
    "execution_id = 'YyOJBgy10lWw8'\n",
    "ex_details = tp.actors.getExecution(actorId=actor_id, executionId=execution_id)\n",
    "ex_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_links': {'execution': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7/executions/YyOJBgy10lWw8',\n",
       "  'owner': 'https://api.tacc.utexas.edu/profiles/v2/jstubbs',\n",
       "  'self': 'https://api.tacc.utexas.edu/actors/v2/pm0vLq1aEJW7/executions/YyOJBgy10lWw8/logs'},\n",
       " 'logs': 'Contents of MSG: https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/12231410/Labrador-Retriever-On-White-01.jpg\\n2019-07-14 20:28:54.762906: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX\\n2019-07-14 20:28:55.219176: W tensorflow/core/framework/op_def_util.cc:343] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().\\n/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\\n  from ._conv import register_converters as _register_converters\\n\\nSuccessfully downloaded inception-2015-12-05.tgz 88931400 bytes.\\nLabrador retriever (score = 0.97471)\\ngolden retriever (score = 0.00324)\\nkuvasz (score = 0.00099)\\nbull mastiff (score = 0.00095)\\nSaint Bernard, St Bernard (score = 0.00067)\\n'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_logs = tp.actors.getExecutionLogs(actorId=actor_id, executionId=execution_id)\n",
    "ex_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see from the above that our classifier image classified the doa as a Lobrador retrieve with 97.47% certainty. Pretty good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We have seen how to execute an actors on the Abaco cloud using the `tapispy` library, but we have only scratched the surface of what is capable with the Abaco API. In subsequent tutorials, we will explore additional topics such as sending messages of different content types, including JSON and binary data, permissions and sharing actors with other researchers, actor events and building complex networks of actors, and more. Details on all of these topics can also be found on Abaco's official documentation site on ReadTheDocs: https://abaco.readthedocs.io"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}