nanowire-plugin for python tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center> Guide to using nanowire-plugin to build plugins for the nanowire data management system </center></h1>\n",
    "\n",
    "This is a guide to using the nanowire plugin to develop custom plugins for the nanowire data processing system. The nanowire system works by creating a server which performs some user function on code passed to it. The result of the user code is then passed to the next plugin in the pipeline.\n",
    "\n",
    "<center><img src=\"./nanowire-drawing.png\"><center>\n",
    "\n",
    "In this system the function to be turned into a plugin becomes an individual dockerised server which examines a rabbitMQ queue waiting for information to become avalible on the queue. When an object is queued on the apropriate rabbitMQ queue the plugin takes the object off the queue and passes it to the queue for the next plugin in the pipeline. In this way multiple plugins can be chained together to produce a workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h2>Creating a plugin with python</h2>\n",
    "\n",
    "In order to create your own plugin using python you will need to first install docker on your development machine. Instructions as to how to do this can be found at:-\n",
    "\n",
    "https://docs.docker.com/engine/installation/\n",
    "\n",
    "Once you have installed docker it may be worth giving docker sudo permissions as detailed here:-\n",
    "\n",
    "https://docs.docker.com/engine/installation/linux/linux-postinstall/\n",
    "\n",
    "This is necessary since all nanowire plugins are run as self contained servers from within docker containers. You will now need to convert the processing you require into a single main function that takes \"nmo\", \"jsonld\" and \"url\" as arguments. The nmo is a system tool containing information such as the filename and job id, the jsonld is the tool which contains the data produced by previous plugins in the pipeline (this will be None if it is the first plugin in the pipeline) and url is the minio link to a file to be processed. This is used in instances such as image processing plugins. An example skeleton for your plugin is shown below:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "\n",
    "#Example plugin template\n",
    "\n",
    "import example_plugin_functions as epf\n",
    "\n",
    "from nanowire_plugin.single_file_plugin import bind\n",
    "\n",
    "# The function which the plugin is bound to. This function will be run on the\n",
    "#inputs from the pipeline.\n",
    "\n",
    "#metadata is used for backend tasks and should only be usefull to a few advanced\n",
    "#users. It is a dictionary containing system information.\n",
    "\n",
    "#jsonld is the object most plugin developers will be interested in as it contains\n",
    "#the results from previous plugins. These plugins may include text extraction or\n",
    "#social media scraping meaning that the jsonld will contain the text to be processed\n",
    "#or the interaction statistics to be examined. If you are creating a plugin which is the\n",
    "#first in a pipeline this will be \"None\" and you should create a jsonld to return\n",
    "\n",
    "#The url is for processing of non-document files. When a plugin is designed to\n",
    "#process an image or a pdf this url will be a link to that file. The developer should\n",
    "#grab the image and then perform their processing\n",
    "def main(metadata, jsonld, url):\n",
    "\n",
    "\n",
    "    ######################\n",
    "    ### YOUR CODE HERE ###\n",
    "    ######################\n",
    "\n",
    "\n",
    "    return jsonld\n",
    "    \n",
    "#The bind function converts your plugin into a server. This means that all you need to worry \n",
    "# about is the \"main\" function and the nanowire-plugin module will handle all of the nanowire\n",
    "# specific requirements \n",
    "bind(main, debug_mode=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the basic structure of the plugin. This python script must be baked into a docker file as demonstrated in the following code segment (please note that nanowire-plugin also supports python 2.7):-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Initialise the docker container by grabbing the official python repo. Advanced users may like to use their \n",
    "#own custom repos to reduce build time\n",
    "FROM python:3.6\n",
    "\n",
    "#install the python libraries that your plugin requires\n",
    "ADD requirements.txt /\n",
    "RUN pip3 install -r /requirements.txt\n",
    "\n",
    "ADD example_plugin_template.py /\n",
    "\n",
    "CMD [\"python3\", \"/example_plugin_template.py\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that the modules required by our plugin have been installed using a file called requirements.txt. An example of this file is shown below:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "requests==2.19.1\n",
    "nanowire-plugin==0.14.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice above that the most recent version of nanowire-plugin has also been installed here. Once the code has been built and pushed to jenkins <b>[MORE DETAILS NEEDED HERE WHEN WE GO PUBLIC, ARE WE GOING TO MAKE AN INTERFACE TO HELP 3rd PARTY DEVS?]</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@Library('PipelineHelpers') _\n",
    "\n",
    "pipeline {\n",
    "    agent any\n",
    "\n",
    "    environment {\n",
    "        PIPELINE_NAME = pipelineHelpers.getPipelineName(currentBuild)\n",
    "        DOCKER_IMAGE = \"plugins/example_plugin:${env.BRANCH_NAME}-${env.BUILD_NUMBER}\"\n",
    "    }\n",
    "\n",
    "    stages {\n",
    "        stage('Build') {\n",
    "            steps {\n",
    "                checkout scm\n",
    "\n",
    "                script {\n",
    "                    docker.build(env.DOCKER_IMAGE)\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "\n",
    "        stage('Test') {\n",
    "            steps {\n",
    "                script {\n",
    "                    dockerHelpers.runInDockerImage(\n",
    "                        env.DOCKER_IMAGE,\n",
    "                        \"python3 ./unit_tests.py\", [\n",
    "                        ]\n",
    "                    )\n",
    "\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "\n",
    "        stage('Push to Dockerhub') {\n",
    "            steps {\n",
    "                script {\n",
    "                    dockerHelpers.pushImageToDefaultRegistry(env.DOCKER_IMAGE, [env.BRANCH_NAME])\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "        stage('Deploy') {\n",
    "            steps{\n",
    "                script{\n",
    "                    kubeHelpers.deployPlugin(env.DOCKER_IMAGE, env.BRANCH_NAME)\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "    post {\n",
    "        always {\n",
    "            script {\n",
    "                pipelineHelpers.notifySlack(currentBuild)\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the code is built within nanowire you will need to convert your container into a nanowire-plugin. When you find yourself on the create plugin page you will be asked for a schema. An example of this is shown below <i>[WARNING, THE FOLLOWING IS TRUE FOR NANOWIRE V0.2. THIS IS SUBJECT TO CHANGE]</i>:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "{\n",
    "    \"name\": \"example-plugin\",\n",
    "    \"description\": \"This is an example nanowire.json. This will be used to turn your plugin into a component of nanowire. Please note that this may be subject to future change.\",\n",
    "    \"image\": \"docker.spotlightdata.co.uk/plugins/example_plugin:dev\",\n",
    "    \"author\": \"Your name here\",\n",
    "    \"email\": \"test@example.co.uk\",\n",
    "    \"types\": [\"SINGLE_FILE\"], \n",
    "    \"inputs\": [\n",
    "        \"source\"\n",
    "    ],\n",
    "    \"outputs\": [\n",
    "        \"plugin\"\n",
    "    ],\n",
    "    \"cmd\": [\n",
    "        \"python3\",\n",
    "        \"example_plugin_template.py\"\n",
    "    ],\n",
    "    \"env\": {},\n",
    "    \"cpu\": \"10m\",\n",
    "    \"memory\": \"100M\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### THE NEXT SECTION WAS WRITTEN BEFORE AUTOMATIC SETTING OF THE ENVIROMENTAL VARIABLES WAS SET UP. IT IS SUBJECT TO CHANGE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the plugin id from the plugin setup page as shown below\n",
    "\n",
    "<img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}