german_credit_train_and_config.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Package installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!rm -rf $PIP_BUILD\n",
    "!pip install --upgrade watson-machine-learning-client --no-cache | tail -n 1\n",
    "!pip install --upgrade ibm-ai-openscale --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Restart the kernel to assure the new libraries are being used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For OpenScale credentials, if you have your AIOS GUID, enter it below. If not, navigate to your [Cloud console](https://cloud.ibm.com/resources), click the word **default** in the same row your OpenScale instance from the list of services, and paste copy the **GUID** from the panel that pops up on the right side of the screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "AIOS_GUID = \"PASTE HERE\"\n",
    "print(AIOS_GUID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your Cloud API key can be generated by going to the [Bluemix console](https://console.bluemix.net/) and clicking Manage->Account->Users, selecting \"Platform API Keys\" from the menu on the left, and then clicking the \"Create\" button."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = \"PASTE KEY HERE\"\n",
    "\n",
    "AIOS_CREDENTIALS = {\n",
    "    \"instance_guid\": AIOS_GUID,\n",
    "    \"apikey\": CLOUD_API_KEY,\n",
    "    \"url\": \"https://api.aiopenscale.cloud.ibm.com\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy and paste your WML credentials into the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "WML_CREDENTIALS = {\n",
    "  \"PASTE HERE\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy and paste your Db2 Warehouse credentials in the cell below.\n",
    "\n",
    "If you have previously configured OpenScale, this notebook will use your existing datamart.\n",
    "\n",
    "__If you previously configured OpenScale to use the free internal version of PostgreSQL, you will be unable to complete the continuous learning/model lifecycle portion of the notebook.__ If you would like to delete this configuration and create a new one using Db2 Warehouse, set the __KEEP_MY_INTERNAL_POSTGRES__ variable to __False__ below. In this case, the notebook will remove your existing internal PostgreSQL datamart and create a new one with the supplied credentials. __*NO DATA MIGRATION WILL OCCUR.*__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DB2_CREDENTIALS = {\n",
    "    \"PASTE HERE\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEEP_MY_INTERNAL_POSTGRES = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a schema in your Db2 instance you would like to use for OpenScale data, specify it below. If you leave the variable set to None, OpenScale will use the default Db2 schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SCHEMA_NAME = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Paste your Spark credentials in the cell below. If you do not wish to do the model evaluation and retraining portion of the lab, you may skip this part and leave the variable set to None."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SPARK_CREDENTIALS = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and explore data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the training data from github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!rm credit_risk_training.csv\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/credit_risk_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "import json\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "df_data = spark.read.csv(path=\"credit_risk_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store the training data in Db2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if SCHEMA_NAME is None:\n",
    "    SCHEMA_NAME = DB2_CREDENTIALS['username']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Connect to Db2 database and schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import ibm_db\n",
    "\n",
    "dsn = (\n",
    "    \"DRIVER={{IBM DB2 ODBC DRIVER}};\"\n",
    "    \"DATABASE={0};\"\n",
    "    \"HOSTNAME={1};\"\n",
    "    \"PORT={2};\"\n",
    "    \"PROTOCOL=TCPIP;\"\n",
    "    \"UID={3};\"\n",
    "    \"PWD={4};\").format(DB2_CREDENTIALS['db'], DB2_CREDENTIALS['hostname'], DB2_CREDENTIALS['port'], DB2_CREDENTIALS['username'], DB2_CREDENTIALS['password'])\n",
    "\n",
    "conn = ibm_db.connect(dsn, \"\", \"\")\n",
    "\n",
    "query = \"SET SCHEMA {0}\".format(SCHEMA_NAME)\n",
    "stmt = ibm_db.exec_immediate(conn, query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check to see if our training data table exists; delete if it does"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    query = \"DROP TABLE GERMAN_CREDIT_TRAINING_DATA;\"\n",
    "    stmt = ibm_db.exec_immediate(conn, query)\n",
    "    print('Dropping existing training data table')\n",
    "except:\n",
    "    print('Training data table does not exist')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ingest.Connectors import Connectors\n",
    "\n",
    "dashdbsaveoption = {\n",
    "    Connectors.DASHDB.HOST              : DB2_CREDENTIALS['hostname'],\n",
    "    Connectors.DASHDB.DATABASE          : DB2_CREDENTIALS['db'],\n",
    "    Connectors.DASHDB.USERNAME          : DB2_CREDENTIALS['username'],\n",
    "    Connectors.DASHDB.PASSWORD          : DB2_CREDENTIALS['password'],\n",
    "    Connectors.DASHDB.TARGET_TABLE_NAME : 'GERMAN_CREDIT_TRAINING_DATA',\n",
    "    Connectors.DASHDB.TARGET_WRITE_MODE : 'insert'\n",
    "}\n",
    "\n",
    "df_data.write.format(\"com.ibm.spark.discover\").options(**dashdbsaveoption).save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Number of records: \" + str(df_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "spark_df = df_data\n",
    "(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n",
    "\n",
    "MODEL_NAME = \"AIOS Spark German Risk Model - Final\"\n",
    "DEPLOYMENT_NAME = \"AIOS Spark German Risk Deployment - Final\"\n",
    "\n",
    "print(\"Number of records for training: \" + str(train_data.count()))\n",
    "print(\"Number of records for evaluation: \" + str(test_data.count()))\n",
    "\n",
    "spark_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml import Pipeline, Model\n",
    "\n",
    "si_CheckingStatus = StringIndexer(inputCol = 'CheckingStatus', outputCol = 'CheckingStatus_IX')\n",
    "si_CreditHistory = StringIndexer(inputCol = 'CreditHistory', outputCol = 'CreditHistory_IX')\n",
    "si_LoanPurpose = StringIndexer(inputCol = 'LoanPurpose', outputCol = 'LoanPurpose_IX')\n",
    "si_ExistingSavings = StringIndexer(inputCol = 'ExistingSavings', outputCol = 'ExistingSavings_IX')\n",
    "si_EmploymentDuration = StringIndexer(inputCol = 'EmploymentDuration', outputCol = 'EmploymentDuration_IX')\n",
    "si_Sex = StringIndexer(inputCol = 'Sex', outputCol = 'Sex_IX')\n",
    "si_OthersOnLoan = StringIndexer(inputCol = 'OthersOnLoan', outputCol = 'OthersOnLoan_IX')\n",
    "si_OwnsProperty = StringIndexer(inputCol = 'OwnsProperty', outputCol = 'OwnsProperty_IX')\n",
    "si_InstallmentPlans = StringIndexer(inputCol = 'InstallmentPlans', outputCol = 'InstallmentPlans_IX')\n",
    "si_Housing = StringIndexer(inputCol = 'Housing', outputCol = 'Housing_IX')\n",
    "si_Job = StringIndexer(inputCol = 'Job', outputCol = 'Job_IX')\n",
    "si_Telephone = StringIndexer(inputCol = 'Telephone', outputCol = 'Telephone_IX')\n",
    "si_ForeignWorker = StringIndexer(inputCol = 'ForeignWorker', outputCol = 'ForeignWorker_IX')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "va_features = VectorAssembler(inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\", \"EmploymentDuration_IX\", \"Sex_IX\", \\\n",
    "                                         \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\", \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \\\n",
    "                                         \"LoanDuration\", \"LoanAmount\", \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\", \\\n",
    "                                         \"Dependents\"], outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "classifier = RandomForestClassifier(featuresCol=\"features\")\n",
    "\n",
    "pipeline = Pipeline(stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker, si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\\\n",
    "                               si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n",
    "model = pipeline.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "predictions = model.transform(test_data)\n",
    "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "area_under_curve = evaluatorDT.evaluate(predictions)\n",
    "\n",
    "#default evaluation is areaUnderROC\n",
    "print(\"areaUnderROC = %g\" % area_under_curve)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save and deploy the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from watson_machine_learning_client import WatsonMachineLearningAPIClient\n",
    "import json\n",
    "\n",
    "wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing model and deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_deployment_ids = wml_client.deployments.get_uids()\n",
    "for deployment_id in model_deployment_ids:\n",
    "    deployment = wml_client.deployments.get_details(deployment_id)\n",
    "    model_id = deployment['entity']['deployable_asset']['guid']\n",
    "    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n",
    "        print('Deleting deployment id', deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print('Deleting model id', model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "training_data_reference = {\n",
    "    \"name\": \"German Credit Training Data\",\n",
    "    \"connection\": DB2_CREDENTIALS,\n",
    "    \"source\": {\n",
    "        \"tablename\": \"GERMAN_CREDIT_TRAINING_DATA\",\n",
    "        \"type\": \"dashdb\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "OUTPUT_DATA_SCHEMA = {'fields': [{'metadata': {'measure': 'discrete',\n",
    "      'modeling_role': 'feature'},\n",
    "     'name': 'CheckingStatus',\n",
    "     'nullable': True,\n",
    "     'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'}, 'name': 'LoanDuration', 'nullable': True, 'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'}, 'name': 'CreditHistory', 'nullable': True, 'type': 'string'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'}, 'name': 'LoanPurpose', 'nullable': True, 'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'}, 'name': 'LoanAmount', 'nullable': True, 'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'}, 'name': 'ExistingSavings', 'nullable': True, 'type': 'string'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'}, 'name': 'EmploymentDuration', 'nullable': True, 'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'}, 'name': 'InstallmentPercent', 'nullable': True, 'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'}, 'name': 'Sex','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'OthersOnLoan','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'},'name': 'CurrentResidenceDuration','nullable': True,'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'OwnsProperty','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'},'name': 'Age','nullable': True,'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'InstallmentPlans','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'Housing','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'},'name': 'ExistingCreditsCount','nullable': True,'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'Job','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'feature'},'name': 'Dependents','nullable': True,'type': 'integer'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'Telephone','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'measure': 'discrete', 'modeling_role': 'feature'},'name': 'ForeignWorker','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'probability'},'name': 'probability','nullable': True,'type': {'containsNull': True, 'elementType': 'double', 'type': 'array'}},\n",
    "    {'metadata': {'modeling_role': 'prediction'},'name': 'prediction','nullable': True,'type': 'double'},\n",
    "    {'metadata': {'modeling_role': 'decoded-target'},'name': 'predictedLabel','nullable': True,'type': 'string'},\n",
    "    {'metadata': {'modeling_role': 'debiased-prediction'},'name': 'debiased_prediction','nullable': True,'type': 'double'},\n",
    "    {'metadata': {'modeling_role': 'debiased-probability'},'name': 'debiased_probability','nullable': True,'type': {'containsNull': True,'elementType': 'double','type': 'array'}}],\n",
    "   'type': 'struct'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_props = {\n",
    "    wml_client.repository.ModelMetaNames.NAME: \"{}\".format(MODEL_NAME),\n",
    "    wml_client.repository.ModelMetaNames.TRAINING_DATA_REFERENCE: training_data_reference,\n",
    "    wml_client.repository.ModelMetaNames.EVALUATION_METHOD: \"binary\",\n",
    "    wml_client.repository.ModelMetaNames.EVALUATION_METRICS: [\n",
    "        {\n",
    "           \"name\": \"areaUnderROC\",\n",
    "           \"value\": area_under_curve,\n",
    "           \"threshold\": 0.7\n",
    "        }\n",
    "    ],\n",
    "    wml_client.repository.ModelMetaNames.OUTPUT_DATA_SCHEMA: OUTPUT_DATA_SCHEMA\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wml_models = wml_client.repository.get_details()\n",
    "model_uid = None\n",
    "for model_in in wml_models['models']['resources']:\n",
    "    if MODEL_NAME == model_in['entity']['name']:\n",
    "        model_uid = model_in['metadata']['guid']\n",
    "        break\n",
    "\n",
    "if model_uid is None:\n",
    "    print(\"Storing model ...\")\n",
    "\n",
    "    published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props, training_data=train_data, pipeline=pipeline)\n",
    "    model_uid = wml_client.repository.get_model_uid(published_model_details)\n",
    "    print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_uid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wml_deployments = wml_client.deployments.get_details()\n",
    "deployment_uid = None\n",
    "for deployment in wml_deployments['resources']:\n",
    "    if DEPLOYMENT_NAME == deployment['entity']['name']:\n",
    "        deployment_uid = deployment['metadata']['guid']\n",
    "        break\n",
    "\n",
    "if deployment_uid is None:\n",
    "    print(\"Deploying model...\")\n",
    "\n",
    "    deployment = wml_client.deployments.create(artifact_uid=model_uid, name=DEPLOYMENT_NAME, asynchronous=False)\n",
    "    deployment_uid = wml_client.deployments.get_uid(deployment)\n",
    "    \n",
    "print(\"Model id: {}\".format(model_uid))\n",
    "print(\"Deployment id: {}\".format(deployment_uid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure OpenScale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale import APIClient\n",
    "from ibm_ai_openscale.engines import *\n",
    "from ibm_ai_openscale.utils import *\n",
    "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n",
    "from ibm_ai_openscale.supporting_classes.enums import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create schema and datamart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ai_client = APIClient(aios_credentials=AIOS_CREDENTIALS)\n",
    "ai_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up datamart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    data_mart_details = ai_client.data_mart.get_details()\n",
    "    if 'internal_database' in data_mart_details['database_configuration'] and data_mart_details['database_configuration']['internal_database']:\n",
    "        if KEEP_MY_INTERNAL_POSTGRES:\n",
    "            print('Using existing internal datamart. YOU WILL NOT BE ABLE TO COMPLETE THE CONTINUOUS LEARNING PORTION OF THE NOTEBOOK.')\n",
    "        else:\n",
    "            print('Switching to external datamart')\n",
    "            ai_client.data_mart.delete(force=True)\n",
    "            ai_client.data_mart.setup(db_credentials=DB2_CREDENTIALS, schema=SCHEMA_NAME)\n",
    "    else:\n",
    "        print('Using existing external datamart')\n",
    "except:\n",
    "    print('Setting up external datamart')\n",
    "    ai_client.data_mart.setup(db_credentials=DB2_CREDENTIALS, schema=SCHEMA_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_mart_details = ai_client.data_mart.get_details()\n",
    "data_mart_details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bind machine learning engines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "binding_uid = ai_client.data_mart.bindings.add('WML instance', WatsonMachineLearningInstance(WML_CREDENTIALS))\n",
    "if binding_uid is None:\n",
    "    binding_uid = ai_client.data_mart.bindings.get_details()['service_bindings'][0]['metadata']['guid']\n",
    "bindings_details = ai_client.data_mart.bindings.get_details()\n",
    "ai_client.data_mart.bindings.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(binding_uid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ai_client.data_mart.bindings.list_assets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing credit risk subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "for subscription in subscriptions_uids:\n",
    "    sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n",
    "    if sub_name == MODEL_NAME:\n",
    "        ai_client.data_mart.subscriptions.delete(subscription)\n",
    "        print('Deleted existing subscription for', MODEL_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset(\n",
    "    model_uid,\n",
    "    label_column='Risk',\n",
    "    prediction_column='predictedLabel',\n",
    "    probability_column='probability'\n",
    "))\n",
    "if subscription is None:\n",
    "    print('Subscription already exists; get the existing one')\n",
    "    subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "    for sub in subscriptions_uids:\n",
    "        if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:\n",
    "            subscription = ai_client.data_mart.subscriptions.get(sub)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get subscription list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n",
    "ai_client.data_mart.subscriptions.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the model so we can configure monitors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "credit_risk_scoring_endpoint = None\n",
    "print(deployment_uid)\n",
    "\n",
    "for deployment in wml_client.deployments.get_details()['resources']:\n",
    "    if deployment_uid in deployment['metadata']['guid']:\n",
    "        credit_risk_scoring_endpoint = deployment['entity']['scoring_url']\n",
    "        \n",
    "print(credit_risk_scoring_endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}\n",
    "scoring_response = wml_client.deployments.score(credit_risk_scoring_endpoint, payload_scoring)\n",
    "\n",
    "print(scoring_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quality and feedback monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enable quality monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.quality_monitoring.enable(problem_type=ProblemType.BINARY_CLASSIFICATION, threshold=0.7, min_records=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedback logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.feedback_logging.store(\n",
    "    [\n",
    "        [\"no_checking\",28,\"outstanding_credit\",\"appliances\",5990,\"500_to_1000\",\"greater_7\",5,\"male\",\"co-applicant\",3,\"car_other\",55,\"none\",\"free\",2,\"skilled\",2,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"greater_200\",22,\"all_credits_paid_back\",\"car_used\",3376,\"less_100\",\"less_1\",3,\"female\",\"none\",2,\"car_other\",32,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",39,\"credits_paid_to_date\",\"vacation\",6434,\"unknown\",\"greater_7\",5,\"male\",\"none\",4,\"car_other\",39,\"none\",\"own\",2,\"skilled\",2,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"0_to_200\",20,\"credits_paid_to_date\",\"furniture\",2442,\"less_100\",\"unemployed\",3,\"female\",\"none\",1,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"greater_200\",4,\"all_credits_paid_back\",\"education\",4206,\"less_100\",\"unemployed\",1,\"female\",\"none\",3,\"savings_insurance\",27,\"none\",\"own\",1,\"management_self-employed\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"greater_200\",23,\"credits_paid_to_date\",\"car_used\",2963,\"greater_1000\",\"greater_7\",4,\"male\",\"none\",4,\"car_other\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",31,\"prior_payments_delayed\",\"vacation\",2673,\"500_to_1000\",\"1_to_4\",3,\"male\",\"none\",2,\"real_estate\",35,\"stores\",\"rent\",1,\"skilled\",2,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",37,\"prior_payments_delayed\",\"other\",6971,\"500_to_1000\",\"1_to_4\",3,\"male\",\"none\",3,\"savings_insurance\",54,\"none\",\"own\",2,\"skilled\",1,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",14,\"all_credits_paid_back\",\"car_new\",1525,\"500_to_1000\",\"4_to_7\",3,\"male\",\"none\",4,\"real_estate\",33,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",10,\"prior_payments_delayed\",\"furniture\",4037,\"less_100\",\"4_to_7\",3,\"male\",\"none\",3,\"savings_insurance\",31,\"none\",\"rent\",1,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"0_to_200\",28,\"credits_paid_to_date\",\"retraining\",1152,\"less_100\",\"less_1\",2,\"female\",\"none\",2,\"savings_insurance\",20,\"stores\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",17,\"credits_paid_to_date\",\"car_new\",1880,\"less_100\",\"less_1\",3,\"female\",\"co-applicant\",2,\"savings_insurance\",41,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"0_to_200\",39,\"prior_payments_delayed\",\"appliances\",5685,\"100_to_500\",\"1_to_4\",4,\"female\",\"none\",2,\"unknown\",37,\"none\",\"own\",2,\"skilled\",1,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",32,\"prior_payments_delayed\",\"radio_tv\",5105,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",5,\"savings_insurance\",44,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",38,\"prior_payments_delayed\",\"appliances\",4990,\"500_to_1000\",\"greater_7\",4,\"male\",\"none\",4,\"car_other\",50,\"bank\",\"own\",2,\"unemployed\",2,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"less_0\",17,\"credits_paid_to_date\",\"furniture\",1017,\"less_100\",\"less_1\",2,\"female\",\"none\",1,\"car_other\",30,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",33,\"all_credits_paid_back\",\"car_new\",3618,\"500_to_1000\",\"4_to_7\",2,\"male\",\"none\",3,\"unknown\",31,\"stores\",\"own\",2,\"unskilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",12,\"no_credits\",\"car_new\",3037,\"less_100\",\"less_1\",1,\"female\",\"none\",2,\"car_other\",31,\"stores\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",23,\"prior_payments_delayed\",\"furniture\",1440,\"100_to_500\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",39,\"stores\",\"own\",1,\"unskilled\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",18,\"prior_payments_delayed\",\"retraining\",4032,\"less_100\",\"1_to_4\",2,\"female\",\"none\",2,\"car_other\",36,\"none\",\"rent\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",11,\"prior_payments_delayed\",\"car_used\",944,\"greater_1000\",\"1_to_4\",3,\"male\",\"none\",4,\"real_estate\",35,\"none\",\"own\",1,\"management_self-employed\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",36,\"prior_payments_delayed\",\"appliances\",5927,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",3,\"savings_insurance\",47,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",50,\"outstanding_credit\",\"other\",4694,\"unknown\",\"greater_7\",4,\"male\",\"none\",4,\"unknown\",37,\"none\",\"own\",1,\"skilled\",2,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",32,\"prior_payments_delayed\",\"radio_tv\",10584,\"100_to_500\",\"1_to_4\",3,\"male\",\"co-applicant\",3,\"unknown\",46,\"stores\",\"own\",2,\"unskilled\",2,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",41,\"prior_payments_delayed\",\"furniture\",8900,\"500_to_1000\",\"4_to_7\",4,\"male\",\"co-applicant\",3,\"car_other\",26,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"0_to_200\",14,\"credits_paid_to_date\",\"car_used\",1144,\"100_to_500\",\"less_1\",2,\"female\",\"none\",2,\"real_estate\",33,\"none\",\"rent\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",14,\"outstanding_credit\",\"appliances\",1680,\"100_to_500\",\"greater_7\",4,\"male\",\"none\",3,\"car_other\",47,\"none\",\"own\",1,\"management_self-employed\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"0_to_200\",23,\"credits_paid_to_date\",\"retraining\",3387,\"less_100\",\"less_1\",3,\"female\",\"none\",3,\"savings_insurance\",28,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",14,\"credits_paid_to_date\",\"furniture\",1269,\"500_to_1000\",\"greater_7\",2,\"male\",\"none\",2,\"savings_insurance\",39,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",36,\"prior_payments_delayed\",\"appliances\",9570,\"100_to_500\",\"4_to_7\",4,\"male\",\"co-applicant\",3,\"car_other\",53,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",16,\"credits_paid_to_date\",\"car_new\",1428,\"less_100\",\"4_to_7\",1,\"male\",\"none\",1,\"car_other\",20,\"bank\",\"rent\",1,\"unemployed\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",24,\"outstanding_credit\",\"car_used\",4620,\"greater_1000\",\"1_to_4\",3,\"male\",\"none\",4,\"savings_insurance\",40,\"none\",\"own\",2,\"skilled\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",34,\"prior_payments_delayed\",\"furniture\",2196,\"500_to_1000\",\"greater_7\",3,\"male\",\"none\",4,\"savings_insurance\",27,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",25,\"prior_payments_delayed\",\"car_used\",8708,\"100_to_500\",\"1_to_4\",4,\"male\",\"none\",5,\"car_other\",43,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",37,\"outstanding_credit\",\"radio_tv\",10550,\"unknown\",\"greater_7\",5,\"male\",\"co-applicant\",4,\"unknown\",48,\"stores\",\"own\",2,\"unemployed\",2,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",27,\"prior_payments_delayed\",\"radio_tv\",4981,\"500_to_1000\",\"4_to_7\",4,\"male\",\"none\",4,\"savings_insurance\",47,\"none\",\"own\",2,\"management_self-employed\",2,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",13,\"all_credits_paid_back\",\"car_new\",2436,\"less_100\",\"less_1\",2,\"female\",\"none\",1,\"savings_insurance\",19,\"stores\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"greater_200\",25,\"outstanding_credit\",\"appliances\",4136,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",2,\"car_other\",46,\"bank\",\"own\",1,\"unemployed\",1,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",15,\"credits_paid_to_date\",\"retraining\",4014,\"less_100\",\"1_to_4\",4,\"male\",\"co-applicant\",4,\"savings_insurance\",33,\"none\",\"own\",1,\"skilled\",1,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"no_checking\",28,\"prior_payments_delayed\",\"appliances\",5440,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",2,\"unknown\",40,\"none\",\"own\",2,\"skilled\",1,\"yes\",\"yes\",\"Risk\"],\n",
    "        [\"less_0\",13,\"prior_payments_delayed\",\"appliances\",250,\"500_to_1000\",\"4_to_7\",2,\"male\",\"none\",3,\"car_other\",28,\"stores\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",19,\"credits_paid_to_date\",\"furniture\",2111,\"less_100\",\"4_to_7\",3,\"male\",\"none\",2,\"savings_insurance\",34,\"bank\",\"own\",1,\"unemployed\",2,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",27,\"prior_payments_delayed\",\"appliances\",6455,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",4,\"car_other\",43,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"less_0\",17,\"credits_paid_to_date\",\"car_used\",250,\"less_100\",\"4_to_7\",3,\"female\",\"none\",2,\"real_estate\",40,\"none\",\"free\",2,\"skilled\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",27,\"prior_payments_delayed\",\"radio_tv\",4521,\"100_to_500\",\"less_1\",4,\"male\",\"none\",4,\"savings_insurance\",28,\"none\",\"own\",1,\"management_self-employed\",2,\"yes\",\"yes\",\"No Risk\"],\n",
    "        [\"no_checking\",37,\"prior_payments_delayed\",\"other\",7945,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",39,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\",\"No Risk\"],\n",
    "        [\"less_0\",6,\"all_credits_paid_back\",\"car_used\",250,\"less_100\",\"1_to_4\",2,\"male\",\"none\",2,\"savings_insurance\",28,\"stores\",\"rent\",1,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"less_0\",14,\"all_credits_paid_back\",\"appliances\",1431,\"less_100\",\"unemployed\",1,\"female\",\"none\",1,\"car_other\",25,\"stores\",\"own\",1,\"skilled\",1,\"none\",\"yes\",\"Risk\"],\n",
    "        [\"greater_200\",5,\"credits_paid_to_date\",\"car_used\",250,\"less_100\",\"4_to_7\",3,\"male\",\"none\",2,\"savings_insurance\",42,\"none\",\"rent\",1,\"skilled\",1,\"none\",\"yes\",\"No Risk\"]\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.feedback_logging.show_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run_details = subscription.quality_monitoring.run()\n",
    "status = run_details['status']\n",
    "id = run_details['id']\n",
    "print(id)\n",
    "\n",
    "print(\"Run status: {}\".format(status))\n",
    "\n",
    "start_time = time.time()\n",
    "elapsed_time = 0\n",
    "\n",
    "while status != 'completed' and elapsed_time < 60:\n",
    "    time.sleep(10)\n",
    "    run_details = subscription.quality_monitoring.get_run_details(run_uid=id)\n",
    "    status = run_details['status']\n",
    "    elapsed_time = time.time() - start_time\n",
    "    print(\"Run status: {}\".format(status))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.quality_monitoring.get_run_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.quality_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.quality_monitoring._get_data_from_rest_api()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ai_client.data_mart.get_deployment_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fairness monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.fairness_monitoring.enable(\n",
    "            features=[\n",
    "                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n",
    "                Feature(\"Age\", majority=[[26,75]], minority=[[18,25]], threshold=0.95)\n",
    "            ],\n",
    "            prediction_column='Risk',\n",
    "            favourable_classes=['No Risk'],\n",
    "            unfavourable_classes=['Risk'],\n",
    "            min_records=200\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score the model again now that monitoring is configured"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm german_credit_feed.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/german_credit_feed.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Score 200 randomly chosen records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "with open('german_credit_feed.json', 'r') as scoring_file:\n",
    "    scoring_data = json.load(scoring_file)\n",
    "\n",
    "fields = scoring_data['fields']\n",
    "values = []\n",
    "for _ in range(200):\n",
    "    values.append(random.choice(scoring_data['values']))\n",
    "payload_scoring = {\"fields\": fields, \"values\": values}\n",
    "\n",
    "scoring_response = wml_client.deployments.score(credit_risk_scoring_endpoint, payload_scoring)\n",
    "print(scoring_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Insert historical payloads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!rm payload_history*.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_1.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_2.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_3.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_4.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_5.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_6.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/payload_history_7.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "historyDays = 7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n",
    "import datetime\n",
    "import time\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Loading day {}'.format(day + 1))\n",
    "    history_file = 'payload_history_' + str(day + 1) + '.json'\n",
    "    with open(history_file) as f:\n",
    "        payloads = json.load(f)\n",
    "        hourly_records = int(len(payloads) / 24)\n",
    "        index = 0\n",
    "        for hour in range(24):\n",
    "            recordsList = []\n",
    "            for i in range(hourly_records):\n",
    "                score_time = str(datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1))))\n",
    "                recordsList.append(PayloadRecord(request=payloads[index]['request'], response=payloads[index]['response'], scoring_timestamp=score_time))\n",
    "                index += 1\n",
    "            subscription.payload_logging.store(records=recordsList)\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_mart_id = subscription.get_details()['metadata']['url'].split('/service_bindings')[0].split('marts/')[1]\n",
    "print(data_mart_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "performance_metrics_url = 'https://api.aiopenscale.cloud.ibm.com' + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/metrics'\n",
    "print(performance_metrics_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical fairness metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm fairness_history.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/fairness_history.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "token_data = {\n",
    "    'grant_type': 'urn:ibm:params:oauth:grant-type:apikey',\n",
    "    'response_type': 'cloud_iam',\n",
    "    'apikey': AIOS_CREDENTIALS['apikey']\n",
    "}\n",
    "\n",
    "response = requests.post('https://iam.bluemix.net/identity/token', data=token_data)\n",
    "iam_token = response.json()['access_token']\n",
    "iam_headers = {\n",
    "    'Content-Type': 'application/json',\n",
    "    'Authorization': 'Bearer %s' % iam_token\n",
    "}\n",
    "\n",
    "with open('fairness_history.json', 'r') as history_file:\n",
    "    payloads = json.load(history_file)\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        \n",
    "        qualityMetric = {\n",
    "            'metric_type': 'fairness',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': model_uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': random.choice(payloads)\n",
    "        }\n",
    "\n",
    "        response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers)\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical quality metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "token_data = {\n",
    "    'grant_type': 'urn:ibm:params:oauth:grant-type:apikey',\n",
    "    'response_type': 'cloud_iam',\n",
    "    'apikey': AIOS_CREDENTIALS['apikey']\n",
    "}\n",
    "\n",
    "response = requests.post('https://iam.bluemix.net/identity/token', data=token_data)\n",
    "iam_token = response.json()['access_token']\n",
    "iam_headers = {\n",
    "    'Content-Type': 'application/json',\n",
    "    'Authorization': 'Bearer %s' % iam_token\n",
    "}\n",
    "\n",
    "measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]\n",
    "for day in range(historyDays):\n",
    "    print('Day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        \n",
    "        qualityMetric = {\n",
    "            'metric_type': 'quality',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': model_uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': {\n",
    "                'quality': measurements[day],\n",
    "                'threshold': 0.7,\n",
    "                'metrics': [\n",
    "                    {\n",
    "                        'name': 'auroc',\n",
    "                        'value': measurements[day],\n",
    "                        'threshold': 0.7\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        }\n",
    "\n",
    "        response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers)\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical performance metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_data = {\n",
    "    'grant_type': 'urn:ibm:params:oauth:grant-type:apikey',\n",
    "    'response_type': 'cloud_iam',\n",
    "    'apikey': AIOS_CREDENTIALS['apikey']\n",
    "}\n",
    "\n",
    "response = requests.post('https://iam.bluemix.net/identity/token', data=token_data)\n",
    "iam_token = response.json()['access_token']\n",
    "iam_headers = {\n",
    "    'Content-Type': 'application/json',\n",
    "    'Authorization': 'Bearer %s' % iam_token\n",
    "}\n",
    "\n",
    "for day in range(historyDays):\n",
    "    print('Day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "        score_count = random.randint(60, 600)\n",
    "        score_resp = random.uniform(60, 300)\n",
    "\n",
    "        performanceMetric = {\n",
    "            'metric_type': 'performance',\n",
    "            'binding_id': binding_uid,\n",
    "            'timestamp': score_time,\n",
    "            'subscription_id': model_uid,\n",
    "            'asset_revision': model_uid,\n",
    "            'deployment_id': deployment_uid,\n",
    "            'value': {\n",
    "                'response_time': score_resp,\n",
    "                'records': score_count\n",
    "            }\n",
    "        }\n",
    "\n",
    "        response = requests.post(performance_metrics_url, json=[performanceMetric], headers=iam_headers)\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Explainability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ibm_ai_openscale.supporting_classes import *\n",
    "subscription.explainability.enable(\n",
    "    problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "            categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"],\n",
    "            label_column='predictedLabel'\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.explainability.get_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run fairness monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kick off a fairness monitor run on current data. Depending on how fast the monitor runs, the table may not contain the most recent results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run_details = subscription.fairness_monitoring.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscription.fairness_monitoring.show_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional data to help debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print('Datamart:', data_mart_id)\n",
    "print('Model:', model_uid)\n",
    "print('Deployment:', deployment_uid)\n",
    "print('Binding:', binding_uid)\n",
    "print('Scoring URL:', credit_risk_scoring_endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription.payload_logging.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "subscription.payload_logging.print_table_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up continuous learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if DB2_CREDENTIALS is not None:\n",
    "    accuracy_details = subscription.quality_monitoring.get_details()\n",
    "    feedback_data_reference = {\n",
    "        \"name\": \"DRUG feedback\",\n",
    "        \"connection\": DB2_CREDENTIALS,\n",
    "        \"source\": {\n",
    "            \"tablename\": accuracy_details['parameters']['feedback_data_reference']['location']['table_name'],\n",
    "            \"type\": \"dashdb\"\n",
    "        }\n",
    "    }\n",
    "    print(feedback_data_reference)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if DB2_CREDENTIALS is not None and SPARK_CREDENTIALS is not None:\n",
    "    system_config = {\n",
    "        wml_client.learning_system.ConfigurationMetaNames.FEEDBACK_DATA_REFERENCE: feedback_data_reference,\n",
    "        wml_client.learning_system.ConfigurationMetaNames.MIN_FEEDBACK_DATA_SIZE: 50,\n",
    "        wml_client.learning_system.ConfigurationMetaNames.SPARK_REFERENCE: SPARK_CREDENTIALS,\n",
    "        wml_client.learning_system.ConfigurationMetaNames.AUTO_RETRAIN: \"conditionally\",\n",
    "        wml_client.learning_system.ConfigurationMetaNames.AUTO_REDEPLOY: \"always\"\n",
    "    }\n",
    "\n",
    "    wml_client.learning_system.setup(model_uid=model_uid, meta_props=system_config)\n",
    "    wml_client.learning_system.get_details(model_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feed in 100 rows of feedback data via OpenScale API to simulate input from apps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!rm additional_feedback_data.json\n",
    "!wget https://raw.githubusercontent.com/emartensibm/german-credit/master/additional_feedback_data.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open('additional_feedback_data.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)\n",
    "subscription.feedback_logging.store(additional_feedback_data['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below will kick off a learning system evaluation, retrain, and redeploy of the model. You may un-comment this line and run it if you would prefer not to perform this portion of the demo in Watson Studio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# wml_client.learning_system.run(model_uid, asynchronous=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify transactions for Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "payload_table_name = 'Payload_{0}'.format(model_uid)\n",
    "# SELECT * FROM \"{0}\".\"{1}\" LIMIT 1;\n",
    "query = \"\"\"\n",
    "SELECT \"scoring_id\" FROM \"{0}\" WHERE \"predictedLabel\"='No Risk' LIMIT 1;\n",
    "\"\"\".format(payload_table_name)\n",
    "stmt = ibm_db.exec_immediate(conn, query)\n",
    "result = ibm_db.fetch_assoc(stmt)\n",
    "print(\"Transaction with prediction of No Risk:\", result['scoring_id'])\n",
    "\n",
    "query = \"\"\"\n",
    "SELECT \"scoring_id\" FROM \"{0}\" WHERE \"predictedLabel\"='Risk' LIMIT 1;\n",
    "\"\"\".format(payload_table_name)\n",
    "stmt = ibm_db.exec_immediate(conn, query)\n",
    "result = ibm_db.fetch_assoc(stmt)\n",
    "print(\"Transaction with prediction of Risk:\", result['scoring_id'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.5",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}