notebook_examples/big_data_service-(BDS)-kerberos.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "a1515f0f",
   "metadata": {},
   "source": [
    "@notebook{big_data_service-(BDS)-kerberos.ipynb,\n",
    "    title: Connect to Oracle Big Data Service,\n",
    "    summary: Connect to Oracle Big Data services using Kerberos.,\n",
    "    developed_on: pyspark30_p37_cpu_v5,\n",
    "    keywords: kerberos, big data service, bds,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6627d58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads kerberos"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4296a3ce",
   "metadata": {},
   "source": [
    "<font color=gray>Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc.  All rights reserved.\n",
    "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
    "</font>\n",
    "\n",
    "***\n",
    "# <font color=red>Connect to Oracle Big Data Service</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=teal> Oracle Cloud Infrastructure Data Science Service Team </font></p>\n",
    "\n",
    "***\n",
    "\n",
    "# Overview:\n",
    "\n",
    "The Oracle Big Data Service ([BDS](https://docs.oracle.com/en-us/iaas/Content/bigdata/home.htm)) is an Oracle Cloud Infrastructure (OCI) service that is designed for big data use cases and supports Hadoop and Spark. BDS has features such as HDFS and Hive. You can use BDS for short-lived clusters used to tackle specific tasks, and long-lived clusters that manage large data lakes. To connect to BDS from a notebook session, the cluster must have Kerberos enabled. This notebook demonstrates how to configure Kerberos authentication using ADS.\n",
    "\n",
    "Compatible conda pack: [PySpark 3.0 and Data Flow](https://docs.oracle.com/en-us/iaas/data-science/using/conda-pyspark-fam.htm) for CPU on Python 3.7 (version 5.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "* <a href='#introduction'>Introduction</a>\n",
    "* <a href='#setup'>Setup</a>\n",
    "* <a href='#BDSSecretKeeper'>BDSSecretKeeper</a>\n",
    "* <a href='#save'>Save to Vault</a>\n",
    "* <a href='#load'>Load from Vault</a>\n",
    "* <a href='#connect'>Connect to Resources</a>\n",
    "    * <a href='#connect_hdfs'>HDFS</a>\n",
    "    * <a href='#connect_hive'>Hive</a>\n",
    "* <a href='#clean-up'>Clean Up</a>\n",
    "* <a href=\"#ref\">References</a>\n",
    "\n",
    "---\n",
    "\n",
    "Datasets are provided as a convenience. Datasets are considered third-party content and are not considered materials under your agreement with Oracle.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9367374b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads\n",
    "import fsspec\n",
    "import os\n",
    "\n",
    "from ads.bds.auth import refresh_ticket\n",
    "from ads.secrets.big_data_service import BDSSecretKeeper\n",
    "from oci.config import from_file\n",
    "from oci.vault import VaultsClient\n",
    "from oci.vault.models import ScheduleSecretDeletionDetails\n",
    "from impala.dbapi import connect\n",
    "\n",
    "ads.set_auth(\"resource_principal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2040aea3",
   "metadata": {},
   "source": [
    "<a id='introduction'></a>\n",
    "# Introduction\n",
    "\n",
    "As a network security protocol, Kerberos authenticates requests between hosts. A trusted third-party and secret-key cryptography are used to perform the authentication. The protocol has three main entities, client, server, and the Key Distribution Center (KDC). The notebook is the client, and the server is the BDS. The role of KDC is to perform authentication and grant tickets. The tickets allow the client and server to verify identities.\n",
    "\n",
    "At a high level, Kerberos works by having the client make a request to authenticate. The KDC verifies the client's credentials and a number of messages are sent back and forth between the client, server and KDC. Ultimately, the KDC creates a service session key that is shared between the client and the server. This session key is used in the communication that follows.\n",
    "\n",
    "Kerberos requires `keytab` and `krb5.conf` files. These are specific to each cluster and are obtained from the master node on the BDS cluster.\n",
    "\n",
    "<a id='setup'></a>\n",
    "# Setup\n",
    "\n",
    "The best practice is to store the Kerberos credentials and configuration parameters in the vault. By doing this, access can be limited and auditing is supported. Further, it allows for the credentials to be rotated and updated in a single place. You don't have to update each notebook session because they pull the configuration parameters and Kerberos files from the vault as needed. ADS provides the `BDSSecretKeeper` class that makes connecting to BDS, Hive, and HDFS simple and secure. \n",
    "\n",
    "You must obtain two files from the master node on the BDS cluster. The keytab file and `krb5.conf` files must be stored on the block volume in the notebook session. The `krb5.conf` file is in the `/etc` directory of the BDS master node. If you're using a vault to store the credentials, these files are stored in the secure and convenient vault.\n",
    "\n",
    "For Kerberos authentication, you provide the following values:\n",
    "\n",
    "- `kerb5_path`: The local path to the `krb5.conf` configuration file.\n",
    "- `keytab_path`: The local path to the principal's Keytab file.\n",
    "- `principal`: The unique identity to which Kerberos can assign tickets.\n",
    "\n",
    "To connect to the BDS cluster's HDFS you need the hostname and the port. These aren't strictly needed to access BDS, but most data science workflows require at least occasional access:\n",
    "\n",
    "- `hdfs_host`: The HDFS hostname to use to connect to the HDFS file system on the BDS cluster.\n",
    "- `hdfs_port`: The HDFS port to use to connect to the HDFS file system on the BDS cluster.\n",
    "\n",
    "BDS provides access to Hive. To connect to Hive you need the hostname and the port:\n",
    "\n",
    "- `hive_host`: The Hive hostname for the BDS cluster.\n",
    "- `hive_port`: The Hive port for the BDS cluster.\n",
    "\n",
    "To save the configuration parameters in the vault, you provide the OCIDs of the vault and the master encryption key:\n",
    "\n",
    "- `vault_id` = OCID of the vault.\n",
    "- `key_id` = OCID of the master encryption key.\n",
    "\n",
    "If the configuration parameters are already stored in the vault, all you need is the secret OCID that has the necessary information. It's safe to hardcode the secret OCID into a notebook because OCI IAM manages access.\n",
    "\n",
    "In the following cell, update the values to use to create your secret:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f9e41b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Kerberos Authentication\n",
    "kerb5_path = \"<kerb5_path>\"\n",
    "keytab_path = \"<keytab_file_path>\"\n",
    "principal = \"<principal>\"\n",
    "\n",
    "# HDFS configuration\n",
    "hdfs_host = \"<hdfs_host>\"\n",
    "hdfs_port = \"<hdfs_port>\"\n",
    "\n",
    "# HIVE configuration\n",
    "hive_host = \"<hive_host>\"\n",
    "hive_port = \"<hive_port>\"\n",
    "\n",
    "# Vault OCIDs\n",
    "vault_id = \"<vault_id>\"\n",
    "key_id = \"<key_id>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bc2e614",
   "metadata": {},
   "source": [
    "<a id='BDSSecretKeeper'></a>\n",
    "# BDSSecretKeeper\n",
    "\n",
    "ADS provides the `BDSSecretKeeper` class to manage your BDS secrets. It's used to store the configuration parameters, including the `keytab` and `krb5.conf` files, in the vault. Use the `BDSSecretKeeper` and Kerberos context manager, `krbcontext()` to manage the connection to BDS, HDFS, and Hive.\n",
    "\n",
    "The following cell creates a `BDSSecretKeeper` object, which contains the configuration parameters to connect with BDS and its services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e44aab73",
   "metadata": {},
   "outputs": [],
   "source": [
    "if vault_id != \"<vault_id>\" and key_id != \"<key_id>\" and principal != \"<principal>\":\n",
    "    secret = BDSSecretKeeper(\n",
    "        vault_id=vault_id,\n",
    "        key_id=key_id,\n",
    "        principal=principal,\n",
    "        hdfs_host=hdfs_host if hdfs_host != \"<hdfs_host>\" else None,\n",
    "        hive_host=hive_host if hive_host != \"<hive_host>\" else None,\n",
    "        hdfs_port=hdfs_port if hdfs_port != \"<hdfs_port>\" else None,\n",
    "        hive_port=hive_port if hive_port != \"<hive_port>\" else None,\n",
    "        keytab_path=keytab_path if keytab_path != \"<keytab_path>\" else None,\n",
    "        kerb5_path=kerb5_path if kerb5_path != \"<kerb5_path>\" else None,\n",
    "    )\n",
    "else:\n",
    "    secret = None\n",
    "    print(\n",
    "        \"BDSSecretKeeper object was not created. Enter configuration values in the Setup section.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79d8fdd5",
   "metadata": {},
   "source": [
    "<a id='save'></a>\n",
    "# Save to Vault\n",
    "\n",
    "The `.save()` method on a `BDSSecretKeeper` object stores the configuration parameters in the vault. You can provide metadata such as a name, description, and tags. The contents of the `keytab` file and `krb5.conf` files are saved if the `save_files` parameter is set to `True`. The best practice is to store these files in the vault.\n",
    "\n",
    "The following cell stores the BDS configuration parameters in the vault."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58010e1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "if secret is not None:\n",
    "    saved_secret = secret.save(\n",
    "        name=\"Sample_BDS_Secret_1\",\n",
    "        description=\"Demo BDSSecretKeeper secret to connect to BDS resourses\",\n",
    "        freeform_tags={\"schema\": \"BDSSecretKeeper\"},\n",
    "        save_files=True,\n",
    "    )\n",
    "    print(f\"Secret OCID: {saved_secret.secret_id}\")\n",
    "else:\n",
    "    saved_secret = None\n",
    "    print(\n",
    "        \"No secrets were saved to the Vault. Enter configuration values in the Setup section.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c502c4c",
   "metadata": {},
   "source": [
    "<a id='load'></a>\n",
    "# Load from Vault\n",
    "\n",
    "The `.load_secret()` method in the `BDSSecretKeeper` class accepts a secret's OCID, and returns the contents of the vault. The default behavior is to save the `keytab` and `krb5.conf` files to the `keytab_path` and `kerb5_path` paths that are specified in the secret. You can change this by passing updated values to the `keytab_path` and `kerb5_path` parameters. Any parameter that can be passed to the `BDSSecretKeeper` constructor can be used in `.load_secret()` to override the values returned from the vault.\n",
    "\n",
    "The `keytab file` and `krb5.conf` file can only be saved to the local block storage if they were saved in the vault.\n",
    "\n",
    "Once the configuration parameters have been obtained and the files are saved, the Kerberos context manager (`.krbcontext()`) is used to create the Kerberos ticket.\n",
    "\n",
    "In the following cell, the `.load_secret()` method obtains the configuration parameters from the vault and setups the `keytab` and `krb5.conf` files. The call to `krbcontext()` obtains a Kerberos ticket. The Kerberos ticket isn't returned from the `krbcontext()` because it's managed by a process running in the notebook session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c72f536",
   "metadata": {},
   "outputs": [],
   "source": [
    "if saved_secret is not None:\n",
    "    with BDSSecretKeeper.load_secret(saved_secret.secret_id) as config:\n",
    "        refresh_ticket(principal=config[\"principal\"], keytab_path=config[\"keytab_path\"])\n",
    "        hdfs_config = {\n",
    "            \"host\": config[\"hdfs_host\"],\n",
    "            \"port\": config[\"hdfs_port\"],\n",
    "            \"protocol\": \"webhdfs\",\n",
    "            \"kerberos\": True,\n",
    "        }\n",
    "        hive_config = {\n",
    "            \"host\": config[\"hive_host\"],\n",
    "            \"port\": config[\"hive_port\"],\n",
    "        }\n",
    "else:\n",
    "    config = None\n",
    "    print(\n",
    "        \"No secret was returned from the Vault. Enter configuration values in the Setup section.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cd1f018",
   "metadata": {},
   "source": [
    "<a id='connect'></a>\n",
    "# Connect to Resources\n",
    "\n",
    "A successful call to `krbcontext` enables to you to connect to resources such as Hive and HDFS.\n",
    "\n",
    "<a id='connect_hdfs'></a>\n",
    "## HDFS\n",
    "\n",
    "WebHDFS protocol was developed to allow an external application, such as a notebook session to access or manage files in the HDFS in BDS. It's based on an industry-standard RESTful mechanism that doesn't require Java binding. It works with operations such as reading files, writing to files, making directories, changing permissions, and renaming. It defines a public HTTP REST API that permits clients to access HDFS over the web. Clients can use common tools such as curl and wget to access the [HDFS](https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#Document+Conventions).\n",
    "\n",
    "Data scientists commonly use `fsspec` to perform file-level operations on HDFS. To obtain a handle for the HDFS file system over WebHDFS, use the `fsspec.filesystem()` method. This method accepts a dictionary that contains the HDFS hostname, and port. It must also specify that the protocol is WebHDFS. To connect to BDS, Kerberos must be enabled. The following cell defines the connection parameters, and obtains a file system handle. Use the `fs` file system handle to list the files in the root directory with `fs.ls(\"/\")`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ba66d50",
   "metadata": {},
   "outputs": [],
   "source": [
    "if (\n",
    "    hdfs_config is not None\n",
    "    and hdfs_config[\"host\"] is not None\n",
    "    and hdfs_config[\"port\"] is not None\n",
    "):\n",
    "    fs = fsspec.filesystem(**hdfs_config)\n",
    "    print(fs.ls(\"/\"))\n",
    "else:\n",
    "    fs = None\n",
    "    print(\n",
    "        \"No connection was made to the HDFS file system. Enter configuration values in the Setup section.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ca3af7c",
   "metadata": {},
   "source": [
    "<a id='connect_hive'></a>\n",
    "## Hive\n",
    "\n",
    "Apache Hive is a data warehouse system that runs on top of Hadoop. It allows you to perform SQL-like on your data using the HGL language. The `.connect()` method on a `impala.dbapi` object makes a connection. This connection needs the Hive hostname and port. For secure BDS, the authentication must be set to `GSSAPI`, and the service name is set to 'hive'.\n",
    "\n",
    "A cursor is generally used to work with HQL. The `.execute()` method allows for HQL commands to be run on Hive. For example, this command uses a Hive cursor to get a copy of all the data in the fictitious `my_database` database.\n",
    "\n",
    "``` python\n",
    "cursor.execute(\"SELECT * FROM my_database\")\n",
    "```\n",
    "\n",
    "To materialize the results and convert it to a Pandas dataframe, use the following command:\n",
    "\n",
    "``` python\n",
    "df = pd.DataFrame(cursor.fetchall(), columns=[col[0] for col in cursor.description])\n",
    "```\n",
    "\n",
    "The following cell makes a connection to Hive and returns a cursor, which is used to list the databases in the Hive cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "942c1108",
   "metadata": {},
   "outputs": [],
   "source": [
    "if (\n",
    "    hive_config is not None\n",
    "    and hive_config[\"host\"] is not None\n",
    "    and hive_config[\"port\"] is not None\n",
    "):\n",
    "    cursor = connect(\n",
    "        host=hive_config[\"host\"],\n",
    "        port=hive_config[\"port\"],\n",
    "        auth_mechanism=\"GSSAPI\",\n",
    "        kerberos_service_name=\"hive\",\n",
    "    ).cursor()\n",
    "    cursor.execute(\"SHOW DATABASES\")\n",
    "    print(cursor.fetchall())\n",
    "else:\n",
    "    cursor = None\n",
    "    print(\n",
    "        \"No connection was made to Hive. Enter configuration values in the Setup section.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec228e5b",
   "metadata": {},
   "source": [
    "<a id='clean-up'></a>\n",
    "# Clean Up\n",
    "\n",
    "The following code removes the `keytab` and `krb5.conf` files from local storage. It also deletes the secret from the vault, and drops connections to HDFS and Hive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cf46b07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Disconnect from Hive\n",
    "if cursor is not None:\n",
    "    cursor.close()\n",
    "\n",
    "# Remove the Keytab file\n",
    "if keytab_path is not None and os.path.exists(keytab_path):\n",
    "    os.remove(keytab_path)\n",
    "\n",
    "# Remove the `krb5.conf` file\n",
    "if kerb5_path is not None and os.path.exists(kerb5_path):\n",
    "    os.remove(kerb5_path)\n",
    "\n",
    "# Delete the secret\n",
    "if saved_secret is not None:\n",
    "    oci_config = from_file(\n",
    "        os.path.join(os.path.expanduser(\"~\"), \".oci\", \"config\"), \"DEFAULT\"\n",
    "    )\n",
    "    try:\n",
    "        VaultsClient(oci_config).schedule_secret_deletion(\n",
    "            saved_secret.secret_id, ScheduleSecretDeletionDetails()\n",
    "        )\n",
    "    except:\n",
    "        print(\"The secret has already been deleted.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a79f3e9",
   "metadata": {},
   "source": [
    "<a id=\"ref\"></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}