Update example notebook

lsst-sqre · May 17, 2024 · 44ddf45 · 44ddf45
1 parent 4544107
commit 44ddf45
Showing 1 changed file with 237 additions and 0 deletions.
diff --git a/examples/RestProxyAPIExample.ipynb b/examples/RestProxyAPIExample.ipynb
@@ -0,0 +1,237 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "51d7ed48-cb32-431d-b4b4-27e2d5d7437b",
+   "metadata": {},
+   "source": [
+    "# Sasquatch REST Proxy API\n",
+    "\n",
+    "This notebook implements the example in the [Sasquatch user guide](https://sasquatch.lsst.io/user-guide/restproxy.html) on using the REST Proxy API to send data to Sasquatch, using the data-int cluster as a sandbox area.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8fecfae9-ebc8-4397-b289-94926a3d7d25",
+   "metadata": {},
+   "source": [
+    "### Get the Kafka cluster ID\n",
+    "First thing you need is to get the cluster ID, this is required to create new topics in Kafka."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8f4274eb-e5a3-4721-b0b4-e82cd144867b",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import requests \n",
+    "\n",
+    "sasquatch_rest_proxy_url = \"https://data-int.lsst.cloud/sasquatch-rest-proxy\"\n",
+    "\n",
+    "headers = {\"content-type\": \"application/json\"}\n",
+    "\n",
+    "r = requests.get(f\"{sasquatch_rest_proxy_url}/v3/clusters\", headers=headers)\n",
+    "\n",
+    "cluster_id = r.json()['data'][0]['cluster_id']\n",
+    "\n",
+    "print(cluster_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e2094edb-b9be-4865-bed9-ac59118dfc23",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-06-14T16:21:01.300021Z",
+     "iopub.status.busy": "2023-06-14T16:21:01.299786Z",
+     "iopub.status.idle": "2023-06-14T16:21:01.303710Z",
+     "shell.execute_reply": "2023-06-14T16:21:01.303117Z",
+     "shell.execute_reply.started": "2023-06-14T16:21:01.300004Z"
+    },
+    "tags": []
+   },
+   "source": [
+    "### Create a new Kafka topic\n",
+    "Here we'll use the REST Proxy API on data-int to create a new Kafka topic named `skyFluxMetric` in the `lsst.example` namespace.\n",
+    "Note that if the topic already exists in Kafka the REST Proxy will return an error, but that's ok."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "68331acd-cb4f-4d63-b7de-b59d9d9985c4",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "namespace = \"lsst.example\"\n",
+    "topic_name = \"skyFluxMetric\"\n",
+    "\n",
+    "# The topic is created with one partition and a replication factor of 3 by default, this configuration is fixed for the Sasquatch Kafka cluster.\n",
+    "topic_config = {  \n",
+    "        \"topic_name\": f\"{namespace}.{topic_name}\",\n",
+    "        \"partitions_count\": 1,\n",
+    "        \"replication_factor\": 3\n",
+    "    }\n",
+    "\n",
+    "headers = {\"content-type\": \"application/json\"}\n",
+    "\n",
+    "response = requests.post(f\"{sasquatch_rest_proxy_url}/v3/clusters/{cluster_id}/topics\", json=topic_config, headers=headers)\n",
+    "print(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d015c77b-4721-4275-864e-f0ba12d30387",
+   "metadata": {},
+   "source": [
+    "### Send data\n",
+    "Now we'll create a payload with the schema and two records based on the example in https://sasquatch.lsst.io/user-guide/restproxy.html#kafka-rest-proxy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f275ffc4-ee8d-4e9c-b124-80d9bcdcbbaf",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "value_schema = json.dumps({\"namespace\": f\"{namespace}\", \n",
+    "                           \"type\": \"record\", \n",
+    "                           \"name\": f\"{topic_name}\", \n",
+    "                           \"fields\": [{\"name\": \"timestamp\", \"type\": \"long\"}, \n",
+    "                                      {\"name\": \"band\", \"type\": \"string\"}, \n",
+    "                                      {\"name\": \"instrument\", \"type\": \"string\", \"default\": \"LSSTCam-imSim\"}, \n",
+    "                                      {\"name\": \"meanSky\",\"type\": \"float\"}, \n",
+    "                                      {\"name\": \"stdevSky\",\"type\": \"float\"}]\n",
+    "                          })\n",
+    "\n",
+    "payload = {\n",
+    "    \"value_schema\": value_schema,\n",
+    "    \"records\": [\n",
+    "        {\n",
+    "            \"value\": {\n",
+    "                \"timestamp\": 1715967558000000,\n",
+    "                \"band\": \"y\",\n",
+    "                \"instrument\": \"LSSTCam-imSim\",\n",
+    "                \"meanSky\": -213.75839364883444,\n",
+    "                \"stdevSky\": 2328.906118708811,\n",
+    "            }\n",
+    "        },\n",
+    "        {\n",
+    "            \"value\": {\n",
+    "                \"timestamp\": 1715967559000000,\n",
+    "                \"band\": \"y\",\n",
+    "                \"instrument\": \"LSSTCam-imSim\",\n",
+    "                \"meanSky\": -213.75839364883444,\n",
+    "                \"stdevSky\": 2328.906118708811,\n",
+    "            }\n",
+    "        }\n",
+    "    ],\n",
+    "}\n",
+    "payload"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f53d1fb-2406-496d-a5b0-99ce701248a0",
+   "metadata": {},
+   "source": [
+    "This is the URL for sending data to our topic\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bec461e1-cf1b-43bc-92e8-c7c32441fefb",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "\n",
+    "url = f\"{sasquatch_rest_proxy_url}/topics/{namespace}.{topic_name}\"\n",
+    "url"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "620c56db-d4d2-4581-81b6-e6d88933582f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2024-05-17T17:55:59.193919Z",
+     "iopub.status.busy": "2024-05-17T17:55:59.193437Z",
+     "iopub.status.idle": "2024-05-17T17:55:59.199253Z",
+     "shell.execute_reply": "2024-05-17T17:55:59.198713Z",
+     "shell.execute_reply.started": "2024-05-17T17:55:59.193902Z"
+    }
+   },
+   "source": [
+    "Finally, make a POST request to send data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "340ab7c0-1b0f-4ba0-8270-39345c6f188e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "headers = {\n",
+    "    \"Content-Type\": \"application/vnd.kafka.avro.v2+json\",\n",
+    "    \"Accept\": \"application/vnd.kafka.v2+json\",\n",
+    "}\n",
+    "\n",
+    "response = requests.request(\"POST\", url, json=payload, headers=headers)\n",
+    "\n",
+    "print(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01d63903-1766-4034-88ca-51593bbf09f4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2024-05-17T17:57:50.781582Z",
+     "iopub.status.busy": "2024-05-17T17:57:50.781067Z",
+     "iopub.status.idle": "2024-05-17T17:57:50.784429Z",
+     "shell.execute_reply": "2024-05-17T17:57:50.783976Z",
+     "shell.execute_reply.started": "2024-05-17T17:57:50.781565Z"
+    }
+   },
+   "source": [
+    "You can inspect the data in Kafka using the Kafdrop UI https://data-int.lsst.cloud/kafdrop/topic/lsst.example.skyFluxMetric and also in InlfuxDB using the Chronograf UI https://data-int.lsst.cloud/chronograf"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "LSST",
+   "language": "python",
+   "name": "lsst"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}