diff --git a/notebooks/official/feature_store/feature_monitoring_with_feature_registry.ipynb b/notebooks/official/feature_store/feature_monitoring_with_feature_registry.ipynb new file mode 100644 index 000000000..76a28e15a --- /dev/null +++ b/notebooks/official/feature_store/feature_monitoring_with_feature_registry.ipynb @@ -0,0 +1,953 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e" + }, + "source": [ + "## Feature Monitoring in Vertex AI Feature Store\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Open in Colab\n", + " \n", + " 		\n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Workbench\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "In this tutorial, you will learn how to use the Vertex AI SDK for Python to monitor feature data in Vertex AI Feature Store\n", + "\n", + "This tutorial uses the following Google Cloud ML services and resources:\n", + "\n", + "* Vertex AI Feature Store\n", + "* BigQuery\n", + "\n", + "The steps performed include the following:\n", + "\n", + "* Setup BigQuery data\n", + "* Setup Feature Registry\n", + "* Setup FeatureMonitors, execute FeatureMonitorJobs to observe feature stats and detect drift.\n", + "* Clean up" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "61RBz8LLbxCR" + }, + "source": [ + "## Get started" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "No17Cw5hgx12" + }, + "source": [ + "### Install Vertex AI SDK for Python and other required packages\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tFy3H3aPgx12" + }, + "outputs": [], + "source": [ + "! pip3 install --upgrade --quiet google-cloud-aiplatform bigframes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R5Xep4W9lq-Z" + }, + "source": [ + "### Restart runtime (Colab only)\n", + "\n", + "To use the newly installed packages, you must restart the runtime on Google Colab." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XRvKdaPDTznN" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "da30060e7a7d" + }, + "source": [ + "### Restart kernel (Workbench only)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SbmM4z7FOBpM" + }, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dmWOrTJ3gx13" + }, + "source": [ + "### Authenticate your notebook environment (Colab only)\n", + "\n", + "Authenticate your environment on Google Colab.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NyKGtVQjgx13" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cwr2lW2wo0G0" + }, + "source": [ + "### Grant instance service account permissions (Workbench only)\n", + "\n", + "Grant your workbench instance owner (in format of xxx-compute@developer.gserviceaccount.com) following IAM permissions:\n", + "* Bigquery Admin\n", + "* Vertex AI Feature Store Admin\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DF4l8DTdWgPY" + }, + "source": [ + "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n", + "\n", + "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Nqwi-5ufWp_B" + }, + "outputs": [], + "source": [ + "# change to your own project id\n", + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}\n", + "\n", + "import vertexai\n", + "\n", + "vertexai.init(project=PROJECT_ID, location=LOCATION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "33067053f38b" + }, + "source": [ + "### Imports and IDs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c8abe818393b" + }, + "outputs": [], + "source": [ + "import bigframes\n", + "import bigframes.pandas\n", + "import pandas as pd\n", + "from google.cloud import bigquery\n", + "from vertexai.resources.preview.feature_store import (Feature, FeatureGroup,\n", + " FeatureMonitor)\n", + "from vertexai.resources.preview.feature_store import utils as fs_utils" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4d0295f5d524" + }, + "source": [ + "The following variables set BigQuery and Feature Group resources that will be\n", + "used or created. If you'd like to use your own data source (CSV), please adjust\n", + "`DATA_SOURCE`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ac036ecfbc32" + }, + "outputs": [], + "source": [ + "BQ_DATASET_ID = \"fhfv_dataset_unique\" # @param {type:\"string\"}\n", + "BQ_TABLE_ID = \"fhfv_table_unique\" # @param {type:\"string\"}\n", + "BQ_TABLE_URI = f\"{PROJECT_ID}.{BQ_DATASET_ID}.{BQ_TABLE_ID}\"\n", + "\n", + "FEATURE_GROUP_ID = \"fg_feature_monitoring_tutorial\" # @param {type:\"string\"}\n", + "\n", + "DATA_SOURCE = \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cd580a0679ce" + }, + "source": [ + "## Create BigQuery table containing feature data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1e2c688b844b" + }, + "source": [ + "First we'll use BigQuery DataFrames to load in our CSV data source. Then we'll\n", + "rename the `timestamp` column to `feature_timestamp` to support usage as a\n", + "BigQuery source in Feature Registry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1ff4481243a8" + }, + "outputs": [], + "source": [ + "session = bigframes.connect(\n", + " bigframes.BigQueryOptions(\n", + " project=PROJECT_ID,\n", + " location=LOCATION,\n", + " )\n", + ")\n", + "df = session.read_csv(DATA_SOURCE)\n", + "df[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"], utc=True)\n", + "df = df.rename(columns={\"timestamp\": \"feature_timestamp\"})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "de5008f71567" + }, + "source": [ + "Let's preview the data we'll write to the table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "38b448c47657" + }, + "outputs": [], + "source": [ + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "967ec2a0193f" + }, + "source": [ + "And finally we'll write the DataFrame to the target BigQuery table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4c11b88ab55d" + }, + "outputs": [], + "source": [ + "df.to_gbq(BQ_TABLE_URI, if_exists=\"replace\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "818a36b9da86" + }, + "source": [ + "## Create feature registry resources" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2fd96d0d8628" + }, + "source": [ + "Create a feature group backed by the BigQuery table created above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1f915ddd4669" + }, + "outputs": [], + "source": [ + "fg: FeatureGroup = FeatureGroup.create(\n", + " f\"{FEATURE_GROUP_ID}\",\n", + " fs_utils.FeatureGroupBigQuerySource(\n", + " uri=f\"bq://{BQ_TABLE_URI}\", entity_id_columns=[\"users\"]\n", + " ),\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "s5usvn7_7L5I" + }, + "outputs": [], + "source": [ + "# For existing FeatureGroup, get by passing FEATURE_GROUP_ID\n", + "fg = FeatureGroup(f\"{FEATURE_GROUP_ID}\")\n", + "print(fg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ebb9179572f7" + }, + "source": [ + "Create the `movies` feature which corresponds to the `movies` column in the\n", + "recently created BigQuery table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0dba1c02883c" + }, + "outputs": [], + "source": [ + "movies_feature: Feature = fg.create_feature(\"movies\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a5xBNTA5xQQJ" + }, + "source": [ + "## Setup Feature Monitoring" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hK12BaQ4xaAU" + }, + "source": [ + "### Create Feature Monitor" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7g_Urc4ixoX6" + }, + "outputs": [], + "source": [ + "FEATURE_MONITOR_ID = \"vertex_sdk_fm_cron\" # @param {type:\"string\"}\n", + "fm: FeatureMonitor = fg.create_feature_monitor(\n", + " name=FEATURE_MONITOR_ID,\n", + " feature_selection_configs=[(\"movies\", 0.1)],\n", + " schedule_config=\"0 * * * *\", # Default schedule (hourly)\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MPpOdxBwXYu-" + }, + "source": [ + "List Feature Monitors created in the Feature Group" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "YE6Lz8Y-6vML" + }, + "outputs": [], + "source": [ + "fms: list[FeatureMonitor] = fg.list_feature_monitors()\n", + "print(fms)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dDbI0sgTW1Kh" + }, + "source": [ + "Get FeatureMonitor and it's properties" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XCQEDo4JG-06" + }, + "outputs": [], + "source": [ + "fm = fg.get_feature_monitor(FEATURE_MONITOR_ID)\n", + "print(fm)\n", + "print(\n", + " \"feature selection configs: (feature and it's drift threshold):\",\n", + " fm.feature_selection_configs,\n", + ")\n", + "print(\"schedule config in cron string: \", fm.schedule_config)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bwGyutQfFE8U" + }, + "source": [ + "### Execute a FeatureMonitorJob\n", + "\n", + "FeatureMonitorJob will be executed in two ways:\n", + "1. Automatically executed in scheduled time set the schedule_config in FeatureMonitor.\n", + "2. Manually trigger. In the following sections we will manually trigger monitor job to observe stats and drifts.\n", + "\n", + "Stats are generated on the snapshot of the data in FeatureMonitorJob execution." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s_s2w_a3kb35" + }, + "source": [ + "Manually execute FeatureMonitorJob as following" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Emr9vY7GpHod" + }, + "outputs": [], + "source": [ + "fmj = fm.create_feature_monitor_job()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "syKlCc1SFPb-" + }, + "outputs": [], + "source": [ + "print(fmj)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7k2MQ2anGb29" + }, + "source": [ + "#### Observe Feature Stats in FeatureMonitorJob" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pe7LKlStTRuP" + }, + "source": [ + "Get Feature Monitor Job and observe the feature_stats_and_anomalies. feature_stats refers to tensor flow proto [FeatureNameStatistics](https://www.tensorflow.org/tfx/tf_metadata/api_docs/python/tfmd/proto/statistics_pb2/FeatureNameStatistics)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OEcGwcuckpYZ" + }, + "outputs": [], + "source": [ + "# Note: if feature_stats_and_anomalies not shown, wait for a few seconds to minutes then retry\n", + "import time\n", + "\n", + "while True:\n", + " fmj_get = fm.get_feature_monitor_job(fmj.name)\n", + " if (\n", + " fmj_get.feature_stats_and_anomalies is None\n", + " or len(fmj_get.feature_stats_and_anomalies) == 0\n", + " ):\n", + " time.sleep(5)\n", + " else:\n", + " break\n", + "print(fmj_get)\n", + "print(fmj_get.feature_stats_and_anomalies)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "96JVk7MloQB0" + }, + "source": [ + "At this time, only one job executed, no drift detected." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RcM3uebBn6hY" + }, + "outputs": [], + "source": [ + "for feature_stats_and_anomalies in fmj_get.feature_stats_and_anomalies:\n", + " print(\"feature: \", feature_stats_and_anomalies.feature_id)\n", + " print(\"drift score: \", feature_stats_and_anomalies.distribution_deviation)\n", + " print(\"drift detected: \", feature_stats_and_anomalies.drift_detected)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Kei_nO6iGv4Y" + }, + "source": [ + "#### Get Feature Stats in Feature" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CwokTiZCG3QH" + }, + "outputs": [], + "source": [ + "feature_movie = fg.get_feature(\"movies\", latest_stats_count=5)\n", + "print(feature_movie)\n", + "\n", + "# At this time, only one job executed, no drift detected.\n", + "for feature_stats in feature_movie.feature_stats_and_anomalies:\n", + " print(\"feature monitor job id: \", feature_stats.feature_monitor_job_id)\n", + " print(\"drift score: \", feature_stats.distribution_deviation)\n", + " print(\"drift detected: \", feature_stats.drift_detected)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iRSpTmn4k1Kn" + }, + "source": [ + "Full feature_stats_and_anomalies in feature" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "hcPBOPDnkM_v" + }, + "outputs": [], + "source": [ + "print(feature_movie.feature_stats_and_anomalies)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Hkbe7r0lHdxS" + }, + "source": [ + "### Detect drift" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8u4yzoSDHncf" + }, + "source": [ + "Drifts happen when data in Feature Offline Store (BQ Source) changes overtime. Every Feature Monitor job will calculate drift comparing the data snapshot in the new job with the data snapshot in last job.\n", + "\n", + "Algorithm to calculate drift score:\n", + "* For Categorical type: [L-infinity](https://en.wikipedia.org/wiki/Chebyshev_distance) distance.\n", + "* For Numerical type: [Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence)\n", + "\n", + "In this tutorial, append additional data to the BQ table to simulate the data changes.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-IKohSlrKIm6" + }, + "outputs": [], + "source": [ + "from io import StringIO\n", + "\n", + "data = \"\"\"users,movies,timestamp\n", + "\"new_1\",\"action_1\",2024-08-15T08:28:14Z\n", + "\"new_2\",\"drama_2\",2024-09-15T08:28:14Z\n", + "\"new_3\",\"romance_3\",2024-10-15T08:28:14Z\n", + "\"new_4\",\"science_fiction_4\",2024-11-15T09:29:16Z\n", + "\"new_5\",\"comedy_5\",2024-12-11T07:27:19Z\n", + "\"\"\"\n", + "\n", + "# Read the data into a pandas DataFrame\n", + "df_new = session.read_csv(StringIO(data))\n", + "df_new[\"timestamp\"] = pd.to_datetime(df_new[\"timestamp\"], utc=True)\n", + "df_new = df_new.rename(columns={\"timestamp\": \"feature_timestamp\"})\n", + "df_new.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Afd0JhjvLE2w" + }, + "outputs": [], + "source": [ + "# Append new data to the Bigquery table\n", + "df_new.to_gbq(BQ_TABLE_URI, if_exists=\"append\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "yiQTaXFu2keq" + }, + "outputs": [], + "source": [ + "fmj_new = fm.create_feature_monitor_job(description=\"new job test drift detection\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5mDKU5atocEU" + }, + "source": [ + "List FeatureMonitorJobs, all jobs including the new one are shown" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LpA2a7_nMrNR" + }, + "outputs": [], + "source": [ + "fmjs = fm.list_feature_monitor_jobs()\n", + "print(fmjs)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MGd-zpELdaho" + }, + "source": [ + "Observe drift in Feature Monitor Job" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BcRebMjLPR-e" + }, + "outputs": [], + "source": [ + "while True:\n", + " fmj_with_drift = fm.get_feature_monitor_job(fmj_new.name)\n", + " if (\n", + " fmj_with_drift.feature_stats_and_anomalies is None\n", + " or len(fmj_with_drift.feature_stats_and_anomalies) == 0\n", + " ):\n", + " time.sleep(5)\n", + " else:\n", + " break\n", + "print(fmj_with_drift)\n", + "for feature_stats_and_anomalies in fmj_with_drift.feature_stats_and_anomalies:\n", + " print(\"feature: \", feature_stats_and_anomalies.feature_id)\n", + " print(\n", + " \"drift score (distribution_deviation): \",\n", + " feature_stats_and_anomalies.distribution_deviation,\n", + " )\n", + " print(\"drift detected: \", feature_stats_and_anomalies.drift_detected)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "07i-yw89LsUL" + }, + "source": [ + "Observe the full statistics and drift" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "THUePhdLouBv" + }, + "outputs": [], + "source": [ + "print(fmj_with_drift.feature_stats_and_anomalies)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CtbizaJ0dkQC" + }, + "source": [ + "Observe drift in Feature" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KKIGf3CWdQFo" + }, + "outputs": [], + "source": [ + "feature_movie = fg.get_feature(\"movies\", latest_stats_count=5)\n", + "print(feature_movie)\n", + "\n", + "# There will be stats generated by two jobs, one has no drift, one detected drift\n", + "for feature_stats in feature_movie.feature_stats_and_anomalies:\n", + " print(\"feature monitor job id: \", feature_stats.feature_monitor_job_id)\n", + " print(\"drift score: \", feature_stats.distribution_deviation)\n", + " print(\"drift detected: \", feature_stats.drift_detected)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2a4e033321ad" + }, + "source": [ + "## Cleaning up" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ffd3dc65a25f" + }, + "source": [ + "### Delete feature monitor, feature and feature group" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7_4o4dg73S03" + }, + "outputs": [], + "source": [ + "# Delete Feature Monitor, all FeatureMonitorJobs created under the Feature Monitor will be automatically deleted, but stats kept under Feature.\n", + "fm.delete()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7517048d8510" + }, + "outputs": [], + "source": [ + "# Delete Feature, all stats under the Feature will be automatically deleted.\n", + "movies_feature.delete()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8488287340ca" + }, + "outputs": [], + "source": [ + "# Delete Feature Group.\n", + "fg.delete()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "753e85f60d06" + }, + "source": [ + "### Delete BigQuery dataset and table" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "99e984fe9c53" + }, + "outputs": [], + "source": [ + "client = bigquery.Client()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6cc5fdf51c9e" + }, + "outputs": [], + "source": [ + "client.delete_table(f\"{BQ_TABLE_URI}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4bac93a9ffcb" + }, + "outputs": [], + "source": [ + "client.delete_dataset(f\"{PROJECT_ID}.{BQ_DATASET_ID}\")" + ] + } + ], + "metadata": { + "colab": { + "name": "feature_monitoring_with_feature_registry.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}