diff --git a/notebooks/official/model_monitoring/get_started_with_model_monitoring_xgboost.ipynb b/notebooks/official/model_monitoring/get_started_with_model_monitoring_xgboost.ipynb index 161196c300..62a8e24752 100644 --- a/notebooks/official/model_monitoring/get_started_with_model_monitoring_xgboost.ipynb +++ b/notebooks/official/model_monitoring/get_started_with_model_monitoring_xgboost.ipynb @@ -34,20 +34,25 @@ "# Vertex AI Model Monitoring for XGBoost models\n", "\n", "\n", - " \n", - " \n", + " \n", - " \n", @@ -62,7 +67,7 @@ "source": [ "## Overview\n", "\n", - "This tutorial demonstrates how to use `Vertex AI Model Monitoring` for XGBoost models.\n", + "This tutorial demonstrates how to use Vertex AI Model Monitoring for XGBoost models.\n", "\n", "Learn more about [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring)." ] @@ -75,21 +80,21 @@ "source": [ "### Objective\n", "\n", - "In this notebook, you learn to use the `Vertex AI Model Monitoring` service to detect feature skew and drift in the input predict requests for XGBoost models.\n", + "In this notebook, you learn to use the Vertex AI Model Monitoring service to detect feature skew and drift in the input predict requests for XGBoost models.\n", "\n", "This tutorial uses the following Google Cloud ML services:\n", "\n", - "- `Vertex AI Model Monitoring`\n", - "- `Vertex AI Prediction`\n", - "- `Vertex AI Model` resource\n", - "- `Vertex AI Endpoint` resource\n", + "- Vertex AI Model Monitoring\n", + "- Vertex AI Prediction\n", + "- Vertex AI model resource\n", + "- Vertex AI endpoint resource\n", "\n", "The steps performed include:\n", "\n", "- Download a pre-trained XGBoost model.\n", - "- Upload the pre-trained model as a `Model` resource.\n", - "- Deploy the `Model` resource to the `Endpoint` resource.\n", - "- Configure the `Endpoint` resource for model monitoring:\n", + "- Upload the pre-trained model to Vertex AI model registry.\n", + "- Deploy the model resource to a Vertex AI endpoint resource.\n", + "- Configure the endpoint resource for model monitoring:\n", " - drift detection only -- no access to training data.\n", " - predefine the input schema to map feature alias names to the unnamed array input to the model.\n", "- Generate synthetic prediction requests for drift.\n", @@ -126,8 +131,16 @@ "Learn about [Vertext AI\n", "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", - "Calculator](https://cloud.google.com/products/calculator/)\n", - "to generate a cost estimate based on your projected usage." + "Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f0316df526f8" + }, + "source": [ + "## Get started" ] }, { @@ -136,9 +149,7 @@ "id": "f3848df1e5b0" }, "source": [ - "## Installation\n", - "\n", - "Install the packages required for executing this notebook." + "### Install Vertex AI SDK for Python and other required packages" ] }, { @@ -160,7 +171,9 @@ "id": "restart" }, "source": [ - "### Colab only: Uncomment the following cell to restart the kernel" + "### Restart runtime (Colab only)\n", + "\n", + "To use the newly installed packages, you must restart the runtime on Google Colab." ] }, { @@ -171,148 +184,103 @@ }, "outputs": [], "source": [ - "# Automatically restart kernel after installs so that your environment can access the new packages\n", - "# import IPython\n", - "\n", - "# app = IPython.Application.instance()\n", - "# app.kernel.do_shutdown(True)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "before_you_begin:nogpu" - }, - "source": [ - "## Before you begin\n", + "import sys\n", "\n", - "### Set your project ID\n", + "if \"google.colab\" in sys.modules:\n", "\n", - "**If you don't know your project ID**, try the following:\n", - "* Run `gcloud config list`.\n", - "* Run `gcloud projects list`.\n", - "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "set_project_id" - }, - "outputs": [], - "source": [ - "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + " import IPython\n", "\n", - "# Set the project id\n", - "! gcloud config set project {PROJECT_ID}" + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "metadata": { - "id": "region" - }, - "source": [ - "#### Region\n", - "\n", - "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "2dw8q9fdQEH5" + "id": "c87a2a5d7e35" }, - "outputs": [], "source": [ - "REGION = \"us-central1\"" + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" ] }, { "cell_type": "markdown", "metadata": { - "id": "42c8a7c56abd" + "id": "ff666ce4051c" }, "source": [ - "#### User Email\n", + "### Authenticate your notebook environment (Colab only)\n", "\n", - "Set your user email address to receive monitoring alerts." + "Authenticate your environment on Google Colab." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "ce2589511bb6" + "id": "cc7251520a07" }, "outputs": [], "source": [ - "import os\n", + "import sys\n", "\n", - "USER_EMAIL = \"[your-email-addr]\" # @param {type:\"string\"}\n", + "if \"google.colab\" in sys.modules:\n", "\n", - "if os.getenv(\"IS_TESTING\"):\n", - " USER_EMAIL = \"noreply@google.com\"" + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { - "id": "gcp_authenticate" + "id": "374c1c8dea6d" }, "source": [ - "### Authenticate your Google Cloud account\n", + "### Set Google Cloud project information\n", "\n", - "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n", - "\n", - "**1. Vertex AI Workbench**\n", - "* Do nothing as you are already authenticated.\n", - "\n", - "**2. Local JupyterLab instance, uncomment and run:**" + "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "ce6043da7b33" + "id": "set_project_id" }, "outputs": [], "source": [ - "# ! gcloud auth login" + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}" ] }, { "cell_type": "markdown", "metadata": { - "id": "0367eac06a10" + "id": "42c8a7c56abd" }, "source": [ - "**3. Colab, uncomment and run:**" + "### User Email\n", + "\n", + "Set your user email address to receive monitoring alerts." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "21ad4dbb4a61" + "id": "ce2589511bb6" }, "outputs": [], "source": [ - "# from google.colab import auth\n", - "# auth.authenticate_user()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c13224697bfb" - }, - "source": [ - "**4. Service account or other**\n", - "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples." + "import os\n", + "\n", + "USER_EMAIL = \"[your-email-addr]\" # @param {type:\"string\"}\n", + "\n", + "if os.getenv(\"IS_TESTING\"):\n", + " USER_EMAIL = \"noreply@google.com\"" ] }, { @@ -323,7 +291,7 @@ "source": [ "### Notes about service account and permission\n", "\n", - "**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n", + "**By default no configuration is required**, if you run into any permission related issue, make sure the service accounts above have the required roles:\n", "\n", "|Service account email|Description|Roles|\n", "|---|---|---|\n", @@ -331,14 +299,15 @@ "|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n", "\n", "\n", - "1. Goto https://console.cloud.google.com/iam-admin/iam.\n", - "2. Check the \"Include Google-provided role grants\" checkbox.\n", + "1. Go to [IAM console](https://console.cloud.google.com/iam-admin/iam).\n", + "2. Check the **Include Google-provided role grants** checkbox.\n", "3. Find the above emails.\n", "4. Grant the corresponding roles.\n", "\n", "### Using data source from a different project\n", - "- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n", - "- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role." + "If you're using data sources from a different project:\n", + "- For BigQuery data source, grant the \\\"BigQuery Data Viewer\\\" role to both the service accounts.\n", + "- For CSV data source, grant the \\\"Storage Object Viewer\\\" role to both the service accounts." ] }, { @@ -369,7 +338,7 @@ "id": "autoset_bucket" }, "source": [ - "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." ] }, { @@ -380,7 +349,7 @@ }, "outputs": [], "source": [ - "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI" + "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI" ] }, { @@ -403,7 +372,8 @@ "import os\n", "import random\n", "\n", - "from google.cloud import aiplatform, bigquery\n", + "import google.cloud.aiplatform as aiplatform\n", + "from google.cloud import bigquery\n", "from google.cloud.aiplatform import model_monitoring" ] }, @@ -415,7 +385,7 @@ "source": [ "### Initialize Vertex AI SDK for Python\n", "\n", - "Initialize the Vertex AI SDK for Python for your project and corresponding bucket." + "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)." ] }, { @@ -426,7 +396,7 @@ }, "outputs": [], "source": [ - "aiplatform.init(project=PROJECT_ID, location=REGION)" + "aiplatform.init(project=PROJECT_ID, location=LOCATION)" ] }, { @@ -461,7 +431,7 @@ "\n", "Set the pre-built Docker container image for prediction.\n", "\n", - "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)." + "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/vertex-ai/docs/predictions/pre-built-containers)." ] }, { @@ -474,9 +444,9 @@ "source": [ "DEPLOY_VERSION = \"xgboost-cpu.1-1\"\n", "\n", - "LOCATION = REGION.split(\"-\")[0]\n", - "\n", - "DEPLOY_IMAGE = f\"{LOCATION}-docker.pkg.dev/vertex-ai/prediction/{DEPLOY_VERSION}:latest\"\n", + "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n", + " LOCATION.split(\"-\")[0], DEPLOY_VERSION\n", + ")\n", "\n", "print(\"Deployment:\", DEPLOY_IMAGE)" ] @@ -492,13 +462,13 @@ "Next, set the machine types to use for training and prediction.\n", "\n", "- Set the variable `DEPLOY_COMPUTE` to configure your compute resources for prediction.\n", - " - `machine type`\n", + " - Set a `machine type`:\n", " - `n1-standard`: 3.75GB of memory per vCPU\n", " - `n1-highmem`: 6.5GB of memory per vCPU\n", " - `n1-highcpu`: 0.9 GB of memory per vCPU\n", " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n", "\n", - "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*." + "**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs." ] }, { @@ -530,26 +500,26 @@ "source": [ "## Introduction to Vertex AI Model Monitoring\n", "\n", - "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n", + "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skewness and drift detection of the features in the inbound prediction requests or the feature attributions (Explainable AI) in the outbound prediction response. In other words, you monitor the distribution of the attributions that quantify feature contributions to the output (predictions).\n", "\n", "The following are the basic steps to enable model monitoring:\n", "\n", - "1. Deploy a `Vertex AI` AutoML or custom tabular model to an `Vertex AI Endpoint`.\n", + "1. Deploy a Vertex AI AutoML or custom tabular model to a Vertex AI endpoint.\n", "2. Configure a model monitoring specification.\n", - "3. Upload the model monitoring specification to the `Vertex AI Endpoint`.\n", - "4. Upload or automatic generation of the `input schema` for parsing.\n", - "5. For feature skew detection, upload the training data for automatic generation of the feature distribution.\n", - "6. For feature attributions, upload corresponding `Vertex AI Explainability` specification.\n", + "3. Upload the model monitoring specification to the Vertex AI endpoint.\n", + "4. Upload schema or use automatic generation of the *input schema* for parsing.\n", + "5. For feature skewness detection, upload the training data. This enables automatic generation of the feature distributions.\n", + "6. For feature attributions, upload corresponding *Vertex Explainable AI* specification.\n", "\n", - "Once configured, you can enable/disable monitoring, change alerts and update the model monitoring configuration. \n", + "Once configured, you can enable or disable monitoring, change alerts and update the model monitoring configuration. \n", "\n", - "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n", + "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skewness or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to the model, and the monitoring interval.\n", "\n", "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service attempts to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n", "\n", - "For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n", + "For skewness detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n", "\n", - "For feature attribution skew and drift detection, requires enabling your deployed model for `Vertex AI Explainability` for custom tabular models. For AutoML models, `Vertex AI Explainability` is automatically enabled.\n", + "For feature attribution skewness and drift detection in feature attributions, you're required to enable Vertex Explainable AI feature for your deployed custom tabular models. For AutoML models, Vertex Explainable AI is automatically enabled.\n", "\n", "Learn more about [Introduction to Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)." ] @@ -560,13 +530,13 @@ "id": "9bf06cd476e9" }, "source": [ - "### Upload the model artifacts as a `Vertex AI Model` resource\n", + "### Upload the model artifacts to Vertex AI Model Registry\n", "\n", - "First, you upload the pre-trained XGBoost tabular model artifacts as a `Vertex AI Model` resource using the `upload()` method, with the following parameters:\n", + "First, upload the pre-trained XGBoost tabular model artifacts as a Vertex AI model resource using the `upload()` method, with the following parameters:\n", "\n", - "- `display_name`: The human readable name for the `Model` resource.\n", + "- `display_name`: The human readable name for the model resource.\n", "- `artifact_uri`: The Cloud Storage location of the model artifacts.\n", - "- `serving_container_image`: The serving container image to use when the model is deployed to a `Vertex AI Endpoint` resource.\n", + "- `serving_container_image`: The serving container image to use when the model is deployed to a Vertex AI endpoint resource.\n", "- `sync`: Whether to wait for the process to complete, or return immediately (async)." ] }, @@ -598,9 +568,9 @@ "id": "1069c6f0eba8" }, "source": [ - "### Deploy the `Vertex AI Model` resource to a `Vertex AI Endpoint` resource\n", + "### Deploy the model to an endpoint\n", "\n", - "Next, you deploy your `Vertex AI Model` resource to a `Vertex AI Endpoint` resource using the `deploy()` method, with the following parameters:\n", + "Next, deploy your Vertex AI model resource to a Vertex AI endpoint resource using the `deploy()` method, with the following parameters:\n", "\n", "- `deploy_model_display`: The human reable name for the deployed model.\n", "- `machine_type`: The machine type for each VM node instance.\n", @@ -638,7 +608,7 @@ "\n", "Configuring the monitoring job consists of the following specifications:\n", "\n", - "- `alert_config`: The email address(es) to send monitoring alerts to.\n", + "- `alert_config`: The email address(es) that are supposed to receive the monitoring alerts.\n", "- `schedule_config`: The time window to analyze predictions.\n", "- `logging_sampling_strategy`: The rate for sampling prediction requests. \n", "- `drift_config`: The features and drift thresholds to monitor.\n", @@ -653,9 +623,9 @@ "source": [ "### Configure the alerting specification\n", "\n", - "First, you configure the `alerting_config` specification with the following settings:\n", + "Configure the `alerting_config` specification with the following settings:\n", "\n", - "- `user_emails`: A list of one or more email to send alerts to.\n", + "- `user_emails`: A list of one or more emails that should receive the alerts.\n", "- `enable_logging`: Streams detected anomalies to Cloud Logging. Default is False." ] }, @@ -683,7 +653,7 @@ "\n", "Next, you configure the `schedule_config` specification with the following settings:\n", "\n", - "- `monitor_interval`: Sets the model monitoring job scheduling interval in hours. Minimum time interval is 1 hour. For example, at a one hour interval, the monitoring job will run once an hour." + "- `monitor_interval`: Sets the model monitoring job scheduling interval in hours. Minimum time interval is 1 hour. " ] }, { @@ -802,15 +772,15 @@ "source": [ "### Create the input schema\n", "\n", - "The monitoring service needs to know the features and data types for the the feature inputs to the model, which is referred to as the `input schema`. The `input schema` can either be \n", + "The monitoring service needs to know the features and data types for the the feature inputs to the model, which is referred to as the *input schema*. The *input schema* can either be \n", " - Preloaded to the monitoring service.\n", " - Automatically generated by the monitoring service after receiving first 1000 prediction instances.\n", " \n", - "In this tutorial, you preload the `input schema`.\n", + "In this tutorial, you preload the *input schema*.\n", "\n", "#### Create the predefined input schema\n", "\n", - "The predefined `input schema` is specified as a YAML file. In this example, you generate the YAML specification according to the model's input layer. In this case, the input layer is an array of four floating point numeric values. In the schema, this is represented by:\n", + "The predefined *input schema* is specified as a YAML file. In this example, you generate the YAML specification according to the model's input layer. In this case, the input layer is an array of four floating point numeric values. In the schema, this is represented by:\n", "\n", "- `type: array`: Refers to the input is an array (list)\n", "- `properties`: An ordered list of the inputs in the array\n", @@ -866,17 +836,17 @@ "source": [ "### Create the monitoring job\n", "\n", - "You create a monitoring job, with your monitoring specifications, using the `aiplatform.ModelDeploymentMonitoringJob.create()` method, with the following parameters:\n", + "Create a monitoring job, with your monitoring specifications, using vertex AI's `ModelDeploymentMonitoringJob.create()` method, with the following parameters:\n", "\n", "- `display_name`: The human readable name for the monitoring job.\n", "- `project`: The project ID.\n", - "- `region`: The region.\n", - "- `endpoint`: The fully qualified resource name of the `Vertex AI Endpoint` to enable monitoring.\n", + "- `location`: The location.\n", + "- `endpoint`: The fully qualified resource name of the Vertex AI endpoint to enable monitoring.\n", "- `logging_sampling_strategy`: The specification for the sampling configuration.\n", "- `schedule_config`: The specification for the scheduling configuration.\n", "- `alert_config`: The specification for the alerting configuration.\n", "- `objective_configs`: The specification for the objectives configuration.\n", - "- `analysis_instance_schema_uri`: The location of the YAML file containing the `input schema`." + "- `analysis_instance_schema_uri`: The location of the YAML file containing the *input schema*." ] }, { @@ -890,7 +860,7 @@ "monitoring_job = aiplatform.ModelDeploymentMonitoringJob.create(\n", " display_name=\"xgboost_iris\",\n", " project=PROJECT_ID,\n", - " location=REGION,\n", + " location=LOCATION,\n", " endpoint=endpoint,\n", " logging_sampling_strategy=logging_sampling_strategy,\n", " schedule_config=schedule_config,\n", @@ -912,13 +882,13 @@ "\n", "An email notification is sent to the email address in the alerting configuration, notifying that the model monitoring job is now enabled.\n", "\n", - "The contents will appear like:\n", + "The contents of the email appear as below:\n", "\n", "
\n", "Hello Vertex AI Customer,\n", "\n", "You are receiving this mail because you are using the Vertex AI Model Monitoring service.\n", - "This mail is to inform you that we received your request to set up drift or skew detection for the Prediction Endpoint listed below. Starting from now, incoming prediction requests will be sampled and logged for analysis.\n", + "This mail is to inform you that we received your request to set up drift or skew detection for the prediction endpoint listed below. Starting from now, incoming prediction requests will be sampled and logged for analysis.\n", "Raw requests and responses will be collected from prediction service and saved in bq://[your-project-id].model_deployment_monitoring_[endpoint-id].serving_predict .\n", "
" ] @@ -931,13 +901,9 @@ "source": [ "#### Monitoring Job State\n", "\n", - "After you start the `Vertex AI Model Monitoring` job, there are three transition states the job may be in:\n", - "\n", - "- `PENDING`: The job is configured for skew detection and the `skew distribution baseline` is being calculated. The monitoring service will initiate a batch job to generate the distribution baseline from the training data. \n", + "After you start the Vertex AI Model Monitoring job, it stays in a **PENDING** state until `skew distribution baseline` is calculated. The monitoring service initiates a batch job to generate the distribution baseline from the training data. \n", "\n", - "- `OFFLINE`: The monitoring job is between monitoring intervals.\n", - "\n", - "- `RUNNING`: The monitoring job on a per interval basis is analyzing the sampled data." + "Once the baseline distribution is generated, then the monitoring job changes to **OFFLINE** state. On a per interval basis, for example, once an hour, the monitoring job enters **RUNNING** state while analyzing the sampled data. Once completed, it returns to the **OFFLINE** state while awaiting the next scheduled analysis." ] }, { @@ -959,7 +925,7 @@ "id": "e385d103aba6" }, "source": [ - "pause for the monitoring job to be enabled" + "Pause a bit for the monitoring job to be enabled" ] }, { @@ -1011,7 +977,7 @@ "source": [ "### Make the prediction requests\n", "\n", - "Next, you send the the 1000 prediction requests to your `Vertex AI Endpoint` resource using the `predict()` method.\n", + "Next, you send the the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method.\n", "\n", "Note, the model outputs the class as a floating point value. For example, `0.0` is the label `0`." ] @@ -1041,11 +1007,11 @@ "source": [ "### Logging sampled requests\n", "\n", - "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n", + "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service analyzes the sampled data.\n", "\n", "Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 500 entries.\n", "\n", - "*Note*: This may take upto the length of the monitoring interval (e.g., one hour)." + "*Note*: This takes up to the length of the monitoring interval (e.g., one hour)." ] }, { @@ -1080,7 +1046,7 @@ "source": [ "### Generate synthetic prediction requests for drift detection\n", "\n", - "You modify the data (synthetic) to trigger the drift detection in the prediction requests from the previous basline distribution versus the current distribution, as follows:\n", + "Modify the data (synthetic) to trigger the drift detection in the prediction requests from the previous basline distribution versus the current distribution, as follows:\n", "\n", "- `sepal_length`: increase the value 4x." ] @@ -1110,7 +1076,7 @@ "source": [ "### Make the prediction requests\n", "\n", - "Next, you send the the 1000 prediction requests to your `Vertex AI Endpoint` resource using the `predict()` method." + "Next, you send the the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method." ] }, { @@ -1138,13 +1104,13 @@ "source": [ "### Drift detection during monitoring\n", "\n", - "The feature input drift detection will occur at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job will go from `OFFLINE` to `RUNNING`. While running, it will analyze the logged sampled tables from the predictions during this interval and compare them to the previous monitoring interva distribution.\n", + "The feature input drift detection occurs at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job goes from **OFFLINE** to **RUNNING**. While running, it analyzes the logged sampled tables from the predictions during this interval and compares them to the previous monitoring interval distribution.\n", "\n", - "Once the analysis is completed, the monitoring job will send email notifications on the detected drift, in this case `cnt_user_engagement`, and the monitoring job will go into `OFFLINE` state until the next interval.\n", + "Once the analysis is completed, the monitoring job sends email notifications on the detected drift, in this case `cnt_user_engagement`. Then, the monitoring job goes into **OFFLINE** state until the next interval.\n", "\n", "#### Wait for monitoring interval\n", "\n", - "It can take upwards of 40 minutes from when the analyis occurred on the monitoring interval to when you receive an email alert." + "It can take up to 40 minutes or more from the moment the analyis on the monitoring interval is done until you receive an email alert." ] }, { @@ -1167,11 +1133,11 @@ "source": [ "### Logging sampled requests\n", "\n", - "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n", + "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service analyzes the sampled data.\n", "\n", "Next, you wait for the second logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries.\n", "\n", - "*Note*: This may take upto the length of the monitoring interval (e.g., one hour)." + "*Note*: This takes up to the length of the monitoring interval (e.g., one hour)." ] }, { @@ -1206,7 +1172,9 @@ "source": [ "### Delete the monitoring job\n", "\n", - "You can delete the monitoring job using the `delete()` method. " + "Once you've received the email alerts and verified the content, you can:\n", + "- pause the monitoring job using the `pause()` method.\n", + "- delete the monitoring job using the `delete()` method. " ] }, { @@ -1217,33 +1185,12 @@ }, "outputs": [], "source": [ + "# Pause the job\n", "monitoring_job.pause()\n", + "# Delete the job\n", "monitoring_job.delete()" ] }, - { - "cell_type": "markdown", - "metadata": { - "id": "c6280efab664" - }, - "source": [ - "#### Undeploy and delete the `Vertex AI Endpoint` resource\n", - "\n", - "Your `Vertex AI Endpoint` resource can be deleted using the `delete()` method. Prior to deleting, any model deployed to your `Vertex AI Endpoint` resource, must first be undeployed." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ad0d28b762e5" - }, - "outputs": [], - "source": [ - "endpoint.undeploy_all()\n", - "endpoint.delete()" - ] - }, { "cell_type": "markdown", "metadata": { @@ -1266,13 +1213,24 @@ }, "outputs": [], "source": [ - "delete_bucket = False\n", + "# Undeploy the model from endpoint before deletion\n", + "endpoint.undeploy_all()\n", "\n", - "if delete_bucket or os.getenv(\"IS_TESTING\"):\n", + "# Delete the endpoint\n", + "endpoint.delete()\n", + "\n", + "# Delete the model\n", + "model.delete()\n", + "\n", + "# Delete the Cloud Storage bucket\n", + "delete_bucket = False\n", + "if delete_bucket:\n", " ! gsutil rm -rf {BUCKET_URI}\n", "\n", + "# Delete the locally generated files\n", "! rm -f schema.yaml\n", "\n", + "# Delete the BigQuery table\n", "! bq rm -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}" ] }
\n", + " \n", " \n", - " \"Colab Run in Colab\n", + " \"Colab
Run in Colab\n", " \n", " 		\n", + " \n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", " \n", - " \"GitHub\n", + " \"GitHub
\n", " View on GitHub\n", " \n", " 		\n", + " \n", " \n", - " \"Vertex\n", + " \"Vertex
\n", " Open in Vertex AI Workbench\n", " \n", "