diff --git a/notebooks/community/model_garden/model_garden_hf_paligemma2_deployment.ipynb b/notebooks/community/model_garden/model_garden_hf_paligemma2_deployment.ipynb new file mode 100644 index 000000000..cf2aa6a4f --- /dev/null +++ b/notebooks/community/model_garden/model_garden_hf_paligemma2_deployment.ipynb @@ -0,0 +1,422 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "OdZIyZwjgsQcOXnmE8X0xy40" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "language": "markdown", + "metadata": { + "id": "VJWDivOv3OWy" + }, + "source": [ + "# Vertex AI Model Garden - PaliGemma 2 (Deployment)\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Run in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iOmVD9tZXucQ" + }, + "source": [ + "## Overview\n", + "\n", + "This notebook provides a practical introduction to using the PaLiGemma 2 model, a powerful vision-language model developed by Google. We'll demonstrate how to leverage its multimodal capabilities to perform tasks like vision question answering. Consult the [model card](https://huggingface.co/google/paligemma2-3b-pt-224) for more information.\n", + "\n", + "\n", + "### Objective\n", + "\n", + "- Deploy PaliGemma 2 to a Vertex AI Endpoint.\n", + "- Make predictions to the endpoint including:\n", + " - Answering questions about a given image.\n", + "\n", + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2aFHbs1g6Wc-" + }, + "source": [ + "## Before you begin" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "language": "python", + "metadata": { + "cellView": "form", + "id": "QvQjsmIJ6Y3f" + }, + "outputs": [], + "source": [ + "# @title Setup Google Cloud project\n", + "# @markdown ### Prerequisites\n", + "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "# @markdown 2. [Optional] [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n", + "\n", + "# Upgrade Vertex AI SDK.\n", + "! pip3 install --upgrade --quiet 'google-cloud-aiplatform>=1.64.0'\n", + "! git clone https://github.com/GoogleCloudPlatform/vertex-ai-samples.git\n", + "\n", + "# Import the necessary packages\n", + "import importlib\n", + "from typing import Any, Dict, Tuple\n", + "\n", + "from google.cloud import aiplatform\n", + "\n", + "common_util = importlib.import_module(\n", + " \"vertex-ai-samples.community-content.vertex_model_garden.model_oss.notebook_util.common_util\"\n", + ")\n", + "\n", + "\n", + "# @markdown You must provide a Hugging Face User Access Token (read) to access the Llama 3 models. You can follow the [Hugging Face documentation](https://huggingface.co/docs/hub/en/security-tokens) to create a **read** access token and put it in the `HF_TOKEN` field below.\n", + "HF_TOKEN = \"\" # @param {type:\"string\", isTemplate:true}\n", + "\n", + "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "# @markdown 2. **[Optional]** [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n", + "\n", + "BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n", + "\n", + "# @markdown 3. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n", + "\n", + "REGION = \"\" # @param {type:\"string\"}\n", + "\n", + "# @markdown 4. If you want to run predictions with A100 80GB or H100 GPUs, we recommend using the regions listed below. **NOTE:** Make sure you have associated quota in selected regions. Click the links to see your current quota for each GPU type: [Nvidia A100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_a100_80gb_gpus), [Nvidia H100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_h100_gpus).\n", + "\n", + "# @markdown > | Machine Type | Accelerator Type | Recommended Regions |\n", + "# @markdown | ----------- | ----------- | ----------- |\n", + "# @markdown | a2-ultragpu-1g | 1 NVIDIA_A100_80GB | us-central1, us-east4, europe-west4, asia-southeast1, us-east4 |\n", + "# @markdown | a3-highgpu-2g | 2 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n", + "# @markdown | a3-highgpu-4g | 4 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n", + "# @markdown | a3-highgpu-8g | 8 NVIDIA_H100_80GB | us-central1, us-east5, europe-west4, us-west1, asia-southeast1 |\n", + "\n", + "# The pre-built serving docker images.\n", + "SERVE_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-one-serve:latest\"\n", + "\n", + "\n", + "def deploy_model(\n", + " model_id: str = None,\n", + " task: str = \"paligemma_VQA\",\n", + " machine_type: str = \"g2-standard-8\",\n", + " accelerator_type: str = \"NVIDIA_L4\",\n", + " accelerator_count: int = 1,\n", + " service_account: str = None,\n", + " serving_port: int = 7080,\n", + " serving_route: str = \"/predict\",\n", + " serving_docker_uri: str = SERVE_DOCKER_URI,\n", + " hf_token: str = None,\n", + ") -> Tuple[aiplatform.Endpoint, aiplatform.Model]:\n", + " \"\"\"Deploys a model to a real-time prediction endpoint.\n", + "\n", + " Args:\n", + " model_id: The model ID.\n", + " task: The task to perform.\n", + " machine_type: The machine type.\n", + " accelerator_type: The accelerator type.\n", + " accelerator_count: The accelerator count.\n", + " service_account: The service account.\n", + " serving_port: The serving port.\n", + " serving_route: The serving route.\n", + " hf_token: HuggingFace token for model access.\n", + "\n", + " Returns:\n", + " A tuple containing the created endpoint and deployed model objects.\n", + " \"\"\"\n", + "\n", + " common_util.check_quota(\n", + " project_id=PROJECT_ID,\n", + " region=REGION,\n", + " accelerator_type=accelerator_type,\n", + " accelerator_count=accelerator_count,\n", + " is_for_training=False,\n", + " )\n", + "\n", + " endpoint = aiplatform.Endpoint.create(\n", + " display_name=common_util.get_job_name_with_datetime(prefix=\"paligemma-2\")\n", + " )\n", + " serving_env = {\n", + " \"MODEL_ID\": model_id,\n", + " \"DEPLOY_SOURCE\": \"notebook\",\n", + " \"HF_TOKEN\": hf_token,\n", + " \"TASK\": task,\n", + " }\n", + " model = aiplatform.Model.upload(\n", + " display_name=task,\n", + " serving_container_image_uri=serving_docker_uri,\n", + " serving_container_ports=[serving_port],\n", + " serving_container_predict_route=serving_route,\n", + " serving_container_health_route=\"/ping\",\n", + " serving_container_environment_variables=serving_env,\n", + " )\n", + " model.deploy(\n", + " endpoint=endpoint,\n", + " machine_type=machine_type,\n", + " accelerator_type=accelerator_type,\n", + " accelerator_count=accelerator_count,\n", + " service_account=service_account,\n", + " )\n", + " return endpoint, model\n", + "\n", + "\n", + "def vqa_predict(\n", + " endpoint: aiplatform.Endpoint,\n", + " image_url: str,\n", + " text_prompt: str,\n", + " parameters: Dict[str, Any] = None,\n", + ") -> str:\n", + " \"\"\"Predicts the answer to a question about an image using an Endpoint,\n", + "\n", + " and passes parameters in the payload.\n", + "\n", + " Args:\n", + " endpoint: The deployed Vertex AI endpoint.\n", + " image_url: URL of the image to ask about.\n", + " text_prompt: The text prompt question.\n", + " parameters: Additional parameters for the prediction request.\n", + "\n", + " Returns:\n", + " The predicted answer string or None if no prediction.\n", + " \"\"\"\n", + "\n", + " instances = []\n", + " if text_prompt:\n", + " instances.append(\n", + " {\n", + " \"text_prompt\": text_prompt,\n", + " \"image_url\": image_url,\n", + " }\n", + " )\n", + "\n", + " # Construct the prediction payload\n", + " payload = {\"instances\": instances}\n", + " if parameters:\n", + " payload[\"parameters\"] = parameters\n", + "\n", + " response = endpoint.predict(instances=instances, parameters=parameters)\n", + " answer = None\n", + " if response.predictions:\n", + " answer = response.predictions[0][\"text\"].split(\"\\n\")[1]\n", + " return answer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kyMJXkfviWgl" + }, + "source": [ + "## Deploy PaliGemma to a Vertex AI Endpoint" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "toY-WPKDFesF" + }, + "outputs": [], + "source": [ + "# @title Deploy\n", + "\n", + "# @markdown This section uploads the prebuilt PaliGemma 2 models to Model Registry and deploys it to a Vertex AI Endpoint. It takes approximately 15 minutes to finish.\n", + "\n", + "# @markdown Select the desired resolution and precision of prebuilt model to deploy, leaving the optional `custom_paligemma_model_uri` as is. Higher resolution and precision_type can result in better inference results, but may require additional GPU.\n", + "\n", + "# @markdown You can also serve a finetuned PaliGemma model by setting `resolution` and `precision_type` to the resolution and precision type of the original base model and then setting `custom_paligemma_model_uri` to the GCS URI containing the model.\n", + "\n", + "# @markdown **Note**: You cannot use accelerator type `NVIDIA_TESLA_V100` to serve prebuilt or finetuned PaliGemma models with resolution `896` and precision_type `float32`.\n", + "\n", + "\n", + "MODEL_ID = \"google/paligemma2-3b-pt-224\" # @param [\"google/paligemma2-3b-pt-224\", \"google/paligemma2-3b-pt-448\", \"google/paligemma2-10b-ft-docci-448\"]\n", + "TASK = \"paligemma_VQA\" # @param [\"paligemma_VQA\"]\n", + "accelerator_type = \"NVIDIA_L4\" # @param [\"NVIDIA_L4\"]\n", + "accelerator_count = 1 # @param [1]\n", + "machine_type = \"g2-standard-8\" # @param [\"g2-standard-8\"]\n", + "\n", + "\n", + "check_quota(\n", + " project_id=PROJECT_ID,\n", + " region=REGION,\n", + " accelerator_type=accelerator_type,\n", + " accelerator_count=accelerator_count,\n", + " is_for_training=False,\n", + ")\n", + "# @markdown If you want to use other accelerator types not listed above, then check other Vertex AI prediction supported accelerators and regions at https://cloud.google.com/vertex-ai/docs/predictions/configure-compute. You may need to manually set the `machine_type`, `accelerator_type`, and `accelerator_count` in the code by clicking `Show code` first.\n", + "\n", + "model, endpoint = deploy_model(\n", + " hf_token=HF_TOKEN,\n", + " model_id=MODEL_ID,\n", + " task=TASK,\n", + " machine_type=machine_type,\n", + " accelerator_type=accelerator_type,\n", + " accelerator_count=accelerator_count,\n", + " service_account=SERVICE_ACCOUNT,\n", + " serving_port=7080,\n", + " serving_route=\"/predict\",\n", + " serving_docker_uri=SERVE_DOCKER_URI,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "tOtYOhZa3lsx" + }, + "outputs": [], + "source": [ + "# @title [Optional] Loading an existing Endpoint\n", + "# @markdown If you've already deployed an Endpoint, you can load it by filling in the Endpoint's ID below.\n", + "# @markdown You can view deployed Endpoints at [Vertex Online Prediction](https://console.cloud.google.com/vertex-ai/online-prediction/endpoints).\n", + "endpoint_id = \"\" # @param {type: \"string\"}\n", + "\n", + "if endpoint_id:\n", + " endpoint = aiplatform.Endpoint(\n", + " endpoint_name=endpoint_id,\n", + " project=PROJECT_ID,\n", + " location=REGION,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Idtx2ETNQtn" + }, + "source": [ + "### Predict\n", + "\n", + "The following sections will use images from [pexels.com](https://www.pexels.com/) for demoing purposes. All the images have the following license: https://www.pexels.com/license/.\n", + "\n", + "Images will be resized to a width of 1000 pixels by default since requests made to a Vertex Endpoint are limited to 1.500MB." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "FxdKXm6INQtn" + }, + "outputs": [], + "source": [ + "# @title Visual Question Answering\n", + "\n", + "# @markdown This section uses the deployed PaliGemma model to answer questions about a given image.\n", + "\n", + "# @markdown ![](https://goofytails.com/cdn/shop/files/labrador-retriever_1000x.jpg?w=1260&h=750)\n", + "image_url = \"https://goofytails.com/cdn/shop/files/labrador-retriever_1000x.jpg\" # @param {type:\"string\"}\n", + "\n", + "# @markdown You may leave question prompts empty and they will be ignored.\n", + "question_prompt = \"What animal is shown in the picture?\" # @param {type: \"string\"}\n", + "\n", + "# @markdown The question prompt can be non-English languages.\n", + "\n", + "# Using max_new_tokens along with other parameters\n", + "parameters_with_tokens = {\"max_new_tokens\": 50}\n", + "predictions_with_tokens = vqa_predict(\n", + " endpoint=endpoint,\n", + " image_url=image_url,\n", + " text_prompt=question_prompt,\n", + " parameters=parameters_with_tokens,\n", + ")\n", + "\n", + "print(f\"Prediction Response: {predictions_with_tokens}\")\n", + "# @markdown Click \"Show Code\" to see more details." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IrVZ030i4XMY" + }, + "source": [ + "## Clean up resources" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "YsMpOI1kYjil" + }, + "outputs": [], + "source": [ + "# @markdown Delete the experiment models and endpoints to recycle the resources\n", + "# @markdown and avoid unnecessary continuous charges that may incur.\n", + "\n", + "# Undeploy model and delete endpoint.\n", + "for endpoint in endpoints.values():\n", + " endpoint.delete(force=True)\n", + "\n", + "# Delete models.\n", + "for model in models.values():\n", + " model.delete()\n", + "\n", + "delete_bucket = False # @param {type:\"boolean\"}\n", + "if delete_bucket:\n", + " ! gsutil -m rm -r $BUCKET_NAME" + ] + } + ], + "metadata": { + "colab": { + "name": "model_garden_hf_paligemma2_deployment.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}