From 3a725593cb0062030396ba56601132af016fcda5 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 21 Aug 2024 15:12:11 +0530 Subject: [PATCH 1/8] Updated README for NIM Model Deployment --- model-deployment/containers/nim/Dockerfile | 12 ---- .../containers/nim/README-MODEL-CATALOG.md | 67 +++++++++++++++++++ model-deployment/containers/nim/README.md | 19 ++++-- 3 files changed, 79 insertions(+), 19 deletions(-) delete mode 100644 model-deployment/containers/nim/Dockerfile create mode 100644 model-deployment/containers/nim/README-MODEL-CATALOG.md diff --git a/model-deployment/containers/nim/Dockerfile b/model-deployment/containers/nim/Dockerfile deleted file mode 100644 index 76f85c83..00000000 --- a/model-deployment/containers/nim/Dockerfile +++ /dev/null @@ -1,12 +0,0 @@ -FROM nvcr.io/nim/meta/llama3-8b-instruct:latest - -USER root -COPY start.sh /opt/nim/start.sh -RUN chmod +x /opt/nim/start.sh - -ENV NIM_CACHE_PATH /opt/ds/model/deployed_model -ENV NIM_SERVER_PORT 8000 -EXPOSE ${NIM_SERVER_PORT} - -ENTRYPOINT [ "/bin/bash", "--login", "-c"] -CMD ["/opt/nim/start.sh"] \ No newline at end of file diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md new file mode 100644 index 00000000..5eab5e3a --- /dev/null +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -0,0 +1,67 @@ +# Overview + +Utilising Model Catalog to store Models in OCI. We describe two ways to achieve this: + +* Storing zipped model file in Model Catalog +* Utilising Model-by-reference Feature to download model + +# Pre-requisite + +The following are the pre-requisite: +* Notebook session with internet access (Recommended) +* Download the Llama 3 8B Instruct Model from [HuggingFace][https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct] or NGC repository. +* Download NIM Container image and upload to OCIR as described in README.md + +# Method 1: Export Model to Model Catalog + +Follow the steps mentioned [here][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/model-deployment/containers/llama2/README.md], refer the section One time download to OCI Model Catalog. + +We would utilise the above created model in the next steps to create the Model Deployment. + +# Method 2: Model-by-reference + +Follow the steps to upload your model to Object Storage [here][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md] + +Utilise the [section][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md] on Create Model by Reference using ADS to create the model. + +# ### Create Model deploy + +* To deploy the model now in the console, go back to your [OCI Data Science Project](https://cloud.oracle.com/data-science/project) + * Select the project you created earlier and then select `Model Deployment` + * Click on `Create model deployment` + * Under `Default configuration` set following custom environment variables + * Key: `MODEL_DEPLOY_PREDICT_ENDPOINT`, Value: `/v1/completions` + * Key: `MODEL_DEPLOY_HEALTH_ENDPOINT`, Value: `/v1/health/ready` + * Key: `NIM_MODEL_NAME`, Value: `/opt/ds/model/deployed_model` + * Key: `NIM_SERVER_PORT`, Value `8080` + * Key: `SHM_SIZE`, Value: `5g` + * Under `Models` click on the `Select` button and select the Model Catalog entry we created earlier + * Under `Compute` and then `Specialty and previous generation` select the `VM.GPU.A10.1` instance + * Under `Networking` choose the `Custom Networking` option and bring the VCN and subnet, which allows Internet access. + * Under `Logging` select the Log Group where you've created your predict and access log and select those correspondingly + * Click on `Show advanced options` at the bottom + * Select the checkbox `Use a custom container image` + * Select the OCIR repository and image we pushed earlier + * Use port 8000. + * Leave CMD and Entrypoint blank + * Click on `Create` button to create the model deployment + +* Once the model is deployed and shown as `Active`, you can execute inference against it. + * Go to the model you've just deployed and click on it + * Under the left side under `Resources` select `Invoking your model` + * You will see the model endpoint under `Your model HTTP endpoint` copy it. + +## Inference + + ```bash + oci raw-request \ + --http-method POST \ + --target-uri \ + --request-body '{"model": "/opt/ds/model/deployed_model", "messages": [ { "role":"user", "content":"Hello! How are you?" }, { "role":"assistant", "content":"Hi! I am quite well, how can I help you today?" }, { "role":"user", "content":"Can you write me a song?" } ], "top_p": 1, "n": 1, "max_tokens": 200, "stream": false, "frequency_penalty": 1.0, "stop": ["hello"] }' \ + --auth resource_principal + ``` + +## Troubleshooting + +[Reference](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/llama2#troubleshooting) + diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 8c37de9a..0b982457 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -5,6 +5,10 @@ This Readme walks through how to use NIM - [ Meta-Llama-3-8B-Instruct](https://h * [llama3](https://github.com/meta-llama/llama3) from Meta. * [NIM](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama3-8b-instruct) by Nvidia +We describe two approaches to create this Model Deployment on OCI: +* Download Model using API-KEY from NGC Nvidia (described below) +* Utilising Model Catalog to store Models in OCI, later used for model deployment + ## Prerequisites * Access the corresponding NIM container for the model. For example for llama3, fetch the latest available image from [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama3-8b-instruct/tags). If you are a first time user, you need to sign up a developer account and wait for access to be granted to required container image. Click Get Container Button and click Request Access for NIM. At the time of writing this blog, you need a business email address to get access to NIM. @@ -28,13 +32,13 @@ When experimenting with new frameworks and models, it is highly advisable to att * Then click `Create agent configuration` -## Build NIM Container for Model Deploy - -* Build the OCI Model Deploy compatible container image. This process bakes in some of the configurations that makes it easier to deploy on the platform +## Download NIM Container image and upload to OCIR - ```bash - docker build -f Dockerfile -t odsc-nim-llama3 . - ``` +* Pull the latest NIM Image to local machine or through NB session. Tag it with desired name. + ```bash + docker pull nvcr.io/nim/meta/llama3-8b-instruct:latest + docker tag nvcr.io/nim/meta/llama3-8b-instruct:latest odsc-nim-llama3:latest + ``` ## OCI Container Registry @@ -57,7 +61,7 @@ When experimenting with new frameworks and models, it is highly advisable to att Once you built and pushed the NIM container, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. ### Creating Model catalog -NIM container will download the model directly using publicly exposed NGC catalog APIs. To provide authorization token to download, we will save API key in a file and creaate a zip out of it. This zip file will then be used to create a model catalog resource. +NIM container will download the model directly using publicly exposed NGC catalog APIs. To provide authorization token to download, we will save API key in a file and create a zip out of it. This zip file will then be used to create a model catalog resource. Sample file content named `token`: ```bash nvapi-.......... @@ -73,6 +77,7 @@ This file will be available to container on location `/opt/ds/model/deployed_mod * Key: `MODEL_DEPLOY_PREDICT_ENDPOINT`, Value: `/v1/completions` * Key: `MODEL_DEPLOY_HEALTH_ENDPOINT`, Value: `/v1/health/ready` * Key: `NGC_API_KEY_FILE`, Value: `/opt/ds/model/deployed_model/token` + * Key: `NIM_SERVER_PORT`, Value `8080` * Key: `SHM_SIZE`, Value: `5g` * Under `Models` click on the `Select` button and select the Model Catalog entry we created earlier * Under `Compute` and then `Specialty and previous generation` select the `VM.GPU.A10.1` instance From 7889529b7c5171aa0c39bd4c3d93a0e5dd711713 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 21 Aug 2024 15:18:03 +0530 Subject: [PATCH 2/8] Port Value updated --- model-deployment/containers/nim/README-MODEL-CATALOG.md | 2 +- model-deployment/containers/nim/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md index 5eab5e3a..7427ec64 100644 --- a/model-deployment/containers/nim/README-MODEL-CATALOG.md +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -42,7 +42,7 @@ Utilise the [section][https://github.com/oracle-samples/oci-data-science-ai-samp * Click on `Show advanced options` at the bottom * Select the checkbox `Use a custom container image` * Select the OCIR repository and image we pushed earlier - * Use port 8000. + * Use port 8080. * Leave CMD and Entrypoint blank * Click on `Create` button to create the model deployment diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 0b982457..5dd7e64f 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -86,7 +86,7 @@ This file will be available to container on location `/opt/ds/model/deployed_mod * Click on `Show advanced options` at the bottom * Select the checkbox `Use a custom container image` * Select the OCIR repository and image we pushed earlier - * Use port 8000. + * Use port 8080. * Leave CMD and Entrypoint blank * Click on `Create` button to create the model deployment From 8649b35267f047add31373b140662506b628fdc7 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 21 Aug 2024 15:27:33 +0530 Subject: [PATCH 3/8] Minor Updates --- model-deployment/containers/nim/Dockerfile | 12 ++++++++++++ .../containers/nim/README-MODEL-CATALOG.md | 15 ++++++++++++--- model-deployment/containers/nim/README.md | 16 +++++++--------- 3 files changed, 31 insertions(+), 12 deletions(-) create mode 100644 model-deployment/containers/nim/Dockerfile diff --git a/model-deployment/containers/nim/Dockerfile b/model-deployment/containers/nim/Dockerfile new file mode 100644 index 00000000..76f85c83 --- /dev/null +++ b/model-deployment/containers/nim/Dockerfile @@ -0,0 +1,12 @@ +FROM nvcr.io/nim/meta/llama3-8b-instruct:latest + +USER root +COPY start.sh /opt/nim/start.sh +RUN chmod +x /opt/nim/start.sh + +ENV NIM_CACHE_PATH /opt/ds/model/deployed_model +ENV NIM_SERVER_PORT 8000 +EXPOSE ${NIM_SERVER_PORT} + +ENTRYPOINT [ "/bin/bash", "--login", "-c"] +CMD ["/opt/nim/start.sh"] \ No newline at end of file diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md index 7427ec64..395d4bed 100644 --- a/model-deployment/containers/nim/README-MODEL-CATALOG.md +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -10,7 +10,16 @@ Utilising Model Catalog to store Models in OCI. We describe two ways to achieve The following are the pre-requisite: * Notebook session with internet access (Recommended) * Download the Llama 3 8B Instruct Model from [HuggingFace][https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct] or NGC repository. -* Download NIM Container image and upload to OCIR as described in README.md + +## Download NIM Container image and upload to OCIR +* Pull the latest NIM Image to local machine or through NB session. Tag it with desired name. + ```bash + docker pull nvcr.io/nim/meta/llama3-8b-instruct:latest + docker tag nvcr.io/nim/meta/llama3-8b-instruct:latest odsc-nim-llama3:latest + ``` +## OCI Container Registry + +Once you built and pushed the NIM container, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. # Method 1: Export Model to Model Catalog @@ -26,8 +35,8 @@ Utilise the [section][https://github.com/oracle-samples/oci-data-science-ai-samp # ### Create Model deploy -* To deploy the model now in the console, go back to your [OCI Data Science Project](https://cloud.oracle.com/data-science/project) - * Select the project you created earlier and then select `Model Deployment` +* To deploy the model now in the console, navigate to your [OCI Data Science Project](https://cloud.oracle.com/data-science/project) + * Select the project created earlier and then select `Model Deployment` * Click on `Create model deployment` * Under `Default configuration` set following custom environment variables * Key: `MODEL_DEPLOY_PREDICT_ENDPOINT`, Value: `/v1/completions` diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 5dd7e64f..a077513d 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -32,13 +32,12 @@ When experimenting with new frameworks and models, it is highly advisable to att * Then click `Create agent configuration` -## Download NIM Container image and upload to OCIR +## Build NIM Container for Model Deploy -* Pull the latest NIM Image to local machine or through NB session. Tag it with desired name. - ```bash - docker pull nvcr.io/nim/meta/llama3-8b-instruct:latest - docker tag nvcr.io/nim/meta/llama3-8b-instruct:latest odsc-nim-llama3:latest - ``` +* Build the OCI Model Deploy compatible container image. This process bakes in some of the configurations that makes it easier to deploy on the platform + ```bash + docker build -f Dockerfile -t odsc-nim-llama3 . + ``` ## OCI Container Registry @@ -70,14 +69,13 @@ This file will be available to container on location `/opt/ds/model/deployed_mod ### Create Model deploy -* To deploy the model now in the console, go back to your [OCI Data Science Project](https://cloud.oracle.com/data-science/project) - * Select the project you created earlier and then select `Model Deployment` +* To deploy the model now in the console, navigate to your [OCI Data Science Project](https://cloud.oracle.com/data-science/project) + * Select the project created earlier and then select `Model Deployment` * Click on `Create model deployment` * Under `Default configuration` set following custom environment variables * Key: `MODEL_DEPLOY_PREDICT_ENDPOINT`, Value: `/v1/completions` * Key: `MODEL_DEPLOY_HEALTH_ENDPOINT`, Value: `/v1/health/ready` * Key: `NGC_API_KEY_FILE`, Value: `/opt/ds/model/deployed_model/token` - * Key: `NIM_SERVER_PORT`, Value `8080` * Key: `SHM_SIZE`, Value: `5g` * Under `Models` click on the `Select` button and select the Model Catalog entry we created earlier * Under `Compute` and then `Specialty and previous generation` select the `VM.GPU.A10.1` instance From a2b98fd7f9021d2531067df843f5244872f8f967 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 21 Aug 2024 15:28:43 +0530 Subject: [PATCH 4/8] Port Value Updated --- model-deployment/containers/nim/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index a077513d..80dae843 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -35,6 +35,7 @@ When experimenting with new frameworks and models, it is highly advisable to att ## Build NIM Container for Model Deploy * Build the OCI Model Deploy compatible container image. This process bakes in some of the configurations that makes it easier to deploy on the platform + ```bash docker build -f Dockerfile -t odsc-nim-llama3 . ``` @@ -84,7 +85,7 @@ This file will be available to container on location `/opt/ds/model/deployed_mod * Click on `Show advanced options` at the bottom * Select the checkbox `Use a custom container image` * Select the OCIR repository and image we pushed earlier - * Use port 8080. + * Use port 8000. * Leave CMD and Entrypoint blank * Click on `Create` button to create the model deployment From 59cbf9abf42274e781b2fbd112886df3463cda89 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 21 Aug 2024 16:02:45 +0530 Subject: [PATCH 5/8] Review Comments addressed --- .../containers/nim/README-MODEL-CATALOG.md | 11 +++++------ model-deployment/containers/nim/README.md | 4 ++-- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md index 395d4bed..78f35d47 100644 --- a/model-deployment/containers/nim/README-MODEL-CATALOG.md +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -9,7 +9,7 @@ Utilising Model Catalog to store Models in OCI. We describe two ways to achieve The following are the pre-requisite: * Notebook session with internet access (Recommended) -* Download the Llama 3 8B Instruct Model from [HuggingFace][https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct] or NGC repository. +* Download the Llama 3 8B Instruct Model from [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) or NGC repository. ## Download NIM Container image and upload to OCIR * Pull the latest NIM Image to local machine or through NB session. Tag it with desired name. @@ -19,19 +19,19 @@ The following are the pre-requisite: ``` ## OCI Container Registry -Once you built and pushed the NIM container, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. +Once NIM container is pushed, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. # Method 1: Export Model to Model Catalog -Follow the steps mentioned [here][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/model-deployment/containers/llama2/README.md], refer the section One time download to OCI Model Catalog. +Follow the steps mentioned [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/model-deployment/containers/llama2/README.md), refer the section One time download to OCI Model Catalog. We would utilise the above created model in the next steps to create the Model Deployment. # Method 2: Model-by-reference -Follow the steps to upload your model to Object Storage [here][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md] +Follow the steps to upload your model to Object Storage [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md) -Utilise the [section][https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md] on Create Model by Reference using ADS to create the model. +Utilise the [section](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md) on Create Model by Reference using ADS to create the model. # ### Create Model deploy @@ -43,7 +43,6 @@ Utilise the [section][https://github.com/oracle-samples/oci-data-science-ai-samp * Key: `MODEL_DEPLOY_HEALTH_ENDPOINT`, Value: `/v1/health/ready` * Key: `NIM_MODEL_NAME`, Value: `/opt/ds/model/deployed_model` * Key: `NIM_SERVER_PORT`, Value `8080` - * Key: `SHM_SIZE`, Value: `5g` * Under `Models` click on the `Select` button and select the Model Catalog entry we created earlier * Under `Compute` and then `Specialty and previous generation` select the `VM.GPU.A10.1` instance * Under `Networking` choose the `Custom Networking` option and bring the VCN and subnet, which allows Internet access. diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 80dae843..51f2049b 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -7,7 +7,7 @@ This Readme walks through how to use NIM - [ Meta-Llama-3-8B-Instruct](https://h We describe two approaches to create this Model Deployment on OCI: * Download Model using API-KEY from NGC Nvidia (described below) -* Utilising Model Catalog to store Models in OCI, later used for model deployment +* Utilising Model Catalog to store Models in OCI, later used for model deployment. [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) ## Prerequisites * Access the corresponding NIM container for the model. For example for llama3, fetch the latest available image from [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama3-8b-instruct/tags). If you are a first time user, you need to sign up a developer account and wait for access to be granted to required container image. @@ -58,7 +58,7 @@ When experimenting with new frameworks and models, it is highly advisable to att ## Deploy on OCI Data Science Model Deployment -Once you built and pushed the NIM container, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. +Once NIM container is pushed, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. ### Creating Model catalog NIM container will download the model directly using publicly exposed NGC catalog APIs. To provide authorization token to download, we will save API key in a file and create a zip out of it. This zip file will then be used to create a model catalog resource. From 91ad6f484d5fc201db2f13eb981e38aada01b3c9 Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 28 Aug 2024 10:44:00 +0530 Subject: [PATCH 6/8] Hyperlinks changed --- model-deployment/containers/nim/README-MODEL-CATALOG.md | 8 ++++---- model-deployment/containers/nim/README.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md index 78f35d47..034c076e 100644 --- a/model-deployment/containers/nim/README-MODEL-CATALOG.md +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -12,7 +12,7 @@ The following are the pre-requisite: * Download the Llama 3 8B Instruct Model from [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) or NGC repository. ## Download NIM Container image and upload to OCIR -* Pull the latest NIM Image to local machine or through NB session. Tag it with desired name. +* Pull the latest NIM Image to local machine. Tag it with desired name. ```bash docker pull nvcr.io/nim/meta/llama3-8b-instruct:latest docker tag nvcr.io/nim/meta/llama3-8b-instruct:latest odsc-nim-llama3:latest @@ -23,15 +23,15 @@ Once NIM container is pushed, you can now use the `Bring Your Own Container` Dep # Method 1: Export Model to Model Catalog -Follow the steps mentioned [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/model-deployment/containers/llama2/README.md), refer the section One time download to OCI Model Catalog. +Follow the steps mentioned [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/model-deployment/containers/llama2/README.md#model-store-export-api-for-creating-model-artifacts-greater-than-6-gb-in-size)), refer the section One time download to OCI Model Catalog. We would utilise the above created model in the next steps to create the Model Deployment. # Method 2: Model-by-reference -Follow the steps to upload your model to Object Storage [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md) +Follow the steps to upload your model to Object Storage [here](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md#upload-model-to-oci-object-storage) -Utilise the [section](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md) on Create Model by Reference using ADS to create the model. +Utilise the [section](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/LLM/llama3.1-8B-deployment-vLLM-container.md#create-model-by-reference-using-ads) on Create Model by Reference using ADS to create the model. # ### Create Model deploy diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 51f2049b..120a77f9 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -7,7 +7,7 @@ This Readme walks through how to use NIM - [ Meta-Llama-3-8B-Instruct](https://h We describe two approaches to create this Model Deployment on OCI: * Download Model using API-KEY from NGC Nvidia (described below) -* Utilising Model Catalog to store Models in OCI, later used for model deployment. [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) +* Utilising Object storage to store the model and creating a model catalog pointing to Object storage bucket [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) ## Prerequisites * Access the corresponding NIM container for the model. For example for llama3, fetch the latest available image from [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama3-8b-instruct/tags). If you are a first time user, you need to sign up a developer account and wait for access to be granted to required container image. @@ -58,7 +58,7 @@ When experimenting with new frameworks and models, it is highly advisable to att ## Deploy on OCI Data Science Model Deployment -Once NIM container is pushed, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model. +Once you built and pushed the NIM container, you can now use the `Bring Your Own Container` Deployment in OCI Data Science to deploy the Llama3 model ### Creating Model catalog NIM container will download the model directly using publicly exposed NGC catalog APIs. To provide authorization token to download, we will save API key in a file and create a zip out of it. This zip file will then be used to create a model catalog resource. From 1f74851f1cac8700ed76f01855d1cc77774b635a Mon Sep 17 00:00:00 2001 From: gagachau Date: Wed, 28 Aug 2024 10:46:35 +0530 Subject: [PATCH 7/8] Minor Changes --- model-deployment/containers/nim/README-MODEL-CATALOG.md | 2 +- model-deployment/containers/nim/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/model-deployment/containers/nim/README-MODEL-CATALOG.md b/model-deployment/containers/nim/README-MODEL-CATALOG.md index 034c076e..a3a71bb1 100644 --- a/model-deployment/containers/nim/README-MODEL-CATALOG.md +++ b/model-deployment/containers/nim/README-MODEL-CATALOG.md @@ -3,7 +3,7 @@ Utilising Model Catalog to store Models in OCI. We describe two ways to achieve this: * Storing zipped model file in Model Catalog -* Utilising Model-by-reference Feature to download model +* Utilising Object storage to store the model and creating a model catalog pointing to Object storage bucket [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) # Pre-requisite diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index 120a77f9..a4ffd742 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -7,7 +7,7 @@ This Readme walks through how to use NIM - [ Meta-Llama-3-8B-Instruct](https://h We describe two approaches to create this Model Deployment on OCI: * Download Model using API-KEY from NGC Nvidia (described below) -* Utilising Object storage to store the model and creating a model catalog pointing to Object storage bucket [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) +* Utilising Object storage to store the model and creating a model catalog pointing to Object storage bucket [Refer](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/nim/README-MODEL-CATALOG.md) ## Prerequisites * Access the corresponding NIM container for the model. For example for llama3, fetch the latest available image from [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama3-8b-instruct/tags). If you are a first time user, you need to sign up a developer account and wait for access to be granted to required container image. From b3243fbd350f504b4cb30b0e55153f7c52ad727e Mon Sep 17 00:00:00 2001 From: Yash Pandit Date: Wed, 28 Aug 2024 17:24:18 +0530 Subject: [PATCH 8/8] Added README-SOURCE-NIM-TO-OCIR.MD --- .../nim/README-SOURCE-NIM-TO-OCIR.MD | 75 +++++++++++++++++++ model-deployment/containers/nim/README.md | 5 ++ 2 files changed, 80 insertions(+) create mode 100644 model-deployment/containers/nim/README-SOURCE-NIM-TO-OCIR.MD diff --git a/model-deployment/containers/nim/README-SOURCE-NIM-TO-OCIR.MD b/model-deployment/containers/nim/README-SOURCE-NIM-TO-OCIR.MD new file mode 100644 index 00000000..525bde4a --- /dev/null +++ b/model-deployment/containers/nim/README-SOURCE-NIM-TO-OCIR.MD @@ -0,0 +1,75 @@ + + +## Part 1: Pull NIM image from Nvidia to your local machine + +### Step 1: Get access to NIM image login to docker: + +Register to developer account at https://build.nvidia.com/explore/discover. After logging in you can get the ngc token by going in the model page and clicking generate token button. + +In place of password place your newly generated NGC token. + +``` +$ docker login nvcr.io +Username: $oauthtoken +Password: nvapi-6mj...... +``` + +### Step 2: Pull image to your local machine: +``` +Choose a container name for bookkeeping +$ export CONTAINER_NAME=llama3-8b-instruct + +Define the vendor name for the LLM +$ export VENDOR_NAME=meta + +$ export IMG_NAME="nvcr.io/nim/${VENDOR_NAME}/${CONTAINER_NAME}:latest" + +$ docker pull $IMG_NAME +``` + +## Part 2: Push this image to ocir: + +### Step 1: Login to ocir docker: + +To get your registry domain check: https://docs.oracle.com/en-us/iaas/Content/Registry/Concepts/registryprerequisites.htm#regional-availability +If you dont have login details please check: https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrypushingimagesusingthedockercli.htm#:~:text=If%20you%20already%20have%20an%20auth%20token%2C%20go%20to%20the%20next%20step.%20Otherwise%3A + +``` +$ docker login +``` +When prompted for a username, enter your username in the format ``````/``````, where `````` is the auto-generated Object Storage namespace string of your tenancy (as shown on the Tenancy Information page). +For example, ansh81vru1zp/jdoe@acme.com. +When prompted for a password, enter your auth token. + + + +### Step 2: Locate the image on the client machine that you want to push: +``` +$ docker images +``` +``` +Ex output for our image: + +REPOSITORY TAG IMAGE ID +nvcr.io/nim/meta/llama3-8b-instruct latest 106df041c287 +``` + +### Step 3: Tag the docker image +``` +$ docker tag +``` + +`````` is the fully qualified path to the target location in Container Registry where you want to push the image, in the format ```//:``` + +``` +Ex command: + +$ docker tag 106df041c287 ocir.us-ashburn-1.oci.oraclecloud.com/ansh81vru1zp/project01/acme-web-app:v2.0.test +``` + +### Setp 4: push to ocir +``` +$ docker push +``` + +Note: For more information on pushing image to OCIR refer: https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrypushingimagesusingthedockercli.htm \ No newline at end of file diff --git a/model-deployment/containers/nim/README.md b/model-deployment/containers/nim/README.md index a4ffd742..39fa01d1 100644 --- a/model-deployment/containers/nim/README.md +++ b/model-deployment/containers/nim/README.md @@ -40,6 +40,8 @@ When experimenting with new frameworks and models, it is highly advisable to att docker build -f Dockerfile -t odsc-nim-llama3 . ``` +##### To directly get image from Nvidia NIM catalogue and upload to OCIR check: ```./README-SOURCE-NIM-TO-OCIR.MD``` + ## OCI Container Registry * You need to `docker login` to the Oracle Cloud Container Registry (OCIR) first, if you haven't done so before been able to push the image. To login, you have to use your [API Auth Token](https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrygettingauthtoken.htm) that can be created under your `Oracle Cloud Account->Auth Token`. You need to login only once. @@ -55,6 +57,9 @@ When experimenting with new frameworks and models, it is highly advisable to att ```bash docker push `odsc-nim-llama3:latest` ``` + + +##### To directly get image from Nvidia NIM catalogue and upload to OCIR check: ```./README-SOURCE-NIM-TO-OCIR.MD``` ## Deploy on OCI Data Science Model Deployment