feat: OpenAI embeddings with GPU based KNN #2157
Conversation
Added OpenAI embeddings with GPU based KNN using NVIDIA Rapids
|
Hey @vonodiripsa 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
Please clear the output in this notebook before checking it in so that the diff is minimal |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Well also want to try to get this init script to run on the databricks clusters we spin up so that it tests properly. Can you add the init script to a file in say the tools/init_scripts directory. That way we can just link people to it, and we can upload it during the build. Well also want to add this to the GPU tests on databricks, see the nbtest folder for pointers to the GPU databricks test runner |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2157 +/- ##
==========================================
- Coverage 84.49% 84.47% -0.03%
==========================================
Files 325 325
Lines 16959 16959
Branches 1524 1524
==========================================
- Hits 14330 14326 -4
- Misses 2629 2633 +4 ☔ View full report in Codecov by Sentry. |
Added init script to install repids ml using cuda 11.8
vonodiripsa
changed the title
vonodiripsa
changed the title
|
Corrected the semantic prefix and added init script |
vonodiripsa
changed the title
|
can you remove output from the notebook please? |
Removed outputs
|
@microsoft-github-policy-service agree company="Microsoft" |
With GPU KNN notebook test code
Added GPU test code to OpenAI with KNN notebook
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@microsoft-github-policy-service agree company="NVIDIA" |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Fixed style errors
Suggested by Mark
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| .filterNot(_.getAbsolutePath.contains("Fine-tune")) | ||
| .filterNot(_.getAbsolutePath.contains("GPU")) | ||
| .filterNot(_.getAbsolutePath.contains("Explanation Dashboard")) // TODO Remove this exclusion | ||
|
|
||
| val GPUNotebooks: Seq[File] = ParallelizableNotebooks.filter(_.getAbsolutePath.contains("Fine-tune")) | ||
| val GPUNotebooks: Seq[File] = ParallelizableNotebooks | ||
| .filter(file => | ||
| file.getAbsolutePath.contains("GPU")) |
There was a problem hiding this comment.
please keep fine-tine in the filternots, you had it right last time
There was a problem hiding this comment.
you need to set this back to the || expression you had
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
Show resolved
Hide resolved
…bricksUtilities.scala Added "Fine-tune" again Co-authored-by: Mark Hamilton <[email protected]>
Added Fine-tune back
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Create cluster using init script
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Corrected parameters
Added Rapids cluster name
|
|
||
| class DatabricksRapidsTests extends DatabricksTestHelper { | ||
|
|
||
| val clusterId: String = createClusterInPool(GPUClusterName, AdbGpuRuntime, 2, GpuPoolId, RapidsInitScripts) |
There was a problem hiding this comment.
| val clusterId: String = createClusterInPool(GPUClusterName, AdbGpuRuntime, 2, GpuPoolId, RapidsInitScripts) | |
| val clusterId: String = createClusterInPool(GPUClusterName, AdbGpuRuntime, 1, GpuPoolId, RapidsInitScripts) |
Reduced number of nodes to 1
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Fixed imports
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Added new OpenAI embeddings Quickstart demo with GPU based KNN using NVIDIA Rapids
Related Issues/PRs
#xxx
What changes are proposed in this pull request?
It is a new docs notebook demonstrating usage of NVIDIA Rabids KNN on GPU.
How is this patch tested?
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?