Skip to content

Commit 94109b1

Browse files
shapershaper
authoredFeb 3, 2025··
feat (examples): add example and docs on NIM use via openai-compatible provider (#4666)
·
@ai-sdk/xai@1.2.3ai@4.1.18
1 parent c7b6c0c commit 94109b1

File tree

7 files changed

+250
-0
lines changed

7 files changed

+250
-0
lines changed
 

‎content/providers/02-openai-compatible-providers/35-nim.mdx

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
title: NVIDIA NIM
3+
description: Use NVIDIA NIM OpenAI compatible API with the AI SDK.
4+
---
5+
6+
# NVIDIA NIM Provider
7+
8+
[NVIDIA NIM](https://www.nvidia.com/en-us/ai/) provides optimized inference microservices for deploying foundation models. It offers an OpenAI-compatible API that you can use with the AI SDK.
9+
10+
## Setup
11+
12+
The NVIDIA NIM provider is available via the `@ai-sdk/openai-compatible` module as it is compatible with the OpenAI API.
13+
You can install it with:
14+
15+
<Tabs items={['pnpm', 'npm', 'yarn']}>
16+
<Tab>
17+
<Snippet text="pnpm add @ai-sdk/openai-compatible" dark />
18+
</Tab>
19+
<Tab>
20+
<Snippet text="npm install @ai-sdk/openai-compatible" dark />
21+
</Tab>
22+
<Tab>
23+
<Snippet text="yarn add @ai-sdk/openai-compatible" dark />
24+
</Tab>
25+
</Tabs>
26+
27+
## Provider Instance
28+
29+
To use NVIDIA NIM, you can create a custom provider instance with the `createOpenAICompatible` function from `@ai-sdk/openai-compatible`:
30+
31+
```ts
32+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
33+
34+
const nim = createOpenAICompatible({
35+
name: 'nim',
36+
baseURL: 'https://integrate.api.nvidia.com/v1',
37+
headers: {
38+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
39+
},
40+
});
41+
```
42+
43+
<Note>
44+
You can obtain an API key and free credits by registering at [NVIDIA
45+
Build](https://build.nvidia.com/explore/discover). New users receive 1,000
46+
inference credits to get started.
47+
</Note>
48+
49+
## Language Models
50+
51+
You can interact with NIM models using a provider instance. For example, to use [DeepSeek-R1](https://build.nvidia.com/deepseek-ai/deepseek-r1), a powerful open-source language model:
52+
53+
```ts
54+
const model = nim.chatModel('deepseek-ai/deepseek-r1');
55+
```
56+
57+
### Example - Generate Text
58+
59+
You can use NIM language models to generate text with the `generateText` function:
60+
61+
```ts
62+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
63+
import { generateText } from 'ai';
64+
65+
const nim = createOpenAICompatible({
66+
name: 'nim',
67+
baseURL: 'https://integrate.api.nvidia.com/v1',
68+
headers: {
69+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
70+
},
71+
});
72+
73+
const { text, usage, finishReason } = await generateText({
74+
model: nim.chatModel('deepseek-ai/deepseek-r1'),
75+
prompt: 'Tell me the history of the San Francisco Mission-style burrito.',
76+
});
77+
78+
console.log(text);
79+
console.log('Token usage:', usage);
80+
console.log('Finish reason:', finishReason);
81+
```
82+
83+
### Example - Stream Text
84+
85+
NIM language models can also generate text in a streaming fashion with the `streamText` function:
86+
87+
```ts
88+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
89+
import { streamText } from 'ai';
90+
91+
const nim = createOpenAICompatible({
92+
name: 'nim',
93+
baseURL: 'https://integrate.api.nvidia.com/v1',
94+
headers: {
95+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
96+
},
97+
});
98+
99+
const result = streamText({
100+
model: nim.chatModel('deepseek-ai/deepseek-r1'),
101+
prompt: 'Tell me the history of the Northern White Rhino.',
102+
});
103+
104+
for await (const textPart of result.textStream) {
105+
process.stdout.write(textPart);
106+
}
107+
108+
console.log();
109+
console.log('Token usage:', await result.usage);
110+
console.log('Finish reason:', await result.finishReason);
111+
```
112+
113+
NIM language models can also be used with other AI SDK functions like `generateObject` and `streamObject`.
114+
115+
<Note>
116+
Model support for tool calls and structured object generation varies. For
117+
example, the
118+
[`meta/llama-3.3-70b-instruct`](https://build.nvidia.com/meta/llama-3_3-70b-instruct)
119+
model supports object generation capabilities. Check each model's
120+
documentation on NVIDIA Build for specific supported features.
121+
</Note>

‎content/providers/02-openai-compatible-providers/index.mdx

+1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ Below we focus on the general setup and provider instance creation. You can also
1212
We provide detailed documentation for the following OpenAI compatible providers:
1313

1414
- [LM Studio](/providers/openai-compatible-providers/lmstudio)
15+
- [NIM](/providers/openai-compatible-providers/nim)
1516
- [Baseten](/providers/openai-compatible-providers/baseten)
1617

1718
The general setup and provider instance creation is the same for all of these providers.

‎examples/ai-core/src/generate-object/nim.ts

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2+
import { generateObject } from 'ai';
3+
import { z } from 'zod';
4+
import 'dotenv/config';
5+
6+
async function main() {
7+
const nim = createOpenAICompatible({
8+
baseURL: 'https://integrate.api.nvidia.com/v1',
9+
name: 'nim',
10+
headers: {
11+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
12+
},
13+
});
14+
const model = nim.chatModel('meta/llama-3.3-70b-instruct');
15+
const result = await generateObject({
16+
model,
17+
schema: z.array(
18+
z.object({
19+
name: z.string(),
20+
breed: z.string(),
21+
}),
22+
),
23+
prompt:
24+
'Generate 10 cat names and breeds for a fictional book about a world where cats rule shrimp.',
25+
});
26+
27+
console.log(JSON.stringify(result.object, null, 2));
28+
console.log();
29+
console.log('Token usage:', result.usage);
30+
console.log('Finish reason:', result.finishReason);
31+
}
32+
33+
main().catch(console.error);

‎examples/ai-core/src/generate-text/nim.ts

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2+
import { generateText } from 'ai';
3+
import 'dotenv/config';
4+
5+
async function main() {
6+
const nim = createOpenAICompatible({
7+
baseURL: 'https://integrate.api.nvidia.com/v1',
8+
name: 'nim',
9+
headers: {
10+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
11+
},
12+
});
13+
const model = nim.chatModel('deepseek-ai/deepseek-r1');
14+
const result = await generateText({
15+
model,
16+
prompt: 'Tell me the history of the San Francisco Mission-style burrito.',
17+
});
18+
19+
console.log(result.text);
20+
console.log();
21+
console.log('Token usage:', result.usage);
22+
console.log('Finish reason:', result.finishReason);
23+
}
24+
25+
main().catch(console.error);

‎examples/ai-core/src/stream-object/nim.ts

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2+
import { streamObject } from 'ai';
3+
import { z } from 'zod';
4+
import 'dotenv/config';
5+
6+
async function main() {
7+
const nim = createOpenAICompatible({
8+
baseURL: 'https://integrate.api.nvidia.com/v1',
9+
name: 'nim',
10+
headers: {
11+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
12+
},
13+
});
14+
const model = nim.chatModel('meta/llama-3.3-70b-instruct');
15+
const result = streamObject({
16+
model,
17+
schema: z.object({
18+
characters: z.array(
19+
z.object({
20+
name: z.string(),
21+
class: z
22+
.string()
23+
.describe('Character class, e.g. warrior, mage, or thief.'),
24+
description: z.string(),
25+
}),
26+
),
27+
}),
28+
prompt:
29+
'Generate 3 character descriptions for a fantasy role playing game.',
30+
});
31+
32+
for await (const partialObject of result.partialObjectStream) {
33+
console.clear();
34+
console.log(partialObject);
35+
}
36+
37+
console.log();
38+
console.log('Token usage:', await result.usage);
39+
}
40+
41+
main().catch(console.error);

‎examples/ai-core/src/stream-text/nim.ts

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2+
import { streamText } from 'ai';
3+
import 'dotenv/config';
4+
5+
async function main() {
6+
const nim = createOpenAICompatible({
7+
baseURL: 'https://integrate.api.nvidia.com/v1',
8+
name: 'nim',
9+
headers: {
10+
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
11+
},
12+
});
13+
const model = nim.chatModel('deepseek-ai/deepseek-r1');
14+
const result = streamText({
15+
model,
16+
prompt: 'Tell me the history of the Northern White Rhino.',
17+
});
18+
19+
for await (const textPart of result.textStream) {
20+
process.stdout.write(textPart);
21+
}
22+
23+
console.log();
24+
console.log('Token usage:', await result.usage);
25+
console.log('Finish reason:', await result.finishReason);
26+
}
27+
28+
main().catch(console.error);

‎turbo.json

+1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
"LUMA_API_KEY",
3030
"MISTRAL_API_KEY",
3131
"NEXT_RUNTIME",
32+
"NIM_API_KEY",
3233
"NODE_ENV",
3334
"OPENAI_API_KEY",
3435
"OPENAI_API_BASE",

0 commit comments

Comments
 (0)
Please sign in to comment.