Streaming custom LLM with NemoGuardrails

[nicolemzh]

What are the requirements to stream a custom LLM with NemoGuardrails?

The custom LLM is working with NemoGuardrails without the streaming function. However, I'm having an issue implementing streaming with stream_async or the StreamingHandler. I was wondering if anyone has tried this task and if there are extra steps to stream with a custom LLM compared to GPT or LangChain through NemoGuardrails.

[drazvan]

Hi @nicolemzh ! The NeMo LLM provider is a good example that also supports streaming: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/llm/providers/nemollm.py. In a custom LLM, it needs to be implemented similarly. Let me know if you need more guidance.

[trebedea]

@JanviAg , Langchain has an integration with Google models, including Gemini: https://python.langchain.com/docs/integrations/chat/google_generative_ai/

It is way easier to use this wrapper than create one yourself. We can assist you if you really need to create a new one (with minimal functionality).

[JanviAg]

@trebedea I am also using aws. I already have completion end points implemented in my project. By using custom llm I wanted to use my completions. This is possible by using custom llm, is my understanding correct?

[trebedea]

I need more details to understand the LLM providers you are using. Creating custom LLM interfaces for NeMo Guardrails should only be needed when there is no existing Langchain provider. Amazon Bedrock already has a Langchain provider: https://python.langchain.com/docs/integrations/chat/bedrock/

Even the Nemo LLM example for NVIDIA models has been removed as there is a Langchain integration available now.

[JanviAg]

@trebedea

That makes sense. In my case, I’m not using the native Google or Bedrock SDKs—instead, I have custom completion endpoints for both Gemini and AWS models, exposed via my own LLM Gateway service.

These endpoints follow an OpenAI-compatible completion format, so my goal is to let NeMo Guardrails call them directly through a custom LLM provider.

I understand that if I were using the official Google or Bedrock APIs, I could simply rely on LangChain’s existing integrations (like ChatGoogleGenerativeAI or ChatBedrock).

However, since these endpoints are internal and not part of LangChain’s supported list, could you please confirm if creating a minimal CustomLLM subclass (inheriting from BaseLLM) is still the recommended approach for integrating such custom endpoints with NeMo Guardrails?

[trebedea]

If your endpoints are OpenAI compatible, the simplest thing is to use the OpenAI Langchain wrapper and just provide your specific endpoint URL as a parameter for base_url attribute. The docs are not explicit, but this is the intent of base_url: https://python.langchain.com/api_reference/openai/llms/langchain_openai.llms.base.OpenAI.html