Is the Azure regional endpoint format supported?

[sorin-costea]

I'm using CrewAI with a Azure OpenAI endpoint. Different from any documentation in existence (including Azure's) I get the Azure endpoints created in regional format, like this:

https://my_region.api.cognitive.microsoft.com/openai/deployments/my_deployment/chat/completions?api-version=2023-03-15-preview

As Azure moves everything to cognitive, there's no way for me to create an endpoint in the old resource format everybody keeps mentioning. And with this regional one, I get the following error:

2024-10-01 10:27:30,733 - 38524 - llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.APIError: AzureException APIError - argument of type 'NoneType' is not iterableTraceback (most recent call last):
  File ".\pypoetry\Cache\virtualenvs\my-proj-B-au1vhR-py3.12\Lib\site-packages\litellm\llms\AzureOpenAI\azure.py", line 607, in completion
    if "gateway.ai.cloudflare.com" in api_base:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

I'm using CrewAI code from the howtos so that cannot be, but what do I know... CrewAI folks believe the endpoint format is not supported, and the LangChain bot also commented to check the status here.

from langchain_openai import AzureChatOpenAI
azure_llm = AzureChatOpenAI(
    azure_deployment='azure/my_deployment',
    azure_endpoint='https://my_region.api.cognitive.microsoft.com/',
    api_version='2023-03-15-preview',
    api_key='my_key'
)

PS: litellm==1.48.2

[ishaan-jaff]

@sorin-costea this should already work - the error you're seeing looks like your api_base passed to LiteLLM is None

• can you share code of how you're using litellm
• what variables are you passing to litellm / setting for your Azure endpoint ?

[sorin-costea]

Thank you for your time!
The only code, including the initialization variables, is the one above. I'm not using litellm directly, but initializing in CrewAI the AzureChatOpenAI component of Langchain. Of course the initialization might be wrong or incomplete, but both CrewAI and Langchain communities suggested to chat with you about the endpoint format. Now, if you confirm the endpoint is supported and above parameters look okay, or you already notice something missing, I will take it from there...

[sorin-costea]

If I use the CrewAI's direct LLM initialization, it works fine. So I assume the problem is in Langchain's AzureChatOpenAI, thus probably it's safe to close this issue.

Working code:

from crewai import LLM
azure_llm = LLM(
    model="azure/my_deployment",
    base_url="https://my_region.api.cognitive.microsoft.com/openai/deployments/my_deployment/chat/completions?api-version=some_version",
    api_key=api_key
)

[m-o-w]

This works. Thank you very much!!

[cmosguy]

damn, this took a while and it just helped me too

[sglemmestad]

where did you put your updated code, @sorin-costea? :)

[wlhong-allo]

thanks it took me 2 hours.. and now worked!

[sorin-costea]

@sglemmestad as I'm using CrewAI, in its Agent initialization:

return Agent(
    llm = azure_llm,
    ...

[sglemmestad]

Thanks, @sorin-costea!

Works perfectly!

I am using the crewai demo report crew, so i just updated the crew.py file like this:

I did also remove the content inside of the .env file, so its empty.

Again. thank you so much! :D

[sglemmestad]

To not upload my key to git, i updated like this:

crew.py

.env

[sorin-costea]

Idea: if you're using Git Codespaces, you can place your keys in the User Secrets section (in your user/settings/codespaces)