Stream options are not available for azure openAI?

[SAIVENKATARAJU]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

I was following this article on stream option.https://cookbook.openai.com/examples/how_to_stream_completions.

I am using:

"2023-05-15", "gpt-35-turbo"

code:

response = open_ai_client.chat.completions.create(
    model=OpenAIConstants.generator,
    messages=[
        {'role': 'user', 'content': "What's 1+1? Answer in one word."}
    ],
    temperature=0,
    stream=True,
    stream_options={"include_usage": True}, # retrieving token usage for stream response
)

for chunk in response:
    print(f"choices: {chunk.choices}\nusage: {chunk.usage}")
    print("****************")

Traceback (most recent call last):
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\Archive\sample.py", line 15, in <module>
    response = open_ai_client.chat.completions.create(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_utils\_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\resources\chat\completions.py", line 590, in create
    return self._post(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 921, in request
    return self._request(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Unrecognized request argument supplied: stream_options', 'type': 'invalid_request_error', 'param': None, 'code': None}}

To Reproduce

I was following this article on stream option.https://cookbook.openai.com/examples/how_to_stream_completions.

I am using:

"2023-05-15", "gpt-35-turbo"

code:

response = open_ai_client.chat.completions.create(
    model=OpenAIConstants.generator,
    messages=[
        {'role': 'user', 'content': "What's 1+1? Answer in one word."}
    ],
    temperature=0,
    stream=True,
    stream_options={"include_usage": True}, # retrieving token usage for stream response
)

for chunk in response:
    print(f"choices: {chunk.choices}\nusage: {chunk.usage}")
    print("****************")

Traceback (most recent call last):
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\Archive\sample.py", line 15, in <module>
    response = open_ai_client.chat.completions.create(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_utils\_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\resources\chat\completions.py", line 590, in create
    return self._post(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 921, in request
    return self._request(
  File "C:\Users\nandurisai.venkatara\projects\knowledge-base\.venv\lib\site-packages\openai\_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Unrecognized request argument supplied: stream_options', 'type': 'invalid_request_error', 'param': None, 'code': None}}

Code snippets

No response

OS

Win

Python version

3.10

Library version

1.30.1

[kristapratico]

@SAIVENKATARAJU Azure OpenAI doesn't support stream_options yet. You can follow the "what's new" page for updates on when it will be supported: https://learn.microsoft.com/en-us/azure/ai-services/openai/whats-new

[SAIVENKATARAJU]

@kristapratico So there was no other option to get usage when streaming is enabled?

[apmavrin]

Hi @kristapratico, can you share any estimations on when Azure OpenAI Rest API will start supporting the stream_options parameter? The `langchain-openai, starting from version 0.1.9, automatically adds the stream_options parameter to the request if the streaming is set to true. This made all our streaming response services crash.

Workaround for us was to downgrade the langchain-openai to version 0.1.8

[kristapratico]

@apmavrin I do not have an ETA, but I am sharing your feedback with the team.

The `langchain-openai, starting from version 0.1.9, automatically adds the stream_options parameter to the request if the streaming is set to true. This made all our streaming responses service to crash.

Can you share an issue or a repro for this? I wasn't able to reproduce this with version 0.1.10 + AzureChatOpenAI.

[apmavrin]

Hi @kristapratico

langchain-openai 0.1.10

chat_llm is of type ChatOpenAI and we instantiate it without supplying the stream_options parameter, so we anticipate it won't be included in the request at all.

chat_llm=self.model_provider.get_chat_open_ai(
                        tracing_context=tracing_context,
                        model=model,
                        streaming=True,
                    )

//invoking the stream function
for chunk in chat_llm.stream(chat_messages)

When you look into this file from the langchain-openai: langchain_openai/chat_models/base.py, on lines 1664-1669 you'll see that langchain automatically adds the stream_options parameter

    def _stream(
        self, *args: Any, stream_usage: Optional[bool] = None, **kwargs: Any
    ) -> Iterator[ChatGenerationChunk]:
        """Set default stream_options."""
        stream_usage = self._should_stream_usage(stream_usage, **kwargs)
        kwargs["stream_options"] = {"include_usage": stream_usage}

So, the request to AzureOpenAI now looks like this:

b'{"messages": [{"content": "removing our data from here", "role": "system"}, {"content": "Question: Who was the 2 president of the USA?", "role": "user"}], "model": "gpt-35-turbo-16k", "n": 1, "stream": true, "stream_options": {"include_usage": false}, "temperature": 0.0}'

and the AzureOpenAI returns the 400 - Bad Request error due to the presence of the unknown parameter stream_options in the request

Error code: 400 - {'error': {'code': None, 'message': 'Unrecognized request argument supplied: stream_options', 'param': None, 'type': 'invalid_request_error'}}

If I comment out this opinionated line kwargs["stream_options"] = {"include_usage": stream_usage} in the `base.py file, then the parameter won't be included in the request, and everything will work fine. This is not a solution, of course, even not a workaround.

[kristapratico]

chat_llm=self.model_provider.get_chat_open_ai(
                        tracing_context=tracing_context,
                        model=model,
                        streaming=True,
                    )

@apmavrin thanks. Is this custom code or does langchain provide this somewhere? I'm having trouble understanding how we end up with ChatOpenAI instead of the AzureChatOpenAI wrapper while targeting the Azure OpenAI service. Looking at the PR where stream_options was added, it looks like it intentionally wasn't added to the Azure wrapper since the author of the PR understood it wasn't supported there yet. What's the use case for using ChatOpenAI with Azure?

I think it's fair to only add stream_options to the request if it's set to True (I'm happy to open a PR to start the discussion), but I would be interested in understanding what AzureChatOpenAI is lacking in that we need to use ChatOpenAI.

[aemaem]

According to the documentation of Azure OpenAI REST API there simply is no such option.

However, the documentation of the OpenAI REST API lists such an option.

We would also appreciate if Azure would provide this functionality of stream_options as well, as we need the exact usage of a request.

[apmavrin]

Hi @kristapratico,

This is our code. We use the proxy to control which provider to forward the request. As of today, it's AzureOpenAI service. So all requests land on MS territory.

Azure OpenAI REST API lacks support for the stream_options parameter that 'native' OpenAI supports. If MS can implement it, this will solve all issues :)

I think it's fair to only add stream_options to the request if it's set to True

yes, exactly. This was also our assumption. Unless it's specified, do not modify the request, Langchain ;)

[kristapratico]

@apmavrin thanks for confirming the use case. I'm sharing the customer signal we're getting in this thread and working on trying to get an ETA I can share for when stream_options will land for Azure. In the meantime, I'll submit a PR to langchain to see if we can unblock your scenario.

[ccurme]

Hi all, Langchain released @kristapratico's change in langchain-openai==0.1.11.

We maintain AzureChatOpenAI in the same langchain-openai library in part for this reason-- so that we can manage access to features separately for Azure.

I'm interested in knowing: why not use AzureChatOpenAI? This would help us understand if we need to enforce this constraint going forward.

cc @SAIVENKATARAJU @apmavrin

Thanks for your insight.

[sergiogarlez]

Hi all,

Following. I'm also interested in the Azure OpenAI REST API supporting stream_options for token usage.

Thanks in advance!

[rattrayalex]

This is a feature request for the Azure OpenAI API, not a bug report for this SDK, so I'm closing this issue.

[liyangwd]

I hit some error when chat to AzureOpenAI with image langchain-openai==0.1.23 langchain-core==0.2.38
here is my code:

from langchain_openai import AzureChatOpenAI
import asyncio
from langchain_core.messages import HumanMessage

llm = AzureChatOpenAI(
    api_key="xxxx",
    azure_endpoint="https://xxxxxx.openai.azure.com/",
    api_version="2023-03-15-preview",
    openai_api_type="azure",
    azure_deployment="gpt-4o",
    model_name="gpt-4o",
    temperature=0,
    stream=True,
    stream_options={"include_usage": True},
    # model_kwargs={"stream_options": {"include_usage": True}}
)

req = [HumanMessage(content=[{'type': 'text', 'text': "what's the pic describe"}, {'type': 'image_url', 'image_url': {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/7/70/Snow_Man.jpg/500px-Snow_Man.jpg"}}])]

req_2 = [HumanMessage(content=[{'type': 'text', 'text': 'tell me a joke'}])]


async def fetch_joke():
    async for event in llm.astream_events(req, version="v2"):
        if event["event"] == "on_chat_model_end":
            print(f'Token usage: {event["data"]["output"].usage_metadata}\n')
        elif event["event"] == "on_chat_model_stream":
            chunk = event["data"]["chunk"]
            print(chunk)
        else:
            pass

asyncio.run(fetch_joke())

the same code use param req_2 is ok

async def fetch_joke():
    async for event in llm.astream_events(req_2, version="v2"):
        if event["event"] == "on_chat_model_end":
            print(f'Token usage: {event["data"]["output"].usage_metadata}\n')
        elif event["event"] == "on_chat_model_stream":
            chunk = event["data"]["chunk"]
            print(chunk)
        else:
            pass

asyncio.run(fetch_joke())

is this the azure's problem?????