Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Error with StructuredOutputs and Fireworks.AI models #6797

Open
DaveDeCaprio opened this issue Nov 18, 2024 · 1 comment · May be fixed by #6802
Open

[Bug]: Error with StructuredOutputs and Fireworks.AI models #6797

DaveDeCaprio opened this issue Nov 18, 2024 · 1 comment · May be fixed by #6802
Labels
bug Something isn't working

Comments

@DaveDeCaprio
Copy link

What happened?

When using a Pydantic class as a reponse_format with fireworks.ai I'm getting an error:

`class BooleanResponse(BaseModel):
result: bool

def test_fireworks_structured_output():
resp = litellm.completion(
model="fireworks_ai/accounts/fireworks/models/llama-v3p2-90b-vision-instruct",
messages=[{"role": "user", "content": "Is 2+2=4?"}],
response_format=BooleanResponse,
)
logger.info(f"\nResponse: {resp}")
`
Based on the Fireworks docs: https://docs.fireworks.ai/structured-responses/structured-response-formatting

I can see this is being passed as the reponse_format:
{
'type': 'json_schema',
'json_schema': {
'schema': {
'properties': {
'result': {
'title': 'Result',
'type': 'boolean'
}
},
'required': [
'result'
],
'title': 'BooleanResponse',
'type': 'object',
'additionalProperties': False
},
'name': 'BooleanResponse',
'strict': True
}
}

Based on the error, we need to change to:
{
'type': 'json_object',
'schema': {
'schema': {
'properties': {
'result': {
'title': 'Result',
'type': 'boolean'
}
},
'required': [
'result'
],
'title': 'BooleanResponse',
'type': 'object',
'additionalProperties': False
},
'name': 'BooleanResponse',
'strict': True
}
}

Relevant log output

LiteLLM completion() model= accounts/fireworks/models/llama-v3p2-90b-vision-instruct; provider = fireworks_ai
2024-11-18 14:09:34,986 LiteLLM INFO - 
LiteLLM completion() model= accounts/fireworks/models/llama-v3p2-90b-vision-instruct; provider = fireworks_ai
2024-11-18 14:09:35,298 httpx INFO - HTTP Request: POST https://api.fireworks.ai/inference/v1/chat/completions "HTTP/1.1 400 Bad Request"

mbue/shared/llm/llm_client_test.py:50 (test_structured_output[fireworks_ai/accounts/fireworks/models/llama-v3p2-90b-vision-instruct])
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py:911: in acompletion
    headers, response = await self.make_openai_chat_completion_request(
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py:618: in make_openai_chat_completion_request
    raise e
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py:606: in make_openai_chat_completion_request
    await openai_aclient.chat.completions.with_raw_response.create(
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/openai/_legacy_response.py:373: in wrapped
    return cast(LegacyAPIResponse[R], await func(*args, **kwargs))
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/openai/resources/chat/completions.py:1661: in create
    return await self._post(
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/openai/_base_client.py:1839: in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/openai/_base_client.py:1533: in request
    return await self._request(
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/openai/_base_client.py:1634: in _request
    raise self._make_status_error_from_response(err.response) from None
E   openai.BadRequestError: Error code: 400 - {'error': {'object': 'error', 'type': 'invalid_request_error', 'message': "2 request validation errors: Input should be 'text', 'json_object' or 'grammar', field: 'response_format.type', value: 'json_schema'; Extra inputs are not permitted, field: 'response_format.json_schema'"}}

During handling of the above exception, another exception occurred:
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/main.py:477: in acompletion
    response = await init_response
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py:942: in acompletion
    raise OpenAIError(
E   litellm.llms.OpenAI.openai.OpenAIError: Error code: 400 - {'error': {'object': 'error', 'type': 'invalid_request_error', 'message': "2 request validation errors: Input should be 'text', 'json_object' or 'grammar', field: 'response_format.type', value: 'json_schema'; Extra inputs are not permitted, field: 'response_format.json_schema'"}}

During handling of the above exception, another exception occurred:
mbue/shared/llm/llm_client_test.py:58: in test_structured_output
    resp = await client.chat_response("Is 2+2=4?", response_format=BooleanResponse)
mbue/shared/llm/llm_client.py:79: in chat_response
    chat_completion = await self._cached_chat_completion(
mbue/shared/llm/llm_client.py:236: in _cached_chat_completion
    result = await self._create_chat_completion_with_retry(base_url, **kwargs)
mbue/shared/llm/llm_client.py:276: in _create_chat_completion_with_retry
    return await LLMClient._call_vendor_api_client(
mbue/shared/llm/llm_client.py:311: in _call_vendor_api_client
    return await litellm_chat_completion(base_url=base_url, **kwargs)
mbue/shared/llm/litellm_client.py:18: in litellm_chat_completion
    response = await acompletion(**cleaned_args)
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/utils.py:1173: in wrapper_async
    raise e
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/utils.py:1029: in wrapper_async
    result = await original_function(*args, **kwargs)
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/main.py:499: in acompletion
    raise exception_type(
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py:2122: in exception_type
    raise e
/home/daved/.cache/pypoetry/virtualenvs/mbue-8lo_cb6f-py3.11/lib/python3.11/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py:282: in exception_type
    raise BadRequestError(
E   litellm.exceptions.BadRequestError: litellm.BadRequestError: Fireworks_aiException - Error code: 400 - {'error': {'object': 'error', 'type': 'invalid_request_error', 'message': "2 request validation errors: Input should be 'text', 'json_object' or 'grammar', field: 'response_format.type', value: 'json_schema'; Extra inputs are not permitted, field: 'response_format.json_schema'"}}

Twitter / LinkedIn details

No response
@DaveDeCaprio DaveDeCaprio added the bug Something isn't working label Nov 18, 2024
DaveDeCaprio added a commit to DaveDeCaprio/litellm that referenced this issue Nov 18, 2024
@DaveDeCaprio
Fixed BerriAI#6797 Fireworks AI structured outputs
016c631
@DaveDeCaprio DaveDeCaprio linked a pull request Nov 18, 2024 that will close this issue
Open
@DaveDeCaprio
Copy link
Author

This is working for me with this PR. #6802

DaveDeCaprio added a commit to DaveDeCaprio/litellm that referenced this issue Nov 18, 2024
@DaveDeCaprio
Fixed BerriAI#6797 Fireworks AI structured outputs on version 1.52.5
ac06e6e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed #6797 Fireworks AI structured outputs DaveDeCaprio/litellm
1 participant
@DaveDeCaprio