merging consecutive anthropic messages

[morgante]

What happened?

When sending a simple and correct series of messages for Anthropic, I receive this error:

Error code: 400 - {'error': {'message': "AnthropicException - AnthropicException - Invalid role passed in - {'role': 'user', 'content': '\\nWhat is the query for `console.info` => `consdole.heaven`\\n'}", 'type': None, 'param': None, 'code': 400}}

Messages:

[
    {
        "role": "user",
        "content": "\nWhat is the query for `console.log` => `console.error`\n"
    },
    {
        "role": "assistant",
        "content": "\nThis is the GritQL query for the given before/after examples:\n<gritql>\n`console.log` => `console.error`\n</gritql>\n"
    },
    {
        "role": "user",
        "content": "\nWhat is the query for `console.info` => `consdole.heaven`\n"
    }
]

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

working to repro this issue.

[morgante]

It looks like you fixed it here: b9143a0

I would prefer that you not rewrite my messages at all, so I opened #2663

[krrishdholakia]

unable to repro this

[krrishdholakia]

@morgante can you expand on that? If you pass consecutive user or assistant messages, claude will throw an error. To avoid this (since this would work on openai), we merge consecutive messages.

Would you rather we raise an error?

[krrishdholakia]

Hi @morgante,

Re: testing
LiteLLM runs 456 tests on every commit, before publishing something. Here's our testing - https://github.com/BerriAI/litellm/tree/main/litellm/tests.

We would welcome any additional tests you have.

Re: bug
Thanks for raising this, I've added your test to our ci/cd - 2a9fd4c. I was able to see this caught in our testing pipeline, and when i run it on latest this doesn't occur. Let me know if it persists for you.

Re: merging
You're saying - if you passed an invalid anthropic message, let the actual anthropic call fail. That's good to understand.

Sharing my thinking on this - Since the consecutive user/assistant messages format would work on openai, it seems unfair to ask a developer to create a condition block in their code to need to deal with this restriction just for anthropic, where the fix here would be somebody on their end needing to do the merge anyways. As a user of litellm, I would want this library to make sure i don't need to worry about that.

In cases where the change is pretty drastic - example the initial anthropic message needs to be a user message, and if you don't pass that - I would not want a package to be inserting things into my prompt. So we raise the error.

It's definitely a balancing act, so I appreciate the feedback here.

[morgante]

Could you let me know which release had it? I was able to see this caught in our testing pipeline, and when i run it on latest this doesn't occur.

Can you point that out? It seems like v1.33.7 had this bug.

Sharing my thinking on this - Since the consecutive user/assistant messages format would work on openai, it seems unfair to ask a developer to create a condition block in their code to need to deal with this restriction just for anthropic, where the fix here would be somebody on their end needing to do the merge anyways. As a user of litellm, I would want this library to make sure i don't need to worry about that.

You already have a param for allowing rewriting messages. I don't understand why you wouldn't use that param. I think there are clearly two sets of developers:

• Those who carefully control their messages and make sure weird/invalid prompts aren't inserted (two consecutive human/assistant messages doesn't actually make any sense, and I would classify as a bug). In those cases, I do NOT want LiteLLM arbitrarily rewriting my messages and introducing bugs.
• For developers who want LiteLLM to rewrite their messages, they can set the flag and you keep rewriting.

[krrishdholakia]

hmm, it would help to understand why you're using litellm here?

[morgante]

Why are you questioning my use of LiteLLM? The caching, observability, centralized proxy, unified API, etc. are all very valuable. That doesn't mean I want you rewriting my messages.

[ishaan-jaff]

Addressed here @morgante #2671 - added some notes on the PR, feel free to review / leave comments

[krrishdholakia]

moving to a discussion, as this doesn't seem to be an existing bug on litellm