Duplicate item found with id fc_xxxx when using conversation_id with function calling

[kinnari-patwa]

Describe the bug

When using conversation_id in Runner.run_streamed method for an agent with function calling (custom function), I get the the following error:
{'error': {'message': 'Duplicate item found with id fc_68d25eda79d481959bd7e424738091a006b9ae17d036a0e7. Remove duplicate items from your input and try again.', 'type': 'invalid_request_error', 'param': 'input', 'code': None}}

Debug information

• Agents SDK version: 0.3.0
• Python version 3.12

Repro steps

   @function_tool
    async def query_database(wrapper: RunContextWrapper[AgentContext], query: str) -> str:
        """Query the database with the provided SQL query. Only SELECT queries are allowed."""
        return await self.agent_tools.query_database(query, wrapper.context.websocket)
    
    agent = Agent(
        name="Database Assistant",
        instructions="You are a helpful assistant that can query databases. Use the query_database function to execute SQL queries when users ask for data.",
        model="gpt-4o",
        model_settings=ModelSettings(temperature=0.2),
        tools=[query_database]
    )

run_stream = Runner.run_streamed(
                agent,
                [user_message],
            )

Try any user prompt that results in custom function call - query_database. You will get the following error:
Error - {'error': {'message': 'Duplicate item found with id fc_68d25eda79d481959bd7e424738091a006b9ae17d036a0e7. Remove duplicate items from your input and try again.', 'type': 'invalid_request_error', 'param': 'input', 'code': None}}

Expected behavior

run_streamed should run without any errors and allow function calling (custom function) when using conversation_id

[ihower]

According to the docs: https://openai.github.io/openai-agents-python/sessions/#openai-conversations-api-memory
you should be using OpenAIConversationsSession if you want to integrate with the OpenAI Conversations API. That is the officially supported way right now.

But I also confirmed that using conversation_id directly does not work: the same error happens with both run_streamed and run_sync. Here is a minimal reproducible example:

from agents import Agent, function_tool, Runner
from openai import OpenAI
client = OpenAI()

conversation = client.conversations.create()
conv_id = conversation.id

@function_tool
async def foobar() -> str:
    return "41"

agent = Agent(
    name="Assistant",
    instructions="just call foobar and answer me only the number",
    model="gpt-4.1-mini",
    tools=[foobar]
)

Runner.run_sync(agent, "hi", conversation_id=conv_id)

# openai.BadRequestError: Error code: 400 - {'error': {'message': 'Duplicate item found with id fc_68d281ab76108195ac719f9b7c48eac103032f3a88a761f2. Remove duplicate items from your input and try again.', 'type': 'invalid_request_error', 'param': 'input', 'code': None}}

Based on my understanding, the root cause is that when we pass conversation_id to the OpenAI API, we must only send new items in the request.

With function calling, however, the SDK internally makes two API calls. On the second call it currently resends the entire items list, which triggers the “Duplicate item found” API error.

I suggest the SDK could also be improved to properly support the conversation_id approach so that it works as expected. And, I think conversation_id and session memory should very likely not be set at the same time, since the two approaches are conceptually different:

1. OpenAIConversationsSession memory: fetches the whole conversation history from the OpenAI server and resends it on every turn. (it does not actually use conversation_id when calling the LLM API.)
2. conversation_id approach: relies fully on the OpenAI server to maintain state: the SDK only send the latest user input.

[kinnari-patwa]

We were trying to use the conversation_id approach to avoid sending attached images/files with every user prompt in a multi-turn conversation. Imagine user attached an image/file in the fist prompt and asked questions related to it in subsequent turns. To address this, I confirmed that passing conversation_id worked but other function calls are failing.

[ihower]

@kinnari-patwa In that case, I’d recommend using the previous_response_id parameter instead.
It behaves quite similarly, except that after each turn you need to update it with the latest response id.

Personally, I think previous_response_id is even better than conversation_id, since it gives you the ability to fork or retry a conversation thread (for example, when a user edits an earlier message or wants to try again from a specific turn), and it doesn’t require making an extra HTTP request just to obtain a conversation_id up front.

[kinnari-patwa]

Thank you @ihower, I will try both approaches individually - previous_response_id and OpenAIConversationsSession and share my findings.

[kinnari-patwa]

With OpenAIConversationsSession (no conversation_id specified in run_streamed method), function calls work but now we have a new issue. Once I attach a pdf, i get the response, but on subsequent turns, if I ask any question, i get the following error:

{'error': {'message': "Missing required parameter: 'input[8].content[0]'.", 'type': 'invalid_request_error', 'param': 'input[8].content[0]', 'code': 'missing_required_parameter'}}

Not sure, where it is picking up input[8] from?

[kinnari-patwa]

previous_response_id generally works well, it is able to respond to questions related to attached pdf document on subsequent turns and no issues with function calling. But, it was not able to respond to questions related to attached images on subsequent turns.

A fix for the original issue related to conversation_id should result in the best outcome.