OpenAI cache costs not tracked

[LordMoon1337]

Hi all!

I log my requests to Openai as follows:

logger = logging.getLogger(__name__)
config = load_dotenv()

os.environ['OPENAI_API_KEY'] = os.environ.get("OPENAI_API_KEY")
os.environ['ANTHROPIC_API_KEY'] = os.environ.get("ANTHROPIC_API_KEY")


async def gpt_response(messages: list[dict[str: str]], attempts: int, func: str | list, json_format: BaseModel, model: str, temp: float) -> str | dict:
    try:
        async with AsyncOpenAI() as client:
            if func == 'json_mode':
                response = await client.beta.chat.completions.parse(
                    model=model,
                    messages=messages,
                    temperature=temp,
                    seed=52,
                    response_format=json_format,
                )
                result = json.loads(response.choices[0].message.content)
        return result

There is currently no information regarding cached tokens in the user interface, and I have a suspicion that they are not being taken into account. I attempted to review the documentation, however, I was unable to determine how to include information regarding the composition of the price and total cost of a cache.

To verify my findings, I executed the same query both in the absence of a cache (first attempt) and subsequent attempts in which approximately 70% of all tokens were cached. The price remained unchanged.

Please help me, how can I enable the accounting of cached tokens when using OpenAI?

[dosubot[bot]]

Hey there, @LordMoon1337! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

To enable the accounting of cached tokens when using OpenAI with Langfuse, you need to include the cached_tokens in the usage or usage_details object. Here's an example of how you can do this:

generation = langfuse.generation(
  # ...
  usage={
    # usage
    "prompt_tokens": int,
    "completion_tokens": int,
    "total_tokens": int,
    "prompt_tokens_details": {
      "cached_tokens": int,
      "audio_tokens": int,
    },
    "completion_tokens_details": {
      "reasoning_tokens": int,
    },
  },
  # ...
)

In this setup, cached_tokens is included under prompt_tokens_details, which allows Langfuse to account for cached tokens in the usage tracking ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[LordMoon1337]

So it turns out that I have to stop using from langfuse.openai import AsyncOpenAI and set up a custom generation calculation via @observe(as_type="generation")?

[marcklingen]

Langfuse supports tracking cached tokens in both the native OpenAI integration and server-side. Based on the screenshot, I assume you are running a self-hosted version of Langfuse that is not the latest. If that’s the case, please upgrade, as this should resolve the issue. If you are not using a self-hosted version or the upgrade does not fix it, please create an issue or bug report. This functionality is essential, and we are happy to help you resolve it.

[LordMoon1337]

Ty, i updated langfuse and everything worked