How to update generation cost and token usage?

[cimuletz]

Hi, I was reading the documentation and I stumbled upon this doc which states that usage and cost of a generation can be update using generation.update(). Could anyone provide a snippet which exemplifies the usage of generation.update() to update the cost of a generation?

My use case is to update the cost of a generation which belongs to a dataset: I am using an external LLM-as-a-judger with a custom cost function and I want to update the cost of a generation with the cost of the evaluation.

I attempted to update the cost with the following code, but 'Usage' is immutable.

generation = self.langfuse.fetch_observations(trace_id = handler.trace.id, type = 'GENERATION')
generation.data[0].usage.total_cost = cost

I am self-hosting Langfuse, version 2.93.6.

[dosubot[bot]]

To update the cost of a generation in Langfuse, you should use the generation.update() method instead of directly modifying the usage field, which is immutable. Here's an example of how you can update the cost using generation.update():

# Fetch the generation you want to update
generation = self.langfuse.fetch_observations(trace_id=handler.trace.id, type='GENERATION')[0]

# Update the generation's cost
generation.update(usage={
    "total_cost": cost
})

This approach allows you to update the cost without directly modifying the immutable usage field ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[marcklingen]

This is partially correct.

You'd need to use the following in order to update an existing generation:

generations = self.langfuse.fetch_observations(trace_id = handler.trace.id, type = 'GENERATION')
generation_id = generations.data[0].id

langfuse.generation(id=generation_id, trace_id=trace_id, usage=usage)

If possible, I'd try to add the cost information at run-time of the dataset run in order to prevent this update.