v2.0.0: Custom model usage and cost tracking

[marcklingen]

Important

This is a major release of Langfuse as it requires to run a one-off (non-blocking) migration script on historical data to ensure accurate LLM costs in Langfuse.

• No changes are required if you use Langfuse Cloud or are only concerned about newly ingested traces.
• There are no other breaking changes.

What's Changed

• Fully rebuilt usage/cost tracking of LLM generations in Langfuse by @maxdeichmann and @marcklingen
  • New model definition abstraction which allows to:
    • Quickly support newly emerging models.
    • Track changes in model prices over time.
    • Add custom models/prices at the project level.
  • You can also set (token) usage and cost via the API when ingesting traces.
    • If you want to set cost via the SDKs, make sure to upgrade to JS 2.4.0 and Python 2.7.3 or higher.
  • Usage and cost information on all UI tables and APIs
  • See changelog and docs for full details on this change.
• Other changes
  • feat(ui): add improved prompt ui with better versioning by @RichardKruemmel in feat(ui): add improved prompt ui #961
  • feat(ui): filtering by scores to the generations table in Add filtering by scores to the generations table #880
  • feat: add new openai model prices by @maxdeichmann in feat: add new openai model prices #967
  • feat(db): improve seeding script for prompts by @RichardKruemmel in feat(db): improve seeding script for prompts #965
  • fix(security): add validation for html in inputs by @RichardKruemmel in fix(security): add validation for html in inputs #927
  • fix(sessions): composite ids by @marcklingen in fix(sessions): composite ids #946
  • fix(ui): retrieve prompts created via API by @maxdeichmann in fix: retrieve api prompts #963
  • fix: error log for api prompts by @RichardKruemmel in fix: error log for api prompts #966
  • chore(deps): bump dependencies in chore(deps): bump the patches group with 13 updates #970, chore(deps-dev): bump @testing-library/jest-dom from 6.1.5 to 6.2.1 #936, chore(deps): bump dependencies #979, chore(deps): bump clsx from 2.0.0 to 2.1.0 #986, chore(deps): bump the patches group with 1 update #983, chore(deps-dev): bump @typescript-eslint/eslint-plugin from 6.19.1 to 6.20.0 #985
  • refactor: typing of AutoCompleteOption by @marcklingen in refactor: typing of AutoCompleteOption #978
  • refactor: db logging in dev mode by @maxdeichmann in refactor: db logging in dev mode #952

Full Changelog: v1.34.1...v2.0.0

---

This discussion was created from the release v2.0.0.

[marcklingen]

We are very excited about this release as it makes usage and cost tracking super flexible and customizable.

For full details, check out the changelog and docs page.

Why is this a major version when it is generally not breaking? We made this a new major version because we know some teams use Langfuse's usage data for rate-limiting or billing. We wanted to avoid accidentally breaking this logic before these teams ran the migration script for historical data.

Looking forward to any feedback/ideas regarding cost/usage tracking in Langfuse!

[krrishdholakia]

Wasn't able to send cost via langfuse v2.2.1 python

Is this live on the cloud app yet? @marcklingen

[krrishdholakia]

same issue on v2.7.1

[krrishdholakia]

code:

print(f"kwargs response cost: {kwargs['response_cost']}")
        trace.generation(
            name=metadata.get("generation_name", "litellm-completion"),
            id=metadata.get("generation_id", None),
            startTime=start_time,
            endTime=end_time,
            model=kwargs["model"],
            modelParameters=optional_params,
            input=input,
            output=output,
            usage={
                "prompt_tokens": response_obj["usage"]["prompt_tokens"],
                "completion_tokens": response_obj["usage"]["completion_tokens"],
                "input_cost": kwargs["response_cost"],
                "output_cost": kwargs["response_cost"],
            },
            metadata=metadata,
        )

[maxdeichmann]

@krrishdholakia looking into this now.

[maxdeichmann]

@krrishdholakia this only works as of 2.7.3 in langfuse python. I created a litellm PR (BerriAI/litellm#1704) with questions.