Skip to content

Fixes #1582: Handling of structured CompletionUsage response values for token usage #1583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

elondemirok
Copy link

@elondemirok elondemirok commented May 11, 2025

Summary

This PR resolves a compatibility issue caused by recent changes in the Python SDK. Previously, token usage data (completion_tokens, prompt_tokens, total_tokens) was returned as integers. Recent API updates introduced structured CompletionUsage objects, resulting in type errors after eval execution.

  • I have tested my fix locally and verified it resolves the issue. Screenshot of output attached.
image
  • I have minimized changes to keep the code easy to review and maintain.
  • This PR does not introduce new evals or modify existing evaluation logic.

Final checklist 👀

Submission agreement

  • I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.
  • I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.
  • I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.
  • I have filled out all required fields of this form
  • ⚠️ It's a bug fix; I haven't used Git LFS for the Eval JSON data
  • (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that mypy, black, isort, autoflake and ruff are running when I commit and push

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant