[Platform] Standardise token usage #311
Conversation
junaidbinfarooq
requested review from
chr-hertel,
Nyholm and
OskarStark
as code owners
carsonbot
added
Feature
| $result = $agent->call($messages, [ | ||
| 'max_tokens' => 500, // specific options just for this call | ||
| ]); | ||
|
|
||
| $metadata = $result->getMetadata(); | ||
| if (null === $tokenUsage = $result->getTokenUsage()) { |
There was a problem hiding this comment.
Yes, if you mean null === $tokenUsage, when token_usage is marked false in the config, as far as I can see.
| final class TokenUsage implements \JsonSerializable | ||
| { | ||
| public function __construct( | ||
| public ?int $promptTokens = null, |
There was a problem hiding this comment.
| public ?int $promptTokens = null, | |
| public ?int $prompt = null, |
Maybe remove all the Token suffix here? They feel superfluous
There was a problem hiding this comment.
No problem.
I would not have kept the suffix if the object were simply named something like Tokens, but I thought TokenUsage is a better name. Or perhaps the DTO should be named as Tokens.
There was a problem hiding this comment.
To be discussed with @chr-hertel
There was a problem hiding this comment.
I did remove the suffixes BTW.
There was a problem hiding this comment.
Thanks, not sure now that this was the right decision 😅
junaidbinfarooq
force-pushed
the
feat/standardize-token-usage
branch
2 times, most recently
from
7a34c26 to
d771023
Compare
junaidbinfarooq
force-pushed
the
feat/standardize-token-usage
branch
from
d771023 to
75e98c8
Compare
- Makes token usage explicit - Adds support to automatically register token usage extractors so that the token usage information is available in the result when a model is invoked - Makes the token usage auto-registration configurable
junaidbinfarooq
force-pushed
the
feat/standardize-token-usage
branch
from
75e98c8 to
048ac63
Compare
There was a problem hiding this comment.
Sorry, but we need to slim this down a bit - in general smaller PRs, that build on top of each other are favorable to me. so let's break it down like you did:
Makes token usage explicit
I like having the value object, we should definitely have that 👍
But I'm not in favor of having thegetTokenUsage()directly on the result since the result is a high lever abstraction and token usage is quite specific to remote inference of GenAI models - not all models have that. let's keep it in the metadata please.
Adds support to automatically register token usage extractors so that the token usage information is available in the result when a model is invoked
That's a bit misleading since token usage was already sitting on an extension point =>OutputProcessorand your design decision to use that extension point to add another extension point doesn't really resonate with me. why is the new extension point needed? it feels like we're just adding them because we can.
Makes the token usage auto-registration configurable
For me this is too early, i don't think that a use-case, do you? how often do we see the case for someone to bring their own token usage extractor.
all in all, let's start here please with the TokenUsage value object in the metadata class - that's an easy step we see the same picture 👍
edit: another PR could be to fix that issue #208 by registering the corresponding output processor based on the config setting
Changes proposed:
The PR aims to standardize the token usage extraction and its availability in the AI bundle. Essentially, the following is done: