Refactor/tokenizers #840
Refactor/tokenizers #840
Conversation
collindutter
force-pushed
the
refactor/tokenizers
branch
10 times, most recently
from
74437d1 to
b00d200
Compare
collindutter
force-pushed
the
refactor/tokenizers
branch
2 times, most recently
from
eb39a02 to
9e1533e
Compare
Codecov ReportAttention: Patch coverage is 📢 Thoughts on this report? Let us know! |
collindutter
force-pushed
the
refactor/tokenizers
branch
3 times, most recently
from
11cdd03 to
4377389
Compare
collindutter
force-pushed
the
refactor/tokenizers
branch
from
4377389 to
fb52f7b
Compare
collindutter
force-pushed
the
refactor/tokenizers
branch
from
fb52f7b to
ac2b5da
Compare
| def prompt_stack_input_to_message(self, prompt_input: PromptStack.Input) -> dict: | ||
| """Converts a PromptStack Input to a ChatML-style message dictionary for token counting or model input. | ||
|
|
||
| Args: | ||
| prompt_input: The PromptStack Input to convert. | ||
|
|
||
| Returns: | ||
| A dictionary with the role and content of the input. | ||
| """ | ||
| content = prompt_input.content | ||
|
|
||
| if prompt_input.is_system(): | ||
| return {"role": "system", "content": content} | ||
| elif prompt_input.is_assistant(): | ||
| return {"role": "assistant", "content": content} | ||
| else: | ||
| return {"role": "user", "content": content} |
There was a problem hiding this comment.
Should this be a method on PromptInput?
There was a problem hiding this comment.
I don't think so, each Tokenizer will have a slightly different implementation of this. In an upcoming PR i'm changing this to an abstract method.
|
|
||
| def prompt_stack_to_string(self, prompt_stack: PromptStack) -> str: | ||
| """Converts a Prompt Stack to a string for token counting or model input. | ||
| This base implementation will not be very accurate, and should be overridden by subclasses with model-specific tokens. |
There was a problem hiding this comment.
Originally same question here, but I sense that the model-specific method means this should be owned by the tokenizer.
There was a problem hiding this comment.
Honestly we could probably get rid of this method. All of our Prompt Drivers now support message-style APIs.
There was a problem hiding this comment.
Ah but now I remember why I didn't remove in this PR: I didn't want to remove the token counts in the events which relies on this method. My next PR changes this functionality so I will remove this method.
|
|
||
| def _default_max_input_tokens(self) -> int: | ||
| tokens = next((v for k, v in self.MODEL_PREFIXES_TO_MAX_INPUT_TOKENS.items() if self.model.startswith(k)), None) | ||
|
|
||
| if tokens is None: | ||
| raise ValueError(f"Unknown model default max input tokens: {self.model}") | ||
| return self.DEFAULT_MAX_INPUT_TOKENS |
There was a problem hiding this comment.
Seems reasonable to fall back to the default here. Do you think it'd be worth it to throw out a warning if this happens? It seems like a big enough difference in expectations between the known values defined in the mapping and a default 'best guess' value, a behavior we seem to have avoided until now.
There was a problem hiding this comment.
Yeah warning is a good idea.
collindutter
force-pushed
the
refactor/tokenizers
branch
from
62b4f47 to
bc12fa5
Compare
collindutter
force-pushed
the
refactor/tokenizers
branch
from
649dcda to
3e0a24a
Compare
|
|
||
| ### Simple | ||
| Not all LLM providers have a public tokenizer API. In this case, you can use the `SimpleTokenizer` to count tokens based on a simple heuristic. | ||
|
|
||
| ```python | ||
| from griptape.tokenizers import SimpleTokenizer | ||
|
|
||
| tokenizer = SimpleTokenizer(max_input_tokens=1024, max_output_tokens=1024, characters_per_token=6) | ||
| tokenizer = SimpleTokenizer(model="any-model", max_input_tokens=1024, max_output_tokens=1024, characters_per_token=6) |
There was a problem hiding this comment.
Just curious, why can't model be optional?
Added
BaseTokenizer.prompt_stack_to_string()to convert a Prompt Stack to a string.BaseTokenizer.prompt_stack_input_to_string()to convert a Prompt Stack Input to a ChatML-style message dictionary.Changed
BasePromptDriver.count_tokens().BasePromptDriver.max_output_tokens().BasePromptDriver.prompt_stack_to_string()toBaseTokenizer.PromptStack.add_to_conversation_memorytoBaseConversationMemory.add_to_prompt_stack.BaseTokenizer.count_tokens()can now approximately token counts given a Prompt Stack.BasePromptDriver.max_tokensinstead of usingBasePromptDriver.max_output_tokens().griptape.constants.RESPONSE_STOP_SEQUENCEtoToolkitTask.ToolkitTask.RESPONSE_STOP_SEQUENCEis now only added when usingToolkitTask.