Skip to content

Commit

Permalink
Fix streaming for granite tool call when <|tool_call|> is present (#1…
Browse files Browse the repository at this point in the history
…1069)

Signed-off-by: Max de Bayser <[email protected]>
  • Loading branch information
maxdebayser authored Dec 11, 2024
1 parent ffa48c9 commit e39400a
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion vllm/entrypoints/openai/tool_parsers/granite_tool_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,11 @@ def extract_tool_calls_streaming(
) -> Union[DeltaMessage, None]:

start_idx = consume_space(0, current_text)
if not current_text or current_text[start_idx] != '[':
if current_text[start_idx:].startswith(self.bot_token):
start_idx = consume_space(start_idx + len(self.bot_token),
current_text)
if not current_text or start_idx >= len(current_text)\
or current_text[start_idx] != '[':
return DeltaMessage(content=delta_text)

# bit mask flags for partial JSON parsing. If the name hasn't been
Expand Down

0 comments on commit e39400a

Please sign in to comment.