Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BFCL Chore] Ensure Correct Input Format for Eval Checker #860

Merged
merged 3 commits into from
Jan 4, 2025

Conversation

HuanzhiMao
Copy link
Collaborator

In some cases, the model handler’s decode_ast method returns successfully but produces output in an unexpected format, causing issues in downstream evaluations that do not perform argument format validation. This problem is especially common when the model does not output any function calls, resulting in a human-readable string instead of the expected structure.

This PR refines the is_function_calling_format_output function to enforce that outputs must be a list of dictionaries in the following format before calling the checker function:

[
  {func1: {param1: val1, param2: val2, ...}},
  {func2: {param1: val1, param2: val2, ...}},
  ...
]

Note: This PR will not affect the leaderboard score.

@HuanzhiMao HuanzhiMao added the BFCL-General General BFCL Issue label Dec 30, 2024
@HuanzhiMao HuanzhiMao merged commit 79b1c60 into ShishirPatil:main Jan 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-General General BFCL Issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants