Skip to content

Commit

Permalink
add the pass and fail criteria in the slot rubric
Browse files Browse the repository at this point in the history
  • Loading branch information
gannim committed Dec 4, 2024
1 parent 7a98b47 commit 571abec
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion data/rubric_slot.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,14 @@ Evaluate whether it's a "pass" or a "fail".
In the [Query], a user asks the assistant a question or makes a request, and in [Available Functions], there exists a suitable function to perform this task, but there is a lack of required parameter values needed to call the function. In this [Submission], the assistant is required to ask the user for any additional information necessary to invoke the appropriate function and complete the task. You are given a [Ground Truth] for each [Query], so you can refer to this for evaluating the response.
"pass"
- Appropriate questions for slot filling were asked. (It is not an issue if the 'function call' or 'tool call' item is null.)
- All additional information needed to fill in the missing required parameter values (which is clearly outlined in the [Ground Truth]) was specifically requested without any omissions.
"fail"
- Tool call missing required information
- Tool call with incorrect information: Hallucinated values not found in the [Query].
- Fail of function selection: Called a different, inappropriate function instead of the one that should be called (as can be verified through the [Ground Truth]).
- Answered arbitrarily based on their own knowledge without considering the function call.
- Answered arbitrarily based on their own knowledge without considering the function call.
- Slot filling question made with omissions of some required additional information.
- Redundant question related to information already provided.
***
[END DATA]
Does the submission meet the criteria? Begin by explaining your reasoning step by step in Korean, without immediately revealing the outcome. Subsequently, on a separate line, clearly indicate whether it is a "pass" or "fail". For clarity, repeat your final decision once more (without quotes or punctuation, literally).

0 comments on commit 571abec

Please sign in to comment.