-
Notifications
You must be signed in to change notification settings - Fork 8
/
prompts.py
38 lines (28 loc) · 11.7 KB
/
prompts.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
## Assume question format is always "Question:\nxxxxx"
natural_program_format_few_shot = [
"Question:\nJames gets 10 new CDs. Each CD cost $15. He gets them for 40% off. He decides he doesn't like 5 of them and sells them for 40. How much money was he out? \n\nAnswer:\nFirst, let us rewrite the question with labels.\n#1. James gets 10 new CDs. \n#2. Each CD cost $15, and he gets them for 40% off.\n#3. He sells 5 of them for 40.\n#4. How much money was he out?\nNext let us solve the problem step by step with reference to the question and reasoning process:\n\n#5. (by #2) Step 1: Calculate the price of each CD after the 40% discount.\nOriginal price per CD: $15, Discount: 40%\nPrice per CD after discount: $15 * (1 - 0.40) = $15 * 0.60 = $9\n\n#6. (by#1 #5) Step 2: Calculate the total cost of the 10 CDs. \nPrice per CD after discount: $9, Total CDs: 10\nTotal cost of 10 CDs: $9 * 10 = $90\n\n#7. (by #3) Step 3: Calculate the total money he gets back from selling 5 CDs. \nMoney from selling 5 CDs: $40\n\n#8. (by #6 #7) Step 4: Calculate the total amount of money James is out. \nTotal cost of 10 CDs: $90\nMoney from selling 5 CDs: $40\nMoney James is out: $90 - $40 = $50\n\n#9. (by #4 #8) The original question is #4. How much money was he out? We do not miss information on the rewritten labels. So the answer to this question is John is out $50."]
natural_program_format_suffix = "\n\nAnswer:\nFirst, let us rewrite the question with labels.\n\n"
verification_few_shot = [
## prompt for MWP
"Here is some information:\n\"1. Alice gets 2 apples.\n2. Alice gets twice as many oranges as bananas.\n3. Original price per apple: $2\nDiscount: 50%\nPrice per apple after discount: $2 * (1 - 0.50) = $2 * 0.50 = $1\"\n\nBased on the given information, here is a reasoning process:\n\"Calculate Alice's cost of the 5 apples after a 50% discount.\nPrice per apple after discount: $1\nApples be eaten: 3\nOranges be eaten: 6\nTotal apples: 5\nTotal oranges: x\nTotal bananas: 2x\nAlice's total number of fruits: 5 + x + 2x = 5 + 4x\nAlice's total cost of 5 apples: $1 * (5 - 3) = $2\"\n\nDouble-check the reasoning process, let's analyze its correctness, and end with \"yes\" or \"no\".\n\nAnswer:\nLet's think step by step.\n\nGrounding check\nPurpose: Find references for numbers in the reasoning process sequentially\nRule: Any ungrounded number makes the check fails\n\n\"x\" is a variable, we do not care about variables. \n\"y\" is a variable, we do not care about variables.\n\n\"5\" is a quantity from the given information that represents the number of apples.\n\"50%\" is a quantity from the given information that represents the discount.\n\"$1\" is a quantity from the given information that represents the price per apple after the discount.\n\"3\" is a quantity not from the given information, it is ungrounded.\n\"6\" is a quantity not from the given information, it is ungrounded.\n\"$2\" is a quantity from the given information that represents the original price per apple.\nResult: Based on the check, there are ungrounded numbers, so the grounding check fails.\n\nReasoning check:\nPurpose: Check whether the reasoning logic correctly answers the question\nRule: Any logic error makes the check fails\n\nTo calculate the total cost of apples after a 50% discount, we should multiply the number of apples by the discounted price. But to answer the question, we do not need to calculate the total number of fruit Alice gets.\nResult: Based on the check, there are some logic mistakes, so the reasoning check fails.\n\nCalculation check:\nPurpose: Check whether the calculation process is consistent\nRule: Any inconsistent calculation makes the check fails\n\ncalculation1: \nequation: $1 * (5 - 3), answer: $2\n(5 - 3) = 2\n$1 * 2 = $2 is consistent with the answer, so the calculation is correct.\ncalculation2:\nequation: 5 + x + 2x, answer: 5 + 4x\nx + 2x = 3x\n5 + 3x is inconsistent with the answer, so the calculation is incorrect.\nResult: Based on the check, the calculation process is inconsistent, so the calculation check fails.\n\nCheck results: Ground check fails, Reasoning check fails, Calculation check fails.\nRule: Any failed check makes the reasoning incorrect.\nSo the answer is \"no\".\n\n"
## prompt for Last Letter
# "Here is some information:\n\"We need to take the last letter of the word \"Stark\"\nWe need to concatenate the last letters of the words sequentially.\nTaking the last letter of the word \"Tony\" and concatenate the latter.\nThe last letter of the word \"Tony\" is \"y\".\nThe concatenated last letters are \"y\"\"\n\nBased on the given information, here is a reasoning process:\n\"Taking the last letter of the word \"Stark\" and concatenate the latter.\nThe last letter of the word \"Stark\" is \"k\".\nThe concatenated last letters are \"ky\".\"\n\nDouble-check the reasoning process, let's analyze its correctness, and end with \"yes\" or \"no\".\n\nAnswer:\nConcatenation check:\nPurpose: Check correctness for every concatenation, otherwise ignore this check\nRules: keep repeated letters, \"a\" + \"a\" = \"aa\"\n\nconcatenation targets (previous results \"y\", \"k\" from \"Stark\"), original answer \"ky\"\nrethink: \"y\" + \"k\" = \"yk\", compare with the original answer \"ky\", they are not the same words, so the concatenation is incorrect.\nResult: Based on the check, there are incorrect concatenation, so the answer is \"no\".\n\n"
]
grounding_few_shot = [
"Here is some information:\n\"1. Alice gets 5 apples.\n2. Alice gets twice as many oranges as bananas.\n3. Original price per apple: $2\nDiscount: 50%\"\n\nBased on the given information, here is a reasoning process:\n\"Calculate Alice's cost of the 5 apples after a 50% discount.\nPrice per apple after discount: $2 * 50% = $1\nTotal apples: 5\nTotal bananas: x\nTotal oranges: 2x\nextra discount: 80%\nAlice's total cost of 5 apples: $1 * 5 * 80% = $4\"\n\nDouble-check the numbers in the reasoning process, and find out whether their references are in the given information. End with \"yes\" or \"no\" for whether the grounding check passes.\n\nAnswer:\n\nGrounding check\nPurpose: Find references for numbers in the reasoning process sequentially\nRule: Any numbers without references from the given information or the reasoning process will fail the grounding check.\n\n\"x\" is a variable, variables are ignored.\n\"2x\" is a variable, variables are ignored.\n\n\"5\" is from the given information. (grounded)\n\"50%\" is from the given information. (grounded)\n\"$1\" is from the reasoning process. (grounded)\n\"80%\" does not have any references in the given information or the reasoning process. (ungrounded)\n\"$4\" is from the reasoning process. (grounded)\n\nResult: Based on the check, there are ungrounded numbers. So the answer is \"no\", the grounding check fails.\n\n"]
grounding_few_shot_suffix = "\n\nDouble-check the numbers in the reasoning process, and find out whether their references are in the given information. End with \"yes\" or \"no\" for whether the reasoning check passes.\n\nAnswer:\nGrounding check\nPurpose: Find references for numbers in the reasoning process sequentially\nRule: Any numbers without references from the given information or the reasoning process will fail the grounding check.\n"
reasoning_few_shot = [
"Here is some information:\n\"1. Alice gets 5 apples.\n2. Alice gets twice as many oranges as bananas.\n3. Original price per apple: $2\nDiscount: 50%\"\n\nBased on the given information, here is a reasoning process:\n\"Calculate Alice's cost of the 5 apples after a 50% discount.\nPrice per apple after discount: $2 / 50% = $4\nTotal apples: 5\nAlice's total cost of 5 apples: $4 * 5 = $20\nTotal bananas: x\nTotal oranges: 2x\nAlice's total number of fruits: 5 + x + 2x = 5 + 3x\"\n\nLet's review the reasoning process and check:\n1. What's the purpose of the reasoning process? (The first sentence of the reasoning process)\n2. Based on the purpose, do you think the reasoning process contains any irrelevant steps?\n3. Do you think the reasoning process is wrong?\nAny irrelevant step or mistake fails the check. End with \"yes\" or \"no\" for whether the reasoning check passes.\n\nAnswer:\nHere is the review:\nPurpose: \"Calculate Alice's cost of the 5 apples after a 50% discount\"\n\nIrrelevant Steps: The calculation of Alice's total number of fruits is irrelevant to the problem. The problem does not ask for the total number of fruits Alice has, so there's no need to include the calculation about the bananas and oranges.\nResult: The reasoning process contains irrelevant steps.\n\nReasoning Mistakes: The mistake lies in calculating the price per apple after discount. The correct calculation should be: price per apple after discount = $2 * (1 - 50%) = $1. Then, Alice's total cost of 5 apples would be: $1 * 5 = $5.\nResult: The reasoning process contains mistakes.\n\nIn summary:\n\n1. The reasoning process contains irrelevant steps.\n2. The reasoning process is wrong.\nResult: Based on the check, the answer is \"no\", the reasoning check fails.\n\n"
]
reasoning_few_shot_suffix = "\n\nLet's review the reasoning process and check:\n1. What's the purpose of the reasoning process? (The first sentence of the reasoning process)\n2. Based on the purpose, do you think the reasoning process contains any irrelevant steps?\n3. Do you think the reasoning process is wrong?\nAny irrelevant step or mistake fails the check. End with \"yes\" or \"no\" for whether the reasoning check passes.\n\nAnswer:\nHere is the review:"
calculation_few_shot = [
"Here is some information:\n\"1. Alice gets 5 apples and eats 2 of them.\n2. Alice gets twice as many oranges as bananas.\n3. Original price per apple: $2\nDiscount: 50%\"\n\nBased on the given information, here is a reasoning process:\n\"Calculate Alice's cost of the left apples after a 50% discount.\nPrice per apple after discount: $2 * 50% = $1\nAlice's total cost of the left apples: $1 * (5 - 2) = $2\"\nTotal oranges: x\nTotal bananas: 2x\nAlice's total number of fruits: 5 + x + 2x = 5 + 4x\n\nDouble-check the calculation process, let's analyze its consistency. End with \"yes\" or \"no\" for whether the calculation check passes.\n\nAnswer:\nCalculation check:\nPurpose: Check whether the calculation process is consistent\nRule: Any inconsistent calculation makes the check fails\n\ncalculation1: \nequation: $2* 50%, original answer: $1\n$2 * 50% = $1\n$1 is consistent with the original answer $1, so the calculation is correct.\n\ncalculation2: \nequation: $1 * (5 - 2), original answer: $2\n(5 - 2) = 3\n$1 * 3 = $3 is inconsistent with the original answer $2, so the calculation is incorrect.\n\ncalculation3:\nequation: 5 + x + 2x, original answer: 5 + 4x\nx + 2x = 3x\n5 + 3x is inconsistent with the original answer 5 + 4x, so the calculation is incorrect.\n\nResult: Based on the check, the calculation process is inconsistent. So the answer is \"no\", the calculation check fails..\n\n"
]
calculation_few_shot_suffix = "\n\nDouble-check the calculation process, let's analyze its consistency. End with \"yes\" or \"no\" for whether the calculation check passes.\n\nAnswer:\nCalculation check:\nPurpose: Check whether the calculation process is consistent\nRule: Any inconsistent calculation makes the check fails\n"
prompts = {"deductive_reasoning_generation": {"prefix": natural_program_format_few_shot, "suffix": natural_program_format_suffix},
"verification": {"prefix": verification_few_shot, "suffix": ""},
"grounding": {"prefix": grounding_few_shot, "suffix": grounding_few_shot_suffix},
"reasoning": {"prefix": reasoning_few_shot, "suffix": reasoning_few_shot_suffix},
"calculation": {"prefix": calculation_few_shot, "suffix": calculation_few_shot_suffix}}
def prompt_fn(question, mode):
return "\n".join(prompts[mode]["prefix"]) + question + prompts[mode]["suffix"]