You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
with open('examples/CoT/blocksworld/prompts/pool_prompt_v1.json') as f:
prompt = json.load(f)
evaluator = BWEvaluator(config_file='examples/CoT/blocksworld/data/bw_config.yaml',
domain_file='examples/CoT/blocksworld/data/generated_domain.pddl',
data_path='examples/CoT/blocksworld/data/split_v1/split_v1_step_4_data.json',
init_prompt=prompt)
evaluator.evaluate(reasoner_tot, shuffle_prompt=True, num_shot=4, resume=0)
Result: All of the outputs are empty strings?
Error in output extraction, sequence item 1: expected str instance, BWStateToT found
[+]: Saving plan in tmp_plan.txt
RESPONSE:::
Case #1: correct=False, output='', answer={'init': 'the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table', 'goal': 'the orange block is on top of the blue block', 'plan': '\nunstack the blue block from on top of the orange block\nput down the blue block\nunstack the orange block from on top of the red block\nstack the orange block on top of the blue block\n[PLAN END]\n', 'question': '\n[STATEMENT]\nAs initial conditions I have that, the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table.\nMy goal is to have that the orange block is on top of the blue block.\n\nMy plan is as follows:\n\n[PLAN]\n', 'instance_file': 'LLMs-Planning/llm_planning_analysis/instances/blocksworld/generated_basic/instance-176.pddl'};accuracy=0.000 (0/1)
The text was updated successfully, but these errors were encountered:
LordEdison
changed the title
Error in output extraction, sequence item 1: expected str instance, BWStateToT found
demo.ipynb: Error in output extraction, sequence item 1: expected str instance, BWStateToT found
Nov 18, 2024
Hi, sorry for the bug. We have updated the demo.ipynb.
The reason is that output_extractor was missing when initializing the BWEvaluator object. This function will transform the search results into the action sequence which can be understood by the evaluator. For more use cases, please refer to 1, 2
model:
TheBloke/Llama-2-70B-GPTQ
the final code block:
with open('examples/CoT/blocksworld/prompts/pool_prompt_v1.json') as f:
prompt = json.load(f)
evaluator = BWEvaluator(config_file='examples/CoT/blocksworld/data/bw_config.yaml',
domain_file='examples/CoT/blocksworld/data/generated_domain.pddl',
data_path='examples/CoT/blocksworld/data/split_v1/split_v1_step_4_data.json',
init_prompt=prompt)
evaluator.evaluate(reasoner_tot, shuffle_prompt=True, num_shot=4, resume=0)
Result: All of the outputs are empty strings?
Error in output extraction, sequence item 1: expected str instance, BWStateToT found
[+]: Saving plan in tmp_plan.txt
RESPONSE:::
Case #1: correct=False, output='', answer={'init': 'the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table', 'goal': 'the orange block is on top of the blue block', 'plan': '\nunstack the blue block from on top of the orange block\nput down the blue block\nunstack the orange block from on top of the red block\nstack the orange block on top of the blue block\n[PLAN END]\n', 'question': '\n[STATEMENT]\nAs initial conditions I have that, the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table.\nMy goal is to have that the orange block is on top of the blue block.\n\nMy plan is as follows:\n\n[PLAN]\n', 'instance_file': 'LLMs-Planning/llm_planning_analysis/instances/blocksworld/generated_basic/instance-176.pddl'};accuracy=0.000 (0/1)
The text was updated successfully, but these errors were encountered: