Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

demo.ipynb: Error in output extraction, sequence item 1: expected str instance, BWStateToT found #112

Open
LordEdison opened this issue Nov 18, 2024 · 2 comments

Comments

@LordEdison
Copy link

model:
TheBloke/Llama-2-70B-GPTQ

the final code block:

with open('examples/CoT/blocksworld/prompts/pool_prompt_v1.json') as f:
prompt = json.load(f)
evaluator = BWEvaluator(config_file='examples/CoT/blocksworld/data/bw_config.yaml',
domain_file='examples/CoT/blocksworld/data/generated_domain.pddl',
data_path='examples/CoT/blocksworld/data/split_v1/split_v1_step_4_data.json',
init_prompt=prompt)
evaluator.evaluate(reasoner_tot, shuffle_prompt=True, num_shot=4, resume=0)

Result: All of the outputs are empty strings?

Error in output extraction, sequence item 1: expected str instance, BWStateToT found
[+]: Saving plan in tmp_plan.txt
RESPONSE:::
Case #1: correct=False, output='', answer={'init': 'the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table', 'goal': 'the orange block is on top of the blue block', 'plan': '\nunstack the blue block from on top of the orange block\nput down the blue block\nunstack the orange block from on top of the red block\nstack the orange block on top of the blue block\n[PLAN END]\n', 'question': '\n[STATEMENT]\nAs initial conditions I have that, the blue block is clear, the hand is empty, the red block is on top of the yellow block, the blue block is on top of the orange block, the orange block is on top of the red block and the yellow block is on the table.\nMy goal is to have that the orange block is on top of the blue block.\n\nMy plan is as follows:\n\n[PLAN]\n', 'instance_file': 'LLMs-Planning/llm_planning_analysis/instances/blocksworld/generated_basic/instance-176.pddl'};accuracy=0.000 (0/1)

@LordEdison LordEdison changed the title Error in output extraction, sequence item 1: expected str instance, BWStateToT found demo.ipynb: Error in output extraction, sequence item 1: expected str instance, BWStateToT found Nov 18, 2024
@Ber666
Copy link
Collaborator

Ber666 commented Dec 2, 2024

Hi, sorry for the bug. We have updated the demo.ipynb.

The reason is that output_extractor was missing when initializing the BWEvaluator object. This function will transform the search results into the action sequence which can be understood by the evaluator. For more use cases, please refer to 1, 2

@LordEdison
Copy link
Author

thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants