Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: prompt engineering to enhance output stability #119

Closed
wants to merge 7 commits into from
Closed

chore: prompt engineering to enhance output stability #119

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e209264
chore: prompt engineering to enhance output stability
Apr 2, 2024
a31b782
chore: prompt engineering to enhance output stability
FromCSUZhou Apr 2, 2024
1a5549e
Update autotx/utils/agent/build_goal.py
FromCSUZhou Apr 4, 2024
5e132af
Update autotx/utils/agent/define_tasks.py
FromCSUZhou Apr 4, 2024
ff60f91
Update autotx/utils/agent/define_tasks.py
FromCSUZhou Apr 4, 2024
0fa56fe
Update autotx/utils/agent/define_tasks.py
FromCSUZhou Apr 4, 2024
9259e0e
Merge branch 'polywrap:main' into prompt-engineering-on-task-and-goal
FromCSUZhou Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 36 additions & 10 deletions autotx/utils/agent/build_goal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,40 +66,66 @@ def build_goal(prompt: str, agents_information: str, smart_account_address: ETHA
def analyze_user_prompt(chat_history: str, agents_information: str, smart_account_address: ETHAddress) -> DefineGoalResponse:
template = dedent(
"""
Based on the following chat history between you and the user:
```
{chat_history}
```

You are a task assignment expert. You are good at analyzing the user's chat records to complete the task goals and conditions represented in the user's chat records.
nerfZael marked this conversation as resolved.
Show resolved Hide resolved
You must analyze the prompt and define a goal to be executed by the agents.
If the prompt is not clear or missing information, you MUST ask for more information.
If the prompt is invalid, unsupported or outside the scope of the agents, you MUST ask for a new prompt.
Always ensure you have all the information needed to define the goal that can be executed without prior context.
When dealing with Ethereum transactions, assume the following is the user's address: {smart_account_address}
dOrgJelli marked this conversation as resolved.
Show resolved Hide resolved

The available agents and tools:
{agents_information}

Respond ONLY in one of three of the following JSON formats:
1:

{{
"type": "goal",
"goal": "The detailed goal here. No need to mention specific agents or tools. But you MUST mention the user's address."
"goal": "The detailed goal here. No need to mention specific agents or tools."
}}
2:

{{
"type": "missing_info",
"message": "The information that is missing here"
}}
3:

{{
"type": "unsupported",
"message": "Reason why the prompt is unsupported here"
}}

Please note: You can only output content in json format, and do not output any other content!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line is repeated in line 121 - is that intended?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we have found that for small parameter models, emphasizing the most important content before and after a long prompt seems to be helpful for the stability of the output.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or can we use different prompts for strong inference models such as GPT-4 or small parameter models that need to constantly emphasize requirements? It is true that long prompts may not be necessary for gpt-4, which can already handle it well.

Copy link
Contributor

@cbrzn cbrzn Apr 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or can we use different prompts for strong inference models such as GPT-4 or small parameter models that need to constantly emphasize requirements? It is true that long prompts may not be necessary for gpt-4, which can already handle it well.

i think it makes sense to have one prompt for gpt-4 and another for small parameter models

based on the benchmark results from #119 (review) i would recommend updating the examples with the changes i suggested, see if they improve the planning (which they should, since the examples rn are incorrect) and if it does not improve, then use different prompts for the different models 😄 - otherwise, i don't think it is necessary to have different prompts for the models in this PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes for right now I think we should have a single prompt. Adding more will just add another dimension of complexity which will make it harder to get consistent results.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree with you. In the case where gpt-4 can already handle the requirements of concise prompts, more examples may only consume tokens. I think two different sets of prompts can be submitted to adapt to gpt. -4 and our local llms


You can refer to the following examples:
Chat History: User: Swap ETH to 0.05 WBTC, then swap WBTC to 1000 USDC, and finally send 50 USDC to vitalik.eth
Output:
{{
"type": "goal",
"goal": "First, swap Ethereum (ETH) to 0.05 Wrapped Bitcoin (WBTC), ensuring the right amount of ETH is swapped for exactly 0.05 WBTC. After successfully swapping to WBTC, proceed to swap the entire 0.05 WBTC to USD Coin (USDC), making sure to receive exactly 1000 USDC from the swap. Following these swaps, transfer 50 USDC to the Ethereum Name Service (ENS) domain 'vitalik.eth'."
FromCSUZhou marked this conversation as resolved.
Show resolved Hide resolved
}}

Chat History: User: Send 1 ETH to vitalik.eth
Output:
{{
"type": "goal",
"goal": "Send 1 ETH from the address 0x5d15311D760511d89cFad67404131cdc155E9FDB to the ENS domain vitalik.eth."
}}

Chat History: User: Buy 100 USDC with ETH
Output:
{{
"type": "goal",
"goal": "Execute a swap from ETH to 100 USDC for the user's address 0x5d15311D760511d89cFad67404131cdc155E9FDB"
}}

OK, start now!
Please note: You can only output content in json format, and do not output any other content!
nerfZael marked this conversation as resolved.
Show resolved Hide resolved
Chat History: {chat_history}
Output:
"""
)

formatted_template = template.format(
agents_information=agents_information, chat_history=chat_history
agents_information=agents_information, chat_history=chat_history, smart_account_address=smart_account_address
)

response = openai.chat.completions.create(
Expand Down
95 changes: 88 additions & 7 deletions autotx/utils/agent/define_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,104 @@
def define_tasks(goal: str, agents_information: str, agents: list[Agent]) -> list[Task]:
template = dedent(
"""
Based on the following goal: {goal}

You must convert instructions into specific tasks with the following JSON format:
You are an expert in decomposing and assigning tasks.
You are good at converting instructions into specific sequential tasks in the following JSON format:
{{
tasks : [{{
"task": "The description of task to be done with details needed given by user. You MUST include the user's address if needed."
"task_id": int, // Task number, which represents the task number, starting from 0
"task": "Concise description of task to be done with details needed given by user"
"agent": "The agent that best fits to execute the task"
"expected_output":"Description of expected output for the task"
"context": [int] // Index of tasks that will have their output used as context for this task (Always start from 0), if applicable. Eg. [1, 3] or None
"context": [int] // The task_id that task execution depends on indicates which task_id execution result depends on which task execution depends. Eg. [0, 1] or null if no tasks need to be depended on.
"extra_information": Any extra information as string with description given by the user needed to execute the task, if applicable.
}}]
}}
Please note that you are only permitted to output json content.

The specific tasks will be created based on the available agents role, goal and available tools:
{agents_information}

IMPORTANT: After all tasks are executed, the prepared transactions will be sent to the Ethereum network.

Please note: You can only output content in json format, and do not output any other content!
cbrzn marked this conversation as resolved.
Show resolved Hide resolved

You can refer to the following examples:
Goal: Send 1 ETH from the address 0x5d15311D760511d89cFad67404131cdc155E9FDB to the ENS domain vitalik.eth.
Output:
{{
"tasks": [{{
"task": "Check the balance of 1 ETH for the address 0x5d15311D760511d89cFad67404131cdc155E9FDB",
"agent": "send-tokens",
"expected_output": "A numeric value representing the balance of ETH in the account",
"context": null,
"extra_information": "Make sure the balance is at least 1 ETH to proceed with the transaction."
}}, {{
"task": "Prepare a transfer transaction of 1 ETH to the ENS domain vitalik.eth",
"agent": "send-tokens",
"expected_output": "A message confirming that the ETH transfer transaction has been prepared",
"context": [0],
"extra_information": "Use the output of the first task to ensure there is enough balance before proceeding."
}}]
}}

Goal: Execute a swap from ETH to 100 USDC for the user's address 0x5d15311D760511d89cFad67404131cdc155E9FDB
Output:
{{
"tasks": [
{{
"task": "Get current ETH balance of the user",
"agent": "send-tokens",
"expected_output": "Current ETH balance of 0x5d15311D760511d89cFad67404131cdc155E9FDB",
"context": null,
"extra_information": "Use Get ETH balance tool to check current ETH balance of user's address."
}},
{{
"task": "Calculate ETH amount needed to swap for 100 USDC",
"agent": "swap-tokens",
"expected_output": "The amount of ETH needed to execute a swap to receive 100 USDC",
"context": null,
"extra_information": "Assuming the swap service or agent has the capability to estimate, based on existing market rates, how much ETH is required."
}},
{{
"task": "Build transactions needed to execute the swap from ETH to 100 USDC",
"agent": "swap-tokens",
"expected_output": "Transactions prepared for swapping ETH to exactly 100 USDC",
"context": [2],
"extra_information": "Use the calculated ETH amount from the previous step to prepare the necessary transactions for the swap, ensuring that the exact_input is set to True since we want to receive an exact amount of USDC."
}}
]
}}

Goal: First, swap Ethereum (ETH) to 0.05 Wrapped Bitcoin (WBTC), ensuring the right amount of ETH is swapped for exactly 0.05 WBTC. After successfully swapping to WBTC, proceed to swap 0.05 WBTC to 1000 USD Coin (USDC), making sure to receive exactly 1000 USDC from the swap. Following these swaps, transfer 50 USDC to the Ethereum Name Service (ENS) domain 'vitalik.eth'.
FromCSUZhou marked this conversation as resolved.
Show resolved Hide resolved
Output:
{{
"tasks": [
{{
"task": "Swap the required amount of Ethereum (ETH) for exactly 0.05 Wrapped Bitcoin (WBTC)",
"agent": "swap-tokens",
"expected_output": "A successful swap transaction from ETH to WBTC where the user ends up with exactly 0.05 WBTC",
"context": null,
"extra_information": "Use the exact_output mode for the swap, ensuring the exact amount of 0.05 WBTC is received. Calculate the required ETH amount based on current market rates."
}},
{{
"task": "Swap exactly 0.05 Wrapped Bitcoin (WBTC) for 1000 USD Coin (USDC)",
FromCSUZhou marked this conversation as resolved.
Show resolved Hide resolved
"agent": "swap-tokens",
"expected_output": "A successful swap transaction from WBTC to USDC where the user ends up with exactly 1000 USDC",
"context": null,
"extra_information": "Use the exact_output mode for the swap, ensuring the exact amount of 1000 USDC is received. The exact amount of 0.05 WBTC is to be swapped."
cbrzn marked this conversation as resolved.
Show resolved Hide resolved
FromCSUZhou marked this conversation as resolved.
Show resolved Hide resolved
}},
{{
"task": "Transfer 50 USDC to the Ethereum Name Service (ENS) domain 'vitalik.eth'",
"agent": "send-tokens",
"expected_output": "A confirmation message that the transaction to transfer 50 USDC to 'vitalik.eth' has been prepared",
"context": null,
"extra_information": "Ensure to use the USDC token symbol in the transfer command and verify the correct ENS domain name before executing the transfer."
}}
]
}}

OK, start now!
Please note: You can only output content in json format, and do not output any other content!
Goal: {goal}
Output:
"""
)

Expand Down