Query on configuration of Langchain/Langgraph Agent flow for RAG to get structured output from the LLM

[nag-24]

Hello All,

I used the below code to retrieve a record from a table A (Transcription of an audio conversation with customer) in my Airtable Base X and perform a similarity search on a temporary vector store created in Chroma DB (I ingested a table from another Table B (Wholesale Price List) in my Airtable base X into this Chroma DB persistent store) and pass the retrieved doc to an LLM to create a structured output JSON to insert a record into another table C (Sales_Order_Template) of my my Airtable base X.

To make the scenario understandable, the LLM & Agent flow is being used to read the transcription of a sales conversation with a customer which had been inserted into Airtable and create a sales order template by performing a retrieval from the Wholesale Price List (another table).

The issue is that in the conversation the customer talks about ordering multiple say 10 pcs per store for 8 stores and 6 pcs per store for 4 stores. The LLM correctly calls the 'math_operation' tool to perform 10 * 8 = 80 and 6 * 4 = 24 but to get the total order qty
it should also add 80 + 24 = 104 but it hallucinates the answer here instead of again calling the 'math_operation" tool.

I even tried to make the prompt_template explicit with a few shot example but it didn't work.

I created the code based on https://python.langchain.com/v0.2/docs/how_to/migrate_agent/ (How to migrate from legacy LangChain agents to LangGraph)

Would appreciate your inputs to make this work.

import os
import json
from datetime import datetime
import pytz
from pyairtable import Api
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_core.prompts import PromptTemplate
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
from langsmith import traceable
from langsmith.wrappers import wrap_openai

LangSmith setup

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "xxxx"
os.environ["LANGCHAIN_PROJECT"] = "Sales_order"

Define the custom math tool

@tool("math_operation", return_direct=True)
def math_operation(operation: str, a: float, b: float) -> float:
"""Perform basic math operations: addition, subtraction, multiplication, and division."""
if operation == "add":
return a + b
elif operation == "subtract":
return a - b
elif operation == "multiply":
return a * b
elif operation == "divide":
if b == 0:
return "Error: Division by zero"
return a / b
else:
return "Error: Unsupported operation"

Define the retriever tool

@tool("retrieve_documents", return_direct=True)
def retrieve_documents(query: str) -> list:
"""Retrieve documents from the Chroma DB based on the query to create the airtable_pricelist_extract."""
docs = retriever.invoke(query)
return [doc.page_content for doc in docs]

Timezone handling for Singapore

def get_singapore_time():
tz = pytz.timezone('Asia/Singapore')
return datetime.now(tz).strftime("%d/%m/%Y")

Define variables

CUSTOMER_SALES_ORDER_NO = "SO001"
EXPECTED_SHIPMENT_DATE = "21/06/2024" # Format as required
CUSTOMER_ID = "C0001"
SALES_MANAGER_ID = "E0001"
TODAYS_DATE = get_singapore_time()

Initialize OpenAI and Chroma

openai_api_key = os.getenv('OPENAI_API_KEY')
chat_model = ChatOpenAI(model="gpt-4o", api_key=openai_api_key)

embeddings_model = OpenAIEmbeddings(openai_api_key=openai_api_key, model="text-embedding-3-large")
persist_directory = "./chroma_db"
db = Chroma(persist_directory=persist_directory, embedding_function=embeddings_model)
retriever = db.as_retriever(search_kwargs={"k": 69})

Fetch the engagement diary record from Airtable

AIRTABLE_TOKEN = os.getenv('AIRTABLE_TOKEN')
AIRTABLE_BASE_ID = 'xxxx'
ENGAGEMENT_DIARY_TABLE_NAME = 'xxxx'
api = Api(AIRTABLE_TOKEN)
engagement_table = api.table(AIRTABLE_BASE_ID, ENGAGEMENT_DIARY_TABLE_NAME)
record = engagement_table.first(formula={"Engagement_No": "E0001"})

Check if record is fetched successfully

if record:
# Extract the fields from the fetched record, excluding the IDs
filtered_record_content = {key: value for key, value in record['fields'].items() if not isinstance(value, list) or not key.endswith('_ID')}
transcription_summary = str(filtered_record_content)
print(f"Transcription Summary: {transcription_summary}")

# Define the prompt template as a string
prompt_template_str = """

You are an expert Sales Support Associate working for ABC Suppliers.
You will be given a transcription of a conversation between a sales manager of ABC Suppliers and a contact representing a customer of ABC Suppliers
which is retrieved from a record in the 'Customer_Engagement_Diary' Table in Airtable: {transcription_summary}

You will also be given
- 'Customer_Sales_Order_No': {Customer_Sales_Order_No}
- 'Customer_ID': {Customer_ID}
- 'Sales_Manager_ID': {Sales_Manager_ID}
- 'Expected_Shipment_Date': {Expected_Shipment_Date}
- 'Today's_date': {Todays_date}

Use the 'retrieve_documents' tool with the 'transcription_summary' as the query to retrieve relevant documents.
Also, the retrieved documents will be termed as the 'airtable_pricelist_extract' referred to below.
Do not perform any mathematical calculations on your own but use the 'math_operation' tool to perform simple maths calculations.
Remember sometimes you may need to perform multiple simple maths operations such as:
(a * b) + (c * d) = e

Example
if a=10, b=7, c=5, d=3
then you call the 'math_operation'tool 3 times to arrive at the value of e.
First Tool call to Use 'math_operation' tool to perform (10 * 7) = 70
Second Tool call to Use 'math_operation' tool to perform (5 * 3) = 15
Third Tool call to Use 'math_operation' tool to perform (70 + 15) = 85

Thus do not perform any mathematical calculations on your own. Use the 'math_operation' tool as many times as needed.

Your job is to prepare a JSON for inserting a record in Airtable based on the above inputs and the below format & instructions.

Here is the structure for the JSON:
{{
"create_record": [
{{
"fields": {{
"Customer_Sales_Order_No": "{Customer_Sales_Order_No}",
"Customer_ID": "{Customer_ID}",
"Sales_Order_Date": "{Todays_date}",
"Expected_Shipment_Date": "{Expected_Shipment_Date}",
"Sales_Manager_ID": "{Sales_Manager_ID}",
"Product_No": "",
"Quantity": "",
"Price_Inc_GST": ""
}}
}}
]
}}

Instructions for deriving the values:

• "Sales_Order_Date" should be taken as '{Todays_date}'.
• "Expected_Shipment_Date" should be taken as the value provided '{Expected_Shipment_Date}'.
• "Product_No" should be derived from the 'transcription_summary' and the 'airtable_pricelist_extract'.
• "Quantity" should be derived from the 'transcription_summary' and the 'airtable_pricelist_extract'.
• "Price_Inc_GST" should be derived from the 'transcription_summary', the 'airtable_pricelist_extract', and the given price conditions below:
• 'Customer_Sales_Order_No': {Customer_Sales_Order_No}
• 'Customer_ID': {Customer_ID}
• 'Sales_Manager_ID': {Sales_Manager_ID}

Price conditions:

• If a record only has a value for 'Usual_Price_(inc_gst)', then please take 'Price_Inc_GST' = value of 'Usual_Price_(inc_gst)'.
• If a record has a value for 'Special_Price_(inc_gst)', then please take 'Price_Inc_GST' = value of 'Special_Price_(inc_gst)'.

Transcription Summary:
{transcription_summary}

If multiple items are ordered by the customer, then please include all the items to be recorded in the JSON template as separate sections with the JSON as needed.
Ensure that the "Product_No" and "Quantity" are numbers, and "Price_Inc_GST" is a currency with a "$" symbol.
"""

# Create the PromptTemplate
prompt_template = PromptTemplate(input_variables=[
    'transcription_summary', 'Customer_Sales_Order_No', 'Customer_ID', 
    'Sales_Manager_ID', 'Expected_Shipment_Date', 'Todays_date'], 
    template=prompt_template_str)

# Initialize the model and bind the tools
tools = [retrieve_documents, math_operation]
app = create_react_agent(chat_model, tools)

# Prepare the initial prompt with provided values
initial_prompt = prompt_template.invoke({
    "transcription_summary": transcription_summary,
    "Todays_date": TODAYS_DATE,
    "Customer_Sales_Order_No": CUSTOMER_SALES_ORDER_NO,
    "Expected_Shipment_Date": EXPECTED_SHIPMENT_DATE,
    "Customer_ID": CUSTOMER_ID,
    "Sales_Manager_ID": SALES_MANAGER_ID
})

# Define the inputs for the agent
query = initial_prompt['text'] if isinstance(initial_prompt, dict) and 'text' in initial_prompt else str(initial_prompt)

# Invoke the agent
messages = app.invoke({"messages": [("human", query)]})
output = messages["messages"][-1].content

# Output the generated JSON
output_json_filepath = r"C:\Users\....\Sales Order\Customer_SO_Template_6.json"
with open(output_json_filepath, 'w', encoding='utf-8') as file:
    file.write(output)

print(f"Insert JSON saved successfully to: {output_json_filepath}")

else:
print("No matching record found.")