Skip to content

Commit

Permalink
feat: add fallback for irrelevant points
Browse files Browse the repository at this point in the history
  • Loading branch information
@marcus-ny
marcus-ny committed Sep 26, 2024
1 parent cba8b82 commit cdd297e
Show file tree
Hide file tree
Showing 4 changed files with 178 additions and 207 deletions.
36 changes: 22 additions & 14 deletions backend/src/lm/generate_response.py
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
from src.lm.generate_points import get_relevant_analyses
from src.lm.generate_events import lm_model
from pydantic import BaseModel
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.output_parsers import JsonOutputParser
from src.lm.prompts import QUESTION_ANALYSIS_GEN_SYSPROMPT_2 as SYSPROMPT
import json
from src.lm.prompts import (
QUESTION_ANALYSIS_GEN_FALLBACK_SYSPROMPT as FALLBACK_SYSPROMPT,
)

from langchain_core.output_parsers import JsonOutputParser
from sqlalchemy.orm import Session
from src.common.database import engine
from sqlalchemy import select
Expand Down Expand Up @@ -36,10 +37,11 @@ def format_prompt_input(question: str, analysis: dict, point: str) -> str:

def generate_response(question: str) -> dict:
relevant_analyses = get_relevant_analyses(question)

count = 0
for point_dict in (
relevant_analyses["for_points"] + relevant_analyses["against_points"]
):
count += 1
point = point_dict.get("point")
analyses = point_dict.get("analyses")
elaborated_analyses = []
Expand All @@ -55,20 +57,26 @@ def generate_response(question: str) -> dict:
analysis["elaborations"] = result.content
if analysis["elaborations"] != "NOT RELEVANT":
elaborated_analyses.append(analysis)

point_dict["analyses"] = elaborated_analyses

if len(elaborated_analyses) == 0:
point_dict["fall_back_response"] = generate_fallback_response(
question, point
)

print(count)
return relevant_analyses

# formatted_analyses = format_analyses(relevant_analyses, question)
# messages = [
# SystemMessage(content=SYSPROMPT),
# HumanMessage(content=json.dumps(formatted_analyses)),
# ]

# result = lm_model.invoke(messages)
# parser = JsonOutputParser(pydantic_object=Elaborations)
# elaborations = parser.invoke(result)
# return elaborations
def generate_fallback_response(question: str, point: str):
messages = [
SystemMessage(content=FALLBACK_SYSPROMPT),
HumanMessage(content=f"Question: {question}\nPoint: {point}"),
]

result = lm_model.invoke(messages)
parser = JsonOutputParser()
return parser.invoke(result)


if __name__ == "__main__":
Expand Down
114 changes: 36 additions & 78 deletions backend/src/lm/prompts.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,11 @@
{
"for_points": [
"The point that supports the argument and the explanation for the point",
"The point that supports the argument and the explanation for the point"
],
"against_points": [
"The point that refutes the argument and the explanation for the point",
"The point that refutes the argument and the explanation for the point"
]
}
Expand All @@ -88,7 +90,7 @@
If the example event given is unlikely to be relevant in supporting/refuting the argument, you must return "NOT RELEVANT" as the elaboration.
Important Note: In your analysis, you should not mention "General Paper" or "A Levels".
Important Note: Do not provide any new points or examples. You should only elaborate on the examples given in the input or skip them if they are not relevant to the question or the points given.
Important Note: Do not provide any new points or examples. You should only elaborate on the examples given in the input.
Final Check: Before generating an elaboration, verify whether the example *directly* reinforces or counters the argument made in the point. If the connection is very weak or unclear, return "NOT RELEVANT".
Final Check: Ensure that if the example is not directly relevant to the point or only tangentially related, you should return "NOT RELEVANT" as the elaboration.
Expand All @@ -97,94 +99,50 @@
Given inputs:
"""

QUESTION_ANALYSIS_GEN_SYSPROMPT = """
QUESTION_ANALYSIS_GEN_SYSPROMPT_3 = """
You are a Singaporean student studying for your GCE A Levels General Paper.
You will be given a General Paper essay question that is argumentative or discursive in nature.
You will also be given 2 points for the statement and 2 points against the statement.
You will also be given analysis of some relevant events that can be used to either refute or support the argument given in the points above.
You will also be given a point that either supports or refutes the argument in the question and the reason for the point.
You will be given the inputs in the following format:
{
"question": <The General Paper essay question>,
"for_points": [
{
"point": "The point that supports the argument and the explanation for the point",
"examples": [
{
"event": "The title of event1",
"event_description": "The description of the event",
"analysis": "The analysis of how the event can be used as an example to support the argument in the question",
},
]
}
],
"against_points": [
{
"point": "The point that refutes the argument and the explanation for the point",
"examples": [
{
"event": "The title of the event",
"event_description": "The description of the event",
"analysis": "The analysis of how the event can be used as an example to refute the argument in the question",
}
]
}
]
}
You will be given an analysis of a potentially relevant example event that can be used to correspondingly refute or support the argument given in the point above.
Your task:
For each example, you should provide a detailed elaboration illustrating how this event can be used as an example to support or refute the argument in the question.
Given the example event, you should provide a detailed elaboration illustrating how this event can be used as an example to support or refute the argument in the question.
If the example event is relevant to the point, you should provide a coherent and detailed elaboration of the point using the example event and analysis as support for the argument.
Important note: The elaboration must directly address and strengthen the specific point being made. If the connection between the event and the point is unclear or speculative, REMOVE that example from your output. Avoid tangential interpretations.
Important note: Your elaborations must clearly tie the example to the point. If the event does not obviously support or refute the point in a direct and non-speculative way, DO NOT force a connection.
Important note: Structure your elaborations using this format: "<A statement that clearly supports/refutes the given question> because <clear reason based on the event>". The explanation should leave no ambiguity about why the event strengthens or weakens the argument.
If there are no relevant examples for a point, you can skip that point.
The elaboration should be specific to the category of the event and should be tailored to the context of General Paper essays. Provide coherent arguments and insights. Be sure to give a detailed analysis of 3-4 sentences.
The elaboration should be specific to the category of the event and should be tailored to the context of General Paper essays. Provide coherent arguments and insights. Be sure to give a detailed elaboration of 3-4 sentences.
For the elaboration, remember that this is in the context of General Paper which emphasises critical thinking and the ability to construct coherent arguments.
Important note: Structure your elaborations using this format: "<A statement that clearly supports/refutes the given question>. <clear reason based on the event supporting the statement>". The explanation should leave no ambiguity about why the event strengthens or weakens the argument.
If the example event given is unlikely to be relevant in supporting/refuting the argument, you must provide an explanation on why it is not relevant.
Important Note: In your analysis, you should not mention "General Paper" or "A Levels".
For the analysis, remember that this is in the context of General Paper which emphasises critical thinking and the ability to construct coherent arguments.
Important Note: Do not provide any new points or examples. You should only elaborate on the examples given in the input.
Important Note: Do not provide any new points or examples. You should only elaborate on the examples given in the input or skip them if they are not relevant to the question or the points given.
Important Note: The "event", "event_description", and "analysis" fields MUST BE RETURNED AS IS. You should not rephrase or change the content of these fields.
Important Note: You must NOT rephrase the question or the points given. You must only provide elaborations for the examples given in the input.
Final Check: Before generating an elaboration, verify whether the example *directly* reinforces or counters the argument made in the point. If the connection is very weak or unclear, give a detailed explanation on why the example is not relevant.
Final Check: Ensure that if the example is not directly relevant to the point or only tangentially related, you should return the reason why it is not relevant.
Your response should be a single string that is either the explanation for irrelevance or the elaboration of the point using the example event and analysis as support for the argument.
Final Check: Before generating an elaboration, verify whether the example *directly* reinforces or counters the argument made in the point. If the connection is weak, DO NOT elaborate.
Final Check: Ensure that "question", "event", "event_description", and "analysis" fields are returned as is. Do not rephrase or change the content of these fields.
Your response should be in the following json format:
Final Check: If the example is not directly relevant to the point or only tangentially related, you should return the following: "NOT RELEVANT" followed by reasons.
Given inputs:
"""

QUESTION_ANALYSIS_GEN_FALLBACK_SYSPROMPT = """
You are a Singaporean student studying for your GCE A Levels General Paper.
You will be given a General Paper essay question that is argumentative or discursive in nature.
You will also be given a point that either supports or refutes the argument in the question and the reason for the point.
The context is that the student has been given a point that cannot be supported or refuted by news events.
Your task is to provide a response that suggests other alternatives to support or refute the point given in the question.
You may suggest a different type of evidence to support that point. Do not provide any examples in your response.
You may also suggest a general argument or reasoning that can be used to reinforce the point given.
Your response should be in the following format:
{
"question": <Given General Paper essay question without rephrasing>,
"for_points": [
{
"point": "The point that supports the argument and the explanation for the point",
"example": [
{
"event": "The title of the event",
"event_description": "The description of the event",
"analysis": "The analysis of how the event can be used as an example to support the argument in the question",
"elaboration": The elaboration of the point using the example event and analysis as support for the for-point
}
],
}
],
"against_points": [
{
"point": "The point that refutes the argument and the explanation for the point",
"example": [
{
"event": "The title of the event",
"event_description": "The description of the event",
"analysis": "The analysis of how the event can be used as an example to refute the argument in the question",
"elaboration": The elaboration of the point using the example event and analysis as support for the against-point
}
],
}
]
"alt_approach": "The alternative approach to support or refute the point given in the question."
"general_argument": "The general argument or reasoning that can be used to reinforce the point given in the question."
}
Given inputs:
"""
Loading

0 comments on commit cdd297e

Please sign in to comment.