You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When "criteria" is used in the evaluation engine, during the automagic generation of evaluation_steps, the intermediate JSON/dict is sometimes improperly formatted, leading to runtime errors.
Steps to Reproduce:
Running this as-is will work (sending something to the evaluation steps directly). Commenting "evaluation_steps" out, and "criteria" in, shows the problem:
from griptape.structures import Pipeline
from griptape.engines import EvalEngine
from griptape.tasks import PromptTask
from griptape.rules import Rule
from dotenv import load_dotenv
load_dotenv() # Load the environment variables
rules = [
"Answer with a json object, with no additional markup.",
"Talk like a pirate.",
]
pipeline = Pipeline(
tasks = [
PromptTask(
"Respond to this user: '{{ args[0] }}'"
"{% if args[1] %}Use this feedback when answering.{{ args[1] }}{% endif %}"
),
],
rules=[ Rule( rule ) for rule in rules ],
)
engine = EvalEngine(
#criteria=[
evaluation_steps=[
f"Determine whether the following rules have been met: {rules}",
]
)
pipeline.run( "Hi there" )
score, reason = engine.evaluate(
input = pipeline.tasks[0].input.value,
actual_output = pipeline.output.value,
)
Expected Behavior: The intermediate JSON/dict for evaluation_steps should always include a steps key, an be properly formatted for use.
Description:
When "criteria" is used in the evaluation engine, during the automagic generation of evaluation_steps, the intermediate JSON/dict is sometimes improperly formatted, leading to runtime errors.
Steps to Reproduce:
Running this as-is will work (sending something to the evaluation steps directly). Commenting "evaluation_steps" out, and "criteria" in, shows the problem:
Expected Behavior: The intermediate JSON/dict for evaluation_steps should always include a steps key, an be properly formatted for use.
Environment:
Griptape version: 1.1.1
Python version: 3.10-3.12
OS: OSX
Debug text pulled to look at the json directly
How the error reads:
The text was updated successfully, but these errors were encountered: