Refactor/task memory cleanup

CHANGELOG.md

@@ -15,6 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Global event bus, `griptape.events.event_bus`, for publishing and subscribing to events.
 - Global config, `griptape.config.config`, for setting global configuration defaults.
 - Unique name generation for all `RagEngine` modules.
+- `ExtractionTool` Tool for having the LLM extract structured data from text.
+- `PromptSummaryTool` Tool for having the LLM summarize text.
+- `QueryTool` Tool for having the LLM query text.
 - Support for bitshift composition in `BaseTask` for adding parent/child tasks.
 
 ### Changed
@@ -32,11 +35,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **BREAKING**: Dropped `Client` from all Tool names for better naming consistency. 
 - **BREAKING**: Dropped `_client` suffix from all Tool packages. 
 - **BREAKING**: Added `Tool` suffix to all Tool names for better naming consistency. 
+- **BREAKING**: Removed `TextArtifactStorage.query` and `TextArtifactStorage.summarize`. 
+- **BREAKING**: Removed `TextArtifactStorage.rag_engine`, and `TextArtifactStorage.retrieval_rag_module_name`.
+- **BREAKING**: Removed `TextArtifactStorage.summary_engine`, `TextArtifactStorage.csv_extraction_engine`, and `TextArtifactStorage.json_extraction_engine`.
+- **BREAKING**: Removed `TaskMemory.summarize_namespace` and `TaskMemory.query_namespace`.
+- **BREAKING**: Removed `Structure.rag_engine`.
+- **BREAKING**: Split `JsonExtractionEngine.template_generator` into `JsonExtractionEngine.system_template_generator` and `JsonExtractionEngine.user_template_generator`.
+- **BREAKING**: Split `CsvExtractionEngine.template_generator` into `CsvExtractionEngine.system_template_generator` and `CsvExtractionEngine.user_template_generator`.
+- **BREAKING**: Changed `JsonExtractionEngine.template_schema` from a `run` argument to a class attribute. 
+- **BREAKING**: Changed `CsvExtractionEngine.column_names` from a `run` argument to a class attribute. 
+- **BREAKING**: Removed `JsonExtractionTask`, and `CsvExtractionTask` use `ExtractionTask` instead.
+- **BREAKING**: Removed `TaskMemoryClient`, use `RagClient`, `ExtractionTool`, or `PromptSummaryTool` instead.
 - **BREAKING**: `BaseTask.add_parent/child` now take a `BaseTask` instead of `str | BaseTask`.
 - Engines that previously required Drivers now pull from `griptape.config.config.drivers` by default.
 - `BaseTask.add_parent/child` will now call `self.structure.add_task` if possible.
 - `BaseTask.add_parent/child` now returns `self`, allowing for chaining.
 
+### Fixed
+- `JsonExtractionEngine` failing to parse json when the LLM outputs more than just the json.
+
 ## [0.29.1] - 2024-08-02
 
 ### Changed
@@ -442,7 +459,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `JsonExtractionTask` for convenience over using `ExtractionTask` with a `JsonExtractionEngine`.
 - `CsvExtractionTask` for convenience over using `ExtractionTask` with a `CsvExtractionEngine`.
 - `OpenAiVisionImageQueryDriver` to support queries on images using OpenAI's Vision model.
-- `ImageQueryClient` allowing an Agent to make queries on images on disk or in Task Memory.
+- `ImageQueryTool` allowing an Agent to make queries on images on disk or in Task Memory.
 - `ImageQueryTask` and `ImageQueryEngine`.
 
 ### Fixed 

README.md

@@ -89,13 +89,13 @@ With Griptape, you can create Structures, such as Agents, Pipelines, and Workflo
 
 ```python
 from griptape.structures import Agent
-from griptape.tools import WebScraperTool, FileManagerTool, TaskMemoryTool
+from griptape.tools import WebScraperTool, FileManagerTool, PromptSummaryTool
 
 agent = Agent(
     input="Load {{ args[0] }}, summarize it, and store it in a file called {{ args[1] }}.",
     tools=[
         WebScraperTool(off_prompt=True),
-        TaskMemoryTool(off_prompt=True),
+        PromptSummaryTool(off_prompt=True),
         FileManagerTool()
     ]
 )
@@ -104,42 +104,63 @@ agent.run("https://griptape.ai", "griptape.txt")
 
 And here is the output:
 ```
-[04/02/24 13:51:09] INFO     ToolkitTask 85700ec1b0594e1a9502c0efe7da6ef4
+[08/12/24 14:48:15] INFO     ToolkitTask c90d263ec69046e8b30323c131ae4ba0
                              Input: Load https://griptape.ai, summarize it, and store it in a file called griptape.txt.
-[04/02/24 13:51:15] INFO     Subtask db6a3e7cb2f549128c358149d340f91c
-                             Thought: First, I need to load the content of the website using the WebScraper action. Then, I will use the TaskMemoryClient action to
-                             summarize the content. Finally, I will save the summarized content to a file using the FileManager action.
+[08/12/24 14:48:16] INFO     Subtask ebe23832cbe2464fb9ecde9fcee7c30f
                              Actions: [
                                {
-                                 "name": "WebScraper",
+                                 "tag": "call_62kBnkswnk9Y6GH6kn1GIKk6",
+                                 "name": "WebScraperTool",
                                  "path": "get_content",
                                  "input": {
                                    "values": {
                                      "url": "https://griptape.ai"
                                    }
-                                 },
-                                 "tag": "load_website_content"
+                                 }
                                }
                              ]
-[04/02/24 13:51:16] INFO     Subtask db6a3e7cb2f549128c358149d340f91c
-                             Response: Output of "WebScraper.get_content" was stored in memory with memory_name "TaskMemory" and artifact_namespace
-                             "752b38bb86da4baabdbd9f444eb4a0d1"
-[04/02/24 13:51:19] INFO     Subtask c3edba87ebf845d4b85e3a791f8fde8d
-                             Thought: Now that the website content is loaded into memory, I need to summarize it using the TaskMemoryClient action.
-                             Actions: [{"tag": "summarize_content", "name": "TaskMemoryClient", "path": "summarize", "input": {"values": {"memory_name": "TaskMemory",
-                             "artifact_namespace": "752b38bb86da4baabdbd9f444eb4a0d1"}}}]
-[04/02/24 13:51:25] INFO     Subtask c3edba87ebf845d4b85e3a791f8fde8d
-                             Response: Output of "TaskMemoryClient.summarize" was stored in memory with memory_name "TaskMemory" and artifact_namespace
-                             "c4f131c201f147dcab07be3925b46294"
-[04/02/24 13:51:33] INFO     Subtask 06fe01ca64a744b38a8c08eb152aaacb
-                             Thought: Now that the content has been summarized and stored in memory, I need to save this summarized content to a file named 'griptape.txt'
-                             using the FileManager action.
-                             Actions: [{"tag": "save_summarized_content", "name": "FileManager", "path": "save_memory_artifacts_to_disk", "input": {"values": {"dir_name":
-                             ".", "file_name": "griptape.txt", "memory_name": "TaskMemory", "artifact_namespace": "c4f131c201f147dcab07be3925b46294"}}}]
-                    INFO     Subtask 06fe01ca64a744b38a8c08eb152aaacb
-                             Response: saved successfully
-[04/02/24 13:51:35] INFO     ToolkitTask 85700ec1b0594e1a9502c0efe7da6ef4
-                             Output: The summarized content of the website https://griptape.ai has been successfully saved to a file named 'griptape.txt'.
+[08/12/24 14:48:17] INFO     Subtask ebe23832cbe2464fb9ecde9fcee7c30f
+                             Response: Output of "WebScraperTool.get_content" was stored in memory with memory_name "TaskMemory" and artifact_namespace
+                             "cecca28eb0c74bcd8c7119ed7f790c95"
+[08/12/24 14:48:18] INFO     Subtask dca04901436d49d2ade86cd6b4e1038a
+                             Actions: [
+                               {
+                                 "tag": "call_o9F1taIxHty0mDlWLcAjTAAu",
+                                 "name": "PromptSummaryTool",
+                                 "path": "summarize",
+                                 "input": {
+                                   "values": {
+                                     "summary": {
+                                       "memory_name": "TaskMemory",
+                                       "artifact_namespace": "cecca28eb0c74bcd8c7119ed7f790c95"
+                                     }
+                                   }
+                                 }
+                               }
+                             ]
+[08/12/24 14:48:21] INFO     Subtask dca04901436d49d2ade86cd6b4e1038a
+                             Response: Output of "PromptSummaryTool.summarize" was stored in memory with memory_name "TaskMemory" and artifact_namespace
+                             "73765e32b8404e32927822250dc2ae8b"
+[08/12/24 14:48:22] INFO     Subtask c233853450fb4fd6a3e9c04c52b33bf6
+                             Actions: [
+                               {
+                                 "tag": "call_eKvIUIw45aRYKDBpT1gGKc9b",
+                                 "name": "FileManagerTool",
+                                 "path": "save_memory_artifacts_to_disk",
+                                 "input": {
+                                   "values": {
+                                     "dir_name": ".",
+                                     "file_name": "griptape.txt",
+                                     "memory_name": "TaskMemory",
+                                     "artifact_namespace": "73765e32b8404e32927822250dc2ae8b"
+                                   }
+                                 }
+                               }
+                             ]
+                    INFO     Subtask c233853450fb4fd6a3e9c04c52b33bf6
+                             Response: Successfully saved memory artifacts to disk
+[08/12/24 14:48:23] INFO     ToolkitTask c90d263ec69046e8b30323c131ae4ba0
+                             Output: The content from https://griptape.ai has been summarized and stored in a file called `griptape.txt`.
 ```
 
 During the run, the Griptape Agent loaded a webpage with a [Tool](https://docs.griptape.ai/stable/griptape-tools/), stored its full content in [Task Memory](https://docs.griptape.ai/stable/griptape-framework/structures/task-memory.md), queried it to answer the original question, and finally saved the answer to a file.

_typos.toml

@@ -1,3 +1,6 @@
+[default]
+extend-ignore-re = ["call_[[:alnum:]]+"]
+
 [default.extend-words]
 # Don't correct the state ND
 ND = "ND"

docs/examples/multiple-agent-shared-memory.md

@@ -1,6 +1,6 @@
-This example shows how to use one `Agent` to load content into `TaskMemory` and get that content from another `Agent` using `TaskMemoryClient`.
+This example shows how to use one `Agent` to load content into `TaskMemory` and get that content from another `Agent` using `QueryTool`.
 
-The first `Agent` uses a remote vector store (`MongoDbAtlasVectorStoreDriver` in this example) to handle memory operations. The second `Agent` uses the same instance of `TaskMemory` and the `TaskMemoryClient` with the same `MongoDbAtlasVectorStoreDriver` to get the data.
+The first `Agent` uses a remote vector store (`MongoDbAtlasVectorStoreDriver` in this example) to handle memory operations. The second `Agent` uses the same instance of `TaskMemory` and the `QueryTool` with the same `MongoDbAtlasVectorStoreDriver` to get the data.
 
 The `MongoDbAtlasVectorStoreDriver` assumes that you have a vector index configured where the path to the content is called `vector`, and the number of dimensions set on the index is `1536` (this is a commonly used number of dimensions for embedding models).
 

docs/examples/src/multi_agent_workflow_1.py

@@ -5,7 +5,7 @@
 from griptape.structures import Agent, Workflow
 from griptape.tasks import PromptTask, StructureRunTask
 from griptape.tools import (
-    TaskMemoryTool,
+    PromptSummaryTool,
     WebScraperTool,
     WebSearchTool,
 )
@@ -38,7 +38,7 @@ def build_researcher() -> Agent:
             WebScraperTool(
                 off_prompt=True,
             ),
-            TaskMemoryTool(off_prompt=False),
+            PromptSummaryTool(off_prompt=False),
         ],
         rulesets=[
             Ruleset(

docs/examples/src/multiple_agent_shared_memory_1.py

@@ -4,7 +4,7 @@
 from griptape.config.drivers import AzureOpenAiDriverConfig
 from griptape.drivers import AzureMongoDbVectorStoreDriver, AzureOpenAiEmbeddingDriver
 from griptape.structures import Agent
-from griptape.tools import TaskMemoryTool, WebScraperTool
+from griptape.tools import QueryTool, WebScraperTool
 
 AZURE_OPENAI_ENDPOINT_1 = os.environ["AZURE_OPENAI_ENDPOINT_1"]
 AZURE_OPENAI_API_KEY_1 = os.environ["AZURE_OPENAI_API_KEY_1"]
@@ -47,7 +47,7 @@
 )
 asker = Agent(
     tools=[
-        TaskMemoryTool(off_prompt=False),
+        QueryTool(off_prompt=False),
     ],
     meta_memory=loader.meta_memory,
     task_memory=loader.task_memory,

docs/examples/src/query_webpage_astra_db_1.py

@@ -14,7 +14,7 @@
 from griptape.engines.rag.stages import ResponseRagStage, RetrievalRagStage
 from griptape.loaders import WebLoader
 from griptape.structures import Agent
-from griptape.tools import RagTool, TaskMemoryTool
+from griptape.tools import RagTool
 
 namespace = "datastax_blog"
 input_blogpost = "www.datastax.com/blog/indexing-all-of-wikipedia-on-a-laptop"
@@ -53,5 +53,5 @@
     description="A DataStax blog post",
     rag_engine=engine,
 )
-agent = Agent(tools=[rag_tool, TaskMemoryTool(off_prompt=False)])
+agent = Agent(tools=[rag_tool])
 agent.run("What engine made possible to index such an amount of data, " "and what kind of tuning was required?")

docs/griptape-framework/data/artifacts.md

@@ -40,11 +40,11 @@ Each blob has a [name](../../reference/griptape/artifacts/base_artifact.md#gript
 
 ## Image
 
-An [ImageArtifact](../../reference/griptape/artifacts/image_artifact.md) is used for passing images back to the LLM. In addition to binary image data, an Image Artifact includes image metadata like MIME type, dimensions, and prompt and model information for images returned by [image generation Drivers](../drivers/image-generation-drivers.md). It inherits from [BlobArtifact](#blobartifact).
+An [ImageArtifact](../../reference/griptape/artifacts/image_artifact.md) is used for passing images back to the LLM. In addition to binary image data, an Image Artifact includes image metadata like MIME type, dimensions, and prompt and model information for images returned by [image generation Drivers](../drivers/image-generation-drivers.md). It inherits from [BlobArtifact](#blob).
 
 ## Audio
 
-An [AudioArtifact](../../reference/griptape/artifacts/audio_artifact.md) allows the Framework to interact with audio content. An Audio Artifact includes binary audio content as well as metadata like format, duration, and prompt and model information for audio returned generative models. It inherits from [BlobArtifact](#blobartifact).
+An [AudioArtifact](../../reference/griptape/artifacts/audio_artifact.md) allows the Framework to interact with audio content. An Audio Artifact includes binary audio content as well as metadata like format, duration, and prompt and model information for audio returned generative models. It inherits from [BlobArtifact](#blob).
 
 ## Boolean
 

docs/griptape-framework/data/loaders.md

@@ -75,7 +75,7 @@ Inherits from the [TextLoader](../../reference/griptape/loaders/text_loader.md)
 !!! info
     This driver requires the `loaders-image` [extra](../index.md#extras).
 
-The Image Loader is used to load an image as an [ImageArtifact](./artifacts.md#imageartifact). The Loader operates on image bytes that can be sourced from files on disk, downloaded images, or images in memory.
+The Image Loader is used to load an image as an [ImageArtifact](./artifacts.md#image). The Loader operates on image bytes that can be sourced from files on disk, downloaded images, or images in memory.
 
 ```python
 --8<-- "docs/griptape-framework/data/src/loaders_7.py"
@@ -104,7 +104,7 @@ Can be used to load email from an imap server:
 !!! info
     This driver requires the `loaders-audio` [extra](../index.md#extras).
 
-The [Audio Loader](../../reference/griptape/loaders/audio_loader.md) is used to load audio content as an [AudioArtifact](./artifacts.md#audioartifact). The Loader operates on audio bytes that can be sourced from files on disk, downloaded audio, or audio in memory.
+The [Audio Loader](../../reference/griptape/loaders/audio_loader.md) is used to load audio content as an [AudioArtifact](./artifacts.md#audio). The Loader operates on audio bytes that can be sourced from files on disk, downloaded audio, or audio in memory.
 
 The Loader will load audio in its native format and populates the resulting Artifact's `format` field by making a best-effort guess of the underlying audio format using the `filetype` package.
 

docs/griptape-framework/drivers/src/embedding_drivers_10.py

@@ -5,15 +5,20 @@
     VoyageAiEmbeddingDriver,
 )
 from griptape.structures import Agent
-from griptape.tools import TaskMemoryTool, WebScraperTool
+from griptape.tools import PromptSummaryTool, WebScraperTool
+
+config.drivers = DriverConfig(
+    prompt=OpenAiChatPromptDriver(model="gpt-4o"),
+    embedding=VoyageAiEmbeddingDriver(),
+)
 
 config.drivers = DriverConfig(
     prompt=OpenAiChatPromptDriver(model="gpt-4o"),
     embedding=VoyageAiEmbeddingDriver(),
 )
 
 agent = Agent(
-    tools=[WebScraperTool(off_prompt=True), TaskMemoryTool(off_prompt=False)],
+    tools=[WebScraperTool(off_prompt=True), PromptSummaryTool(off_prompt=False)],
 )
 
 agent.run("based on https://www.griptape.ai/, tell me what Griptape is")

docs/griptape-framework/drivers/src/web_scraper_drivers_3.py

@@ -1,15 +1,14 @@
 from griptape.drivers import MarkdownifyWebScraperDriver
 from griptape.loaders import WebLoader
 from griptape.structures import Agent
-from griptape.tools import TaskMemoryTool, WebScraperTool
+from griptape.tools import WebScraperTool
 
 agent = Agent(
     tools=[
         WebScraperTool(
             web_loader=WebLoader(web_scraper_driver=MarkdownifyWebScraperDriver(timeout=1000)),
-            off_prompt=True,
+            off_prompt=False,
         ),
-        TaskMemoryTool(off_prompt=False),
     ],
 )
 agent.run("List all email addresses on griptape.ai in a flat numbered markdown list.")

docs/griptape-framework/drivers/src/web_search_drivers_2.py

@@ -2,7 +2,7 @@
 
 from griptape.drivers import GoogleWebSearchDriver
 from griptape.structures import Agent
-from griptape.tools import TaskMemoryTool, WebSearchTool
+from griptape.tools import PromptSummaryTool, WebSearchTool
 
 agent = Agent(
     tools=[
@@ -12,7 +12,7 @@
                 search_id=os.environ["GOOGLE_API_SEARCH_ID"],
             ),
         ),
-        TaskMemoryTool(off_prompt=False),
+        PromptSummaryTool(off_prompt=False),
     ],
 )
 agent.run("Give me some websites with information about AI frameworks.")