Release/v0.31.0 (#1128)

Co-authored-by: Zach Giordano <[email protected]> Co-authored-by: Ikko Eltociear Ashimine <[email protected]> Co-authored-by: Andrew French <[email protected]> Co-authored-by: dylanholmes <[email protected]> Co-authored-by: Matt Vallillo <[email protected]> Co-authored-by: torabshaikh <[email protected]> Co-authored-by: Aodhan Roche <[email protected]> Co-authored-by: Kyle Roche <[email protected]> Co-authored-by: Emily Danielson <[email protected]> Co-authored-by: CJ Kindel <[email protected]> Co-authored-by: Vasily Vasinov <[email protected]> Co-authored-by: hkhajgiwale <[email protected]> Co-authored-by: Harsh Khajgiwale <[email protected]> Co-authored-by: Anush <[email protected]> Co-authored-by: datashaman <[email protected]> Co-authored-by: Stefano Lottini <[email protected]> Co-authored-by: James Clarendon <[email protected]> Co-authored-by: Michal <[email protected]> Co-authored-by: William Price <[email protected]>
griptape-ai · Sep 3, 2024 · 2ba3e92 · 2ba3e92
1 parent 457259d
commit 2ba3e92
Show file tree

Hide file tree

Showing 105 changed files with 1,688 additions and 1,450 deletions.
diff --git a/.dependabot.yml b/.dependabot.yml
@@ -0,0 +1,10 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+## [0.31.0] - 2024-09-03
+
+**Note**: This release includes breaking changes. Please refer to the [Migration Guide](./MIGRATION.md#030x-to-031x) for details.
+
+### Added
+- Parameter `meta: dict` on `BaseEvent`.
+
+### Changed
+- **BREAKING**: Drivers, Loaders, and Engines now raise exceptions rather than returning `ErrorArtifacts`.
+- **BREAKING**: Parameter `driver` on `BaseConversationMemory` renamed to `conversation_memory_driver`.
+- **BREAKING**: `BaseConversationMemory.add_to_prompt_stack` now takes a `prompt_driver` parameter.
+- **BREAKING**: `BaseConversationMemoryDriver.load` now returns `tuple[list[Run], dict]`. This represents the runs and metadata.
+- **BREAKING**: `BaseConversationMemoryDriver.store` now takes `runs: list[Run]` and `metadata: dict` as input.
+- **BREAKING**: Parameter `file_path` on `LocalConversationMemoryDriver` renamed to `persist_file` and is now type `Optional[str]`.
+- `Defaults.drivers_config.conversation_memory_driver` now defaults to `LocalConversationMemoryDriver` instead of `None`.
+- `CsvRowArtifact.to_text()` now includes the header.
+
+### Fixed
+- Parsing streaming response with some OpenAI compatible services.
+- Issue in `PromptSummaryEngine` if there are no artifacts during recursive summarization.
+- Issue in `GooglePromptDriver` using Tools with no schema.
+- Missing `maxTokens` inference parameter in `AmazonBedrockPromptDriver`.
+- Incorrect model in `OpenAiDriverConfig`'s `text_to_speech_driver`.
+- Crash when using `CohereRerankDriver` with `CsvRowArtifact`s.
+
+
 ## [0.30.2] - 2024-08-26
 
 ### Fixed

diff --git a/MIGRATION.md b/MIGRATION.md
@@ -0,0 +1,87 @@
+# Migration Guide
+
+This document provides instructions for migrating your codebase to accommodate breaking changes introduced in new versions of Griptape.
+
+## 0.30.X to 0.31.X
+
+### Exceptions Over `ErrorArtifact`s
+
+Drivers, Loaders, and Engines now raise exceptions rather than returning `ErrorArtifact`s.
+Update any logic that expects `ErrorArtifact` to handle exceptions instead.
+
+#### Before
+```python
+artifacts = WebLoader().load("https://www.griptape.ai")
+
+if isinstance(artifacts, ErrorArtifact):
+    raise Exception(artifacts.value)
+```
+
+#### After
+```python
+try:
+    artifacts = WebLoader().load("https://www.griptape.ai")
+except Exception as e:
+    raise e
+```
+
+### LocalConversationMemoryDriver `file_path` renamed to `persist_file`
+
+`LocalConversationMemoryDriver.file_path` has been renamed to `persist_file` and is now `Optional[str]`. If `persist_file` is not passed as a parameter, nothing will be persisted and no errors will be raised. `LocalConversationMemoryDriver` is now the default driver in the global `Defaults` object.
+
+#### Before
+```python
+local_driver_with_file = LocalConversationMemoryDriver(
+    file_path="my_file.json"
+)
+
+local_driver = LocalConversationMemoryDriver()
+
+assert local_driver_with_file.file_path == "my_file.json"
+assert local_driver.file_path == "griptape_memory.json"
+```
+
+#### After
+```python
+local_driver_with_file = LocalConversationMemoryDriver(
+    persist_file="my_file.json"
+)
+
+local_driver = LocalConversationMemoryDriver()
+
+assert local_driver_with_file.persist_file == "my_file.json"
+assert local_driver.persist_file is None
+```
+
+### Changes to BaseConversationMemoryDriver
+
+`BaseConversationMemoryDriver.driver` has been renamed to `conversation_memory_driver`. Method signatures for `.store` and `.load` have been changed.
+
+#### Before
+```python
+memory_driver = LocalConversationMemoryDriver()
+
+conversation_memory = ConversationMemory(
+    driver=memory_driver
+)
+
+load_result: BaseConversationMemory = memory_driver.load()
+
+memory_driver.store(conversation_memory)
+```
+
+#### After
+```python
+memory_driver = LocalConversationMemoryDriver()
+
+conversation_memory = ConversationMemory(
+    conversation_memory_driver=memory_driver
+)
+
+load_result: tuple[list[Run], dict[str, Any]] = memory_driver.load()
+
+memory_driver.store(
+    conversation_memory.runs,
+    conversation_memory.meta
+)
+```
diff --git a/README.md b/README.md
@@ -169,6 +169,10 @@ The important thing to note here is that no matter how big the webpage is it can
 
 In the above example, we set [off_prompt](https://docs.griptape.ai/stable/griptape-framework/structures/task-memory.md#off-prompt) to `True`, which means that the LLM can never see the data it manipulates, but can send it to other Tools.
 
+> [!IMPORTANT]  
+> This example uses Griptape's [ToolkitTask](https://docs.griptape.ai/stable/griptape-framework/structures/tasks/#toolkit-task), which requires a highly capable LLM to function correctly. By default, Griptape uses the [OpenAiChatPromptDriver](https://docs.griptape.ai/stable/griptape-framework/drivers/prompt-drivers/#openai-chat); for another powerful LLM try swapping to the [AnthropicPromptDriver](https://docs.griptape.ai/stable/griptape-framework/drivers/prompt-drivers/#anthropic)!
+If you're using a less powerful LLM, consider using the [ToolTask](https://docs.griptape.ai/stable/griptape-framework/structures/tasks/#tool-task) instead, as the `ToolkitTask` might not work properly or at all.
+
 [Check out our docs](https://docs.griptape.ai/stable/griptape-framework/drivers/prompt-drivers/) to learn more about how to use Griptape with other LLM providers like Anthropic, Claude, Hugging Face, and Azure.
 
 ## Versioning

diff --git a/docs/examples/src/amazon_dynamodb_sessions_1.py b/docs/examples/src/amazon_dynamodb_sessions_1.py
@@ -18,7 +18,7 @@
 
 structure = Agent(
     conversation_memory=ConversationMemory(
-        driver=AmazonDynamoDbConversationMemoryDriver(
+        conversation_memory_driver=AmazonDynamoDbConversationMemoryDriver(
             session=boto3.Session(
                 aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
                 aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],

diff --git a/docs/examples/src/load_query_and_chat_marqo_1.py b/docs/examples/src/load_query_and_chat_marqo_1.py
@@ -1,7 +1,6 @@
 import os
 
 from griptape import utils
-from griptape.artifacts.error_artifact import ErrorArtifact
 from griptape.drivers import MarqoVectorStoreDriver, OpenAiEmbeddingDriver
 from griptape.loaders import WebLoader
 from griptape.structures import Agent
@@ -27,9 +26,6 @@
 # Load artifacts from the web
 artifacts = WebLoader().load("https://www.griptape.ai")
 
-if isinstance(artifacts, ErrorArtifact):
-    raise Exception(artifacts.value)
-
 # Upsert the artifacts into the vector store
 vector_store.upsert_text_artifacts(
     {

diff --git a/docs/examples/src/query_webpage_1.py b/docs/examples/src/query_webpage_1.py
@@ -1,14 +1,11 @@
 import os
 
-from griptape.artifacts.error_artifact import ErrorArtifact
 from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver
 from griptape.loaders import WebLoader
 
 vector_store = LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver(api_key=os.environ["OPENAI_API_KEY"]))
 
 artifacts = WebLoader(max_tokens=100).load("https://www.griptape.ai")
-if isinstance(artifacts, ErrorArtifact):
-    raise Exception(artifacts.value)
 
 for a in artifacts:
     vector_store.upsert_text_artifact(a, namespace="griptape")

diff --git a/docs/examples/src/query_webpage_astra_db_1.py b/docs/examples/src/query_webpage_astra_db_1.py
@@ -1,6 +1,5 @@
 import os
 
-from griptape.artifacts import ErrorArtifact
 from griptape.drivers import (
     AstraDbVectorStoreDriver,
     OpenAiChatPromptDriver,
@@ -45,8 +44,7 @@
 )
 
 artifacts = WebLoader(max_tokens=256).load(input_blogpost)
-if isinstance(artifacts, ErrorArtifact):
-    raise Exception(artifacts.value)
+
 vector_store_driver.upsert_text_artifacts({namespace: artifacts})
 
 rag_tool = RagTool(

diff --git a/docs/examples/src/talk_to_a_pdf_1.py b/docs/examples/src/talk_to_a_pdf_1.py
@@ -1,6 +1,5 @@
 import requests
 
-from griptape.artifacts.error_artifact import ErrorArtifact
 from griptape.drivers import LocalVectorStoreDriver, OpenAiChatPromptDriver, OpenAiEmbeddingDriver
 from griptape.engines.rag import RagEngine
 from griptape.engines.rag.modules import PromptResponseRagModule, VectorStoreRetrievalRagModule
@@ -32,8 +31,6 @@
 )
 
 artifacts = PdfLoader().load(response.content)
-if isinstance(artifacts, ErrorArtifact):
-    raise Exception(artifacts.value)
 
 vector_store.upsert_text_artifacts({namespace: artifacts})
 

diff --git a/docs/examples/src/talk_to_a_webpage_1.py b/docs/examples/src/talk_to_a_webpage_1.py
@@ -1,4 +1,3 @@
-from griptape.artifacts.error_artifact import ErrorArtifact
 from griptape.drivers import LocalVectorStoreDriver, OpenAiChatPromptDriver, OpenAiEmbeddingDriver
 from griptape.engines.rag import RagEngine
 from griptape.engines.rag.modules import PromptResponseRagModule, VectorStoreRetrievalRagModule
@@ -28,9 +27,6 @@
 
 artifacts = WebLoader().load("https://en.wikipedia.org/wiki/Physics")
 
-if isinstance(artifacts, ErrorArtifact):
-    raise Exception(artifacts.value)
-
 vector_store_driver.upsert_text_artifacts({namespace: artifacts})
 
 rag_tool = RagTool(

diff --git a/docs/griptape-cloud/data-sources/create-data-source.md b/docs/griptape-cloud/data-sources/create-data-source.md
@@ -0,0 +1,27 @@
+# Data Sources
+
+Data Sources are the first step to Griptape's RAG pipeline. They allow you to bring your own data to ingest and transform. You can then make one or more Data Source available to your AI applications via [Knowledge Bases](../knowledge-bases/create-knowledge-base.md)
+
+## Create a Data Source
+
+You can [create a Data Source in the Griptape Cloud console](https://cloud.griptape.ai/data-sources/create) by specifying the required configuration for your chosen Data Source in the cloud console.
+
+### Web Page
+
+You can scrape and ingest a single, public web page by providing a URL. If you wish to scrape multiple pages, you must create multiple Data Sources. However, you can then add all of the pages to the same Knowledge Base if you wish to access all the pages together.
+
+### Google Drive
+
+You can ingest documents and spreadsheets stored in a Google Drive account. We support all standard file formats such as text, markdown, spreadsheets, and presentations.
+
+### Confluence
+
+You can connect to your personal or company Confluence by providing a URL, [Atlassian API Token](https://id.atlassian.com/manage-profile/security/api-tokens), and the email address for the token holder's account. Each Confluence Data Source can be limited to a single Space in Confluence by specifying the [specific URL for that Space](https://support.atlassian.com/confluence-cloud/docs/use-spaces-to-organize-your-work/).
+
+### Structure (Experimental)
+
+You can specify a [Structure](../structures/create-structure.md) to run as a Data Source as long as your Structure returns a [`TextArtifact` or `ListArtifact` from the Griptape Framework](../../griptape-framework/data/artifacts.md). You can use this as a way to build custom Data Sources.
+
+## Other Data Source Types
+
+If you do not see a Data Source configuration you'd wish to use, you can submit a request via [Discord](https://discord.gg/gnWRz88eym) or `[email protected]`.
diff --git a/docs/griptape-cloud/data-sources/refresh-data.md b/docs/griptape-cloud/data-sources/refresh-data.md
@@ -0,0 +1,17 @@
+# Refresh a Data Source
+
+## Scheduled Refresh
+
+By default your Data Source will not refresh automatically. When creating a Data Source, you can enable scheduled refresh and specify a [CRON expression](https://crontab.guru/). For example, if you wish your Data Source to refresh every day at midnight PDT you can use the following expression: `0 7 * * *`.
+
+## Manual Refresh
+
+If you wish to manually refresh a Data Source you can do so either via the `Refresh` button in the cloud console or by API using the `Data Source ID` on the `Config` tab and a [Griptape Cloud API Key](https://cloud.griptape.ai/configuration/api-keys).
+
+The following shell commands will create a new data refresh job. You will need to specify your API key and data source id.
+
+```shell
+export GT_CLOUD_API_KEY=<your API key here>
+export DATA_SOURCE_ID=<your data source id here>
+curl -H "Authorization: Bearer ${GT_CLOUD_API_KEY}" --json '{}' https://cloud.griptape.ai/api/data-connectors/${DATA_SOURCE_ID}/data-jobs
+```
diff --git a/docs/griptape-cloud/index.md b/docs/griptape-cloud/index.md
@@ -1,5 +1,12 @@
 # Griptape Cloud
 
-Griptape Cloud provides managed services for your AI app stack. Deploy and scale end-to-end solutions, from LLM-powered data prep and retrieval to AI agents, pipelines and workflows.
+[Griptape Cloud](https://cloud.griptape.ai/) provides managed services for your AI app stack. Deploy and scale end-to-end solutions, from LLM-powered data prep and retrieval to AI Agents, Pipelines, and Workflows.
 
-To get started with AI Structures in the Cloud, check out the [managed-structure-template](https://github.com/griptape-ai/managed-structure-template) or deploy one of the [griptape-sample-structures](https://github.com/griptape-ai/griptape-sample-structures/tree/main).
+## Build Your Own RAG Pipeline
+Connect to your data with our [Data Sources](data-sources/create-data-source.md) and prepare them for retrieval with [Knowledge Bases](knowledge-bases/create-knowledge-base.md).
+
+## Host and Run Your Code
+Have Griptape code? Have existing code with another LLM framework? You can host your Python code using [Structures](structures/create-structure.md) whether it uses the Griptape Framework or not.
+
+## APIs
+All of our features can be called via API with a [Griptape Cloud API Key](https://cloud.griptape.ai/configuration/api-keys). See the [API Reference](api/api-reference.md) for detailed information.
diff --git a/docs/griptape-cloud/knowledge-bases/accessing-data.md b/docs/griptape-cloud/knowledge-bases/accessing-data.md
@@ -0,0 +1,33 @@
+# Accessing Data in a Knowledge Base
+
+You can `Search` or `Query` the Knowledge Base for information contained in your Data Sources. `Search` will return a natural language response while `Query` will return the individual entries. Use whichever one best fits your use case.
+
+## From the Cloud Console
+
+You can explore your data with a natural language question on the `Test` tab of your Knowledge Base. Compare and contrast the results of `Search` vs. `Query` to understand which is correct for your application.
+
+## From the API
+
+You can enact both `Search` and `Query` via the API by hitting their respective endpoints using a [Griptape Cloud API Key](https://cloud.griptape.ai/configuration/api-keys) and the Knowledge Base ID found on the `Config` tab of your Knowledge Base. 
+
+The following example commands will send the string `"test question"` and return the results from the Knowledge Base.
+
+### Search
+
+```shell
+export GT_CLOUD_API_KEY=<your API key here>
+export KNOWLEDGE_BASE_ID=<your knowledge base id here>
+curl -H "Authorization: Bearer ${GT_CLOUD_API_KEY}" --json '{"query": "test question"}' https://cloud.griptape.ai/api/knowledge-bases/${KNOWLEDGE_BASE_ID}/search
+```
+
+### Query
+
+```shell
+export GT_CLOUD_API_KEY=<your API key here>
+export KNOWLEDGE_BASE_ID=<your knowledge base id here>
+curl -H "Authorization: Bearer ${GT_CLOUD_API_KEY}" --json '{"query": "test question"}' https://cloud.griptape.ai/api/knowledge-bases/${KNOWLEDGE_BASE_ID}/query
+```
+
+## Using the Griptape Framework
+
+You can use the [GriptapeCloudKnowledgeBaseVectorStoreDriver](../../griptape-framework/drivers/vector-store-drivers.md/#griptape-cloud-knowledge-base) to query your Knowledge Base with Griptape and the [GriptapeCloudKnowledgeBaseTool](../../griptape-tools/official-tools/griptape-cloud-knowledge-base-tool.md) to search.
diff --git a/docs/griptape-cloud/knowledge-bases/create-knowledge-base.md b/docs/griptape-cloud/knowledge-bases/create-knowledge-base.md
@@ -0,0 +1,7 @@
+# Knowledge Bases
+
+Knowledge Bases are the way to organize and access your data ingested from [Data Sources](../data-sources/create-data-source.md). You can specify multiple Data Sources per Knowledge Base in order to access data ingested from different sources all in one place.
+
+## Create a Knowledge Base
+
+You can [create a Knowledge Base in the Griptape Cloud console](https://cloud.griptape.ai/knowledge-bases/create) by specifying which Data Sources you wish to include. Once created, you can [access your data](accessing-data.md).
diff --git a/docs/griptape-cloud/structures/create-structure.md b/docs/griptape-cloud/structures/create-structure.md
@@ -0,0 +1,17 @@
+# Structures
+
+Structures are a primary component in Griptape for organizing and executing Tasks against a LLM.
+
+## Create a Structure
+
+1. [Connect Your GitHub Account in your Griptape Cloud account](https://cloud.griptape.ai/account)
+1. Install the [Griptape Cloud GitHub app to your GitHub account or organization](https://github.com/apps/griptape-cloud/installations/new/)
+    - Be sure to allow the app access to `All Repositories` or select the specific repositories you need
+1. Ensure your repository has a Structure Config YAML file
+    - To learn more see [Structure Config YAML](structure-config.md)
+
+You can now [create a Structure in the Griptape Cloud console](https://cloud.griptape.ai/structures/create) by providing your GitHub repository information.
+
+### Quickstart With Samples and Templates
+
+To get started with Structures in the Cloud, check out the [managed-structure-template on GitHub](https://github.com/griptape-ai/managed-structure-template) or deploy one of the [griptape-sample-structures from GitHub](https://github.com/griptape-ai/griptape-sample-structures/tree/main).
diff --git a/docs/griptape-cloud/structures/run-structure.md b/docs/griptape-cloud/structures/run-structure.md
@@ -0,0 +1,32 @@
+# Running a Structure
+
+Once your Structure is created and deployed, you can run your Structure one of three ways outlined below. You view the output of any of your runs, no matter how you created them, in the `Runs` tab of your Structure.
+
+## From the Cloud Console
+
+In the cloud console, click on the name of the Structure you wish to run and then go to the `Test` tab. Here you can specify arguments to pass to your Structure run and any run-specific environment variables you need.
+
+When passing arguments through the cloud console, pass each new argument on a new line. For example if your local code is ran with the inputs `-i input_file.txt` then the arguments you would pass in the cloud would be:
+
+```
+-i
+input_file.txt
+```
+
+## From the API
+
+You can run your Structure via the API using CURL or any other code that can make HTTP requests. You will need a [Griptape Cloud API Key](https://cloud.griptape.ai/configuration/api-keys) and the `Structure Invocation URL` which is located on the `Config` tab of your Structure. 
+
+The example below will kick off a run with the args you pass as a json object.
+
+```shell
+export GT_CLOUD_API_KEY=<your API key here>
+export INVOCATION_URL=<your structure invocation URL>
+curl -H "Authorization: Bearer ${GT_CLOUD_API_KEY}" --json '{"args": ["arg1"], ""env_vars"": [{"name":"var1", "value": "value"}]}' ${INVOCATION_URL}
+```
+
+For more information on other Structure run APIs, check out the [StructureRuns API docs](../api/api-reference.md/#/StructureRuns).
+
+## Using the Griptape Framework
+
+You can use [StructureRunDrivers](../../griptape-framework/drivers/structure-run-drivers.md/#griptape-cloud) to run your Structure with Griptape.
diff --git a/docs/griptape-framework/drivers/src/conversation_memory_drivers_1.py b/docs/griptape-framework/drivers/src/conversation_memory_drivers_1.py
@@ -2,8 +2,8 @@
 from griptape.memory.structure import ConversationMemory
 from griptape.structures import Agent
 
-local_driver = LocalConversationMemoryDriver(file_path="memory.json")
-agent = Agent(conversation_memory=ConversationMemory(driver=local_driver))
+local_driver = LocalConversationMemoryDriver(persist_file="memory.json")
+agent = Agent(conversation_memory=ConversationMemory(conversation_memory_driver=local_driver))
 
 agent.run("Surfing is my favorite sport.")
 agent.run("What is my favorite sport?")