Skip to content

Commit

Permalink
Prepare notebook for Assistants API hackathon (#1102)
Browse files Browse the repository at this point in the history 
* pin openai version for hackathon

* runnable e2e

* add migration guide
  • Loading branch information
@daniel-huang-1230
daniel-huang-1230 authored Apr 26, 2024
1 parent 47e7d77 commit 46d362e
Showing 1 changed file with 144 additions and 31 deletions.
175 changes: 144 additions & 31 deletions trulens_eval/examples/expositional/frameworks/openai_assistants/openai_assistants_api.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,59 +15,76 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set keys"
"\n",
"[**Important**] Notice in this example notebook, we are using Assistants API V1 (hence the pinned version of `openai` below) so that we can evaluate against retrieved source.\n",
"At some very recent point in time as of April 2024, OpenAI removed the [\"quote\" attribute from file citation object in Assistants API V2](https://platform.openai.com/docs/api-reference/messages/object#messages/object-content) due to stability issue of this feature. See response from OpenAI staff https://community.openai.com/t/assistant-api-always-return-empty-annotations/489285/48\n",
"\n",
"Here's the migration guide for easier navigating between V1 and V2 of Assistants API: https://platform.openai.com/docs/assistants/migration/changing-beta-versions\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
"#!pip install trulens-eval openai==1.14.3 # pinned openai version to avoid breaking changes "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create the assistant\n",
"\n",
"Let's create a new assistant that answers questions about the famous *Paul Graham Essay*.\n",
"\n",
"The easiest way to get it is to download it via this link and save it in a folder called data. You can do so with the following command"
"## Set keys"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -P data/"
"import os\n",
"os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"metadata": {},
"outputs": [],
"source": [
"from trulens_eval import Tru\n",
"## Create the assistant\n",
"\n",
"tru = Tru()\n",
"Let's create a new assistant that answers questions about the famous *Paul Graham Essay*.\n",
"\n",
"tru.reset_database()"
"The easiest way to get it is to download it via this link and save it in a folder called data. You can do so with the following command"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-04-25 18:07:33-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 75042 (73K) [text/plain]\n",
"Saving to: ‘data/paul_graham_essay.txt.2’\n",
"\n",
"paul_graham_essay.t 100%[===================>] 73.28K --.-KB/s in 0.007s \n",
"\n",
"2024-04-25 18:07:33 (9.58 MB/s) - ‘data/paul_graham_essay.txt.2’ saved [75042/75042]\n",
"\n"
]
}
],
"source": [
"from openai import OpenAI"
"!wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -P data/"
]
},
{
Expand All @@ -79,9 +96,18 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
"🛑 Secret keys may be written to the database. See the `database_redact_keys` option of Tru` to prevent this.\n"
]
}
],
"source": [
"from trulens_eval import Tru\n",
"from trulens_eval.tru_custom_app import instrument\n",
Expand All @@ -93,15 +119,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create a thread"
"## Create a thread (V1 Assistants)"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from openai import OpenAI\n",
"\n",
"class RAG_with_OpenAI_Assistant:\n",
" def __init__(self):\n",
" client = OpenAI()\n",
Expand Down Expand Up @@ -174,9 +202,30 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"✅ In Groundedness, input source will be set to __record__.app.retrieve_and_generate.rets[1] .\n",
"✅ In Groundedness, input statement will be set to __record__.app.retrieve_and_generate.rets[0] .\n",
"✅ In Answer Relevance, input prompt will be set to __record__.app.retrieve_and_generate.args.query .\n",
"✅ In Answer Relevance, input response will be set to __record__.app.retrieve_and_generate.rets[0] .\n",
"✅ In Context Relevance, input question will be set to __record__.app.retrieve_and_generate.args.query .\n",
"✅ In Context Relevance, input context will be set to __record__.app.retrieve_and_generate.rets[1] .\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package punkt to /home/daniel/nltk_data...\n",
"[nltk_data] Package punkt is already up-to-date!\n"
]
}
],
"source": [
"from trulens_eval import Feedback, Select\n",
"from trulens_eval.feedback import Groundedness\n",
Expand Down Expand Up @@ -214,7 +263,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -226,7 +275,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -236,9 +285,73 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 10,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Groundedness</th>\n",
" <th>Answer Relevance</th>\n",
" <th>Context Relevance</th>\n",
" <th>latency</th>\n",
" <th>total_cost</th>\n",
" </tr>\n",
" <tr>\n",
" <th>app_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>OpenAI Assistant RAG</th>\n",
" <td>0.307692</td>\n",
" <td>1.0</td>\n",
" <td>0.4</td>\n",
" <td>38.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Groundedness Answer Relevance Context Relevance \\\n",
"app_id \n",
"OpenAI Assistant RAG 0.307692 1.0 0.4 \n",
"\n",
" latency total_cost \n",
"app_id \n",
"OpenAI Assistant RAG 38.0 0.0 "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from trulens_eval import Tru\n",
"\n",
Expand All @@ -251,7 +364,7 @@
"metadata": {},
"outputs": [],
"source": [
"tru.run_dashboard()"
"tru.run_dashboard() # alternatively, you can also run `trulens-eval` from the terminal in the same folder containing the notebook"
]
},
{
Expand Down

0 comments on commit 46d362e

Please sign in to comment.