span types -> feature/traces branch (#1103)

* shuffling * update standard.md to mention python versions tested * de-circularizing * finishing up decircularization * fix imports in tests * more import fixes * starting spans work * more import fixes * updating docs and docstrings with new locations * add temp to bedrock and ignored warning * trying to remove circular import * more debugging * more fixes * more of the same * last few * missed on * nits * typo * fix feedbackmode imports * 2 more fixes * starting spans work * work * working on spans * opentelemetry work * working on spans * span work * organize imports * starting span categorization logic * nits * typo in module doc * spans for custom apps * clear outputs * fix dictnamespace typing * fill in parent relationships, dates, and inputs/outputs .
truera · Apr 27, 2024 · b99afb5 · b99afb5
1 parent 46d362e
commit b99afb5
Show file tree

Hide file tree

Showing 22 changed files with 2,129 additions and 78 deletions.
diff --git a/docs/trulens_eval/api/trace/index.md b/docs/trulens_eval/api/trace/index.md
@@ -0,0 +1,3 @@
+# Trace
+
+::: trulens_eval.trace
diff --git a/docs/trulens_eval/api/trace/span.md b/docs/trulens_eval/api/trace/span.md
@@ -0,0 +1,3 @@
+# Span
+
+::: trulens_eval.trace.span
diff --git a/docs/trulens_eval/api/trace/tracer.md b/docs/trulens_eval/api/trace/tracer.md
@@ -0,0 +1,4 @@
+# Tracer
+
+::: trulens_eval.trace.tracer
+
diff --git a/docs/trulens_eval/contributing/design.md b/docs/trulens_eval/contributing/design.md
@@ -62,11 +62,11 @@ In addition to collecting app parameters, we also collect:
 
 - (subset of components) App class information:
 
-    - This allows us to deserialize some objects. Pydantic models can be
-      deserialized once we know their class and fields, for example.
-    - This information is also used to determine component types without having
-      to deserialize them first. 
-    - See [Class][trulens_eval.utils.pyschema.Class] for details.
+  - This allows us to deserialize some objects. Pydantic models can be
+    deserialized once we know their class and fields, for example.
+  - This information is also used to determine component types without having
+    to deserialize them first. 
+  - See [Class][trulens_eval.utils.pyschema.Class] for details.
 
 ### Functions/Methods
 
@@ -158,7 +158,11 @@ our reliance on info stored on the stack. Therefore we have a limitation:
   [ThreadPoolExecutor][trulens_eval.utils.threading.ThreadPoolExecutor] also
   defined in `utils/threading.py` in order for instrumented methods called in a
   thread to be tracked. As we rely on call stack for call instrumentation we
-  need to preserve the stack before a thread start which python does not do. 
+  need to preserve the stack before a thread start which python does not do.
+
+OpenTelemetry has similar problems and comes with similar solutions. See for
+  example [OpenTelemetry thread
+  instrumentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/threading/threading.html).
 
 #### Async
 

diff --git a/docs/trulens_eval/contributing/techdebt.md b/docs/trulens_eval/contributing/techdebt.md
@@ -32,6 +32,9 @@ See `instruments.py` docstring for discussion why these are done.
   object that implements `__call__` has been instrumented. Hacks to avoid
   warnings about lack of instrumentation.
 
+- "HACK015" -- Add `__hash__` onto `opentelemetry.trace.span.SpanContext` by
+  changing their `__class__`.
+
 ## Thread overriding
 
 See `instruments.py` docstring for discussion why these are done.

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -67,6 +67,7 @@ plugins:
             - https://typing-extensions.readthedocs.io/en/latest/objects.inv
             - https://docs.llamaindex.ai/en/stable/objects.inv
             - https://docs.sqlalchemy.org/en/20/objects.inv
+            - https://opentelemetry-python.readthedocs.io/en/latest/objects.inv
           options:
             extensions:
               - pydantic: { schema: true }
@@ -269,6 +270,10 @@ nav:
           - trulens_eval/api/utils/json.md
           - trulens_eval/api/utils/frameworks.md
           - trulens_eval/api/utils/utils.md
+      - "Experimental: Tracing":
+        - trulens_eval/api/trace/index.md
+        - trulens_eval/api/trace/span.md
+        - trulens_eval/api/trace/tracer.md
   - 🤝 Contributing:
       - trulens_eval/contributing/index.md
       - 🧭 Design: trulens_eval/contributing/design.md

diff --git a/trulens_eval/examples/experimental/dev_notebook.ipynb b/trulens_eval/examples/experimental/dev_notebook.ipynb
@@ -70,53 +70,6 @@
     "# tru.db.migrate_database()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# tru.db.migrate_database()\n",
-    "tru.migrate_database()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for t in tru.db.orm.registry.values():\n",
-    "    print(t)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from trulens_eval.database.utils import copy_database"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tru.db"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "copy_database(\"sqlite:///default.sqlite\", \"sqlite:///default2.sqlite\", src_prefix=\"dev\", tgt_prefix=\"dev\")"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,

diff --git a/trulens_eval/examples/experimental/dummy_example.ipynb b/trulens_eval/examples/experimental/dummy_example.ipynb
@@ -50,7 +50,7 @@
     "from trulens_eval import Feedback\n",
     "from trulens_eval import Tru\n",
     "from trulens_eval.feedback.provider.hugs import Dummy\n",
-    "from trulens_eval.schema import FeedbackMode\n",
+    "from trulens_eval.schema.feedback import FeedbackMode\n",
     "from trulens_eval.tru_custom_app import TruCustomApp\n",
     "from trulens_eval.utils.threading import TP\n",
     "\n",
@@ -102,7 +102,7 @@
     "ta = TruCustomApp(\n",
     "    ca,\n",
     "    app_id=\"customapp\",\n",
-    "    feedbacks=[f_dummy1, f_dummy2, f_dummy3],\n",
+    "    # feedbacks=[f_dummy1, f_dummy2, f_dummy3],\n",
     "    feedback_mode=FeedbackMode.DEFERRED\n",
     ")"
    ]
@@ -116,7 +116,7 @@
     "# Sequential app invocation.\n",
     "\n",
     "if True:\n",
-    "    for i in tqdm(range(128), desc=\"invoking app\"):\n",
+    "    for i in tqdm(range(2), desc=\"invoking app\"):\n",
     "        with ta as recorder:\n",
     "            res = ca.respond_to_query(f\"hello {i}\")\n",
     "\n",
@@ -229,7 +229,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.18"
+   "version": "3.11.6"
   },
   "orig_nbformat": 4
  },