trulens-eval-0.24.0

What's Changed

• cost tracking tests and litellm cost tracking by @piotrm0 in #910
• check packages on init by @piotrm0 in #917
• Increase provider test coverage to Huggingface feedback provider by @venkatkakoju in #919
• upgrade Llama-Index integration to 0.10 by @joshreini1 in #891
• Update issue templates by @joshreini1 in #923
• async handling adjustments by @piotrm0 in #918

New Contributors

• @venkatkakoju made their first contribution in #919

Full Changelog: trulens-eval-0.23.0...trulens-eval-0.24.0

trulens-eval-0.23.0

What's Changed

• Improvements to Comprehensiveness Feedback by @joshreini1 in #901
• Deeper Instrumentation for Hybrid Retrievers by @joshreini1 in #873

Examples

• Evaluating Hybrid and Ensemble Retrievers by @joshreini1 in #873

Bug Fixes

• various documentation fixes by @piotrm0 in #907
• fix forward refs with model_rebuild by @piotrm0 in #905
• cleanup by @piotrm0 in #880
• fix for in-memory sqlite params by @piotrm0 in #904
• Fix use case colab links by @joshreini1 in #900
• few site-related fixes to recently merged pr by @piotrm0 in #903
• Capture multi-line criteria and reasons for COT feedbacks by @joshreini1 in #901

Full Changelog: trulens-eval-0.22.2...trulens-eval-0.23.0

trulens-eval-0.22.2

What's Changed

• Metadata error type validation by @aaronvarghese in #888
• Update prompting to improve agreement_measure reliability by @daniel-huang-1230 in #884
• Temporary pin to llama-index version by @joshreini1 in #893
• Better exceptions optional imports by @piotrm0 in #892
• Fix bug to enable bedrock provider models by @joshreini1 in #887

New Contributors

• @aaronvarghese made their first contribution in #888

Full Changelog: trulens-eval-0.22.1...trulens-eval-0.22.2

trulens-eval-0.22.1

What's Changed

Changelog

• Improved scalability by adding in a non-threaded pacer by @piotrm0 in #874
• Fixed warning due to streamlit params -> query params by @joshreini1 in #860
• Improved deferred evaluation and status checks by @piotrm0 in #807 and #879
• Allow different schemas for Bedrock provider calls by @joshreini1 in #878

Examples

• Randomly run evals based on record_id hash by @joshreini1 in #850

Bugfixes

• Fix bug in cost tracking system by @piotrm0 in #866
• make feedback_results backwards compatible by @piotrm0 in #881
• fix st.query_params by @joshreini1 in #883

Full Changelog: trulens-eval-0.22.0...trulens-eval-0.22.1

trulens-eval-0.22.0

What's Changed

• Releases/rc trulens eval 0.21.0 by @joshreini1 in #830
• factor out common error message from app types by @piotrm0 in #832
• Add user-facing methods to use with custom functions by @joshreini1 in #847
• add testing with older python versions by @piotrm0 in #841
• Add calibration tests for OpenAI by @joshreini1 in #838

Examples

• Update example of usage with AzureOpenAI by @joshreini1 in #847

Bug Fixes

• fix ellipsis issue by @piotrm0 in #840 and #843
• Fix typing issues for stock feedback functions by @joshreini1 in #838

Full Changelog: trulens-eval-0.21.0...trulens-eval-0.22.0

TruLens Eval v0.21.0

What's changed

• Deduplicated sync/async methods by @piotrm0 in #793
• Refactored groundedness methods by @joshreini1 in #801
• Error on deprecated passthrough methods by @piotrm0 in #803
• Virtual models for logging and evaluating existing data by @piotrm0 in #806
• Rename summarization quality to comprehensiveness by @joshreini1 in #816
• Delete long deprecated TruApp and TruDB by @piotrm0 in #817
• Enable async unit tests by @piotrm0 in #831
• Add generation of test cases by @joshreini1 in #705

Examples

• Expand evaluation docs by @joshreini1 in #823 including:
  • Running Feedback Functions
  • Feedback Function Selectors
  • Feedback Function Providers
  • Feedback Implementations
  • Generating Test Cases
  • Feedback Evaluations

Bug Fixes

• Add metadata display and application tag display in UI by @joshreini1 in #797
• Fixed issue with float precision by @joshreini1 in #798
• Fix typo in openai moderation - sexual minors by @joshreini1 in #815
• Include reasoning in summarization eval by @joshreini1 in #815
• Make OpenAI optional by @joshreini1 in #827

New contributors

• @vivekgangasani made their first contribution to update AWS jumpstart examples in #795

Notes

• When feedback mode is set to WITH_APP_THREAD, feedback may be computed more eagerly than expected.

TruLens Eval v0.20.3

What's changed

• Improved system prompt for ground truth evaluations by @daniel-huang-1230 in #782
• Added pydantic validation to feedback functions by @daniel-huang-1230 in #782
• Changed bedrock default model from titan-text-lite to titan-text-express by @joshreini1 in #788
• Documentation on optional packages to use with TruLens by @piotrm0 in #787

Examples

• Update langchain retrieval agents example to show validation of tool selection and tool input, work with openai > 1.0 by @piotrm0 @joshreini1 in #778
• Update AWS examples with changes to Bedrock API, other bug fixes by @joshreini1 in #788

Bug Fixes

• Removed optional imports from utility import (defined by init) by @piotrm0 in #786
• Fix missing f-strings in warning when app_id not yet present in the logging db by @andrewisplinghoff in #790

New contributors

• @andrewisplinghoff made their first contribution in #790

TruLens Eval v0.20.2

What's changed

• Changed bedrock default model to "amazon.titan-text-lite-v1" by @rajib76 in #774
• Documented outstanding tech debt by @piotrm0 in #771

Bug Fixes

• Fix paths to components if an app gets garbage collected by @piotrm0 in #771
• Wrapped openai imports and jupyter notebook related imports in optional contexts by @piotrm0 in #770
• Fix to issubclass by @piotrm0 in #770
• Address issue with select_context where the retriever is hidden by an excluded attribute by @piotrm0 in #768

New contributors

• @rajib76 made their first contribution in #774

TruLens Eval v0.20.1

What's changed

• Added instructions for installing from github by @piotrm0 in #740
• Updated instrumentation docs by @joshreini1 in #737
• Update structured extraction and chain of thought prompting for better results across providers by @joshreini1 in #732

Examples

• Improve clarity of human feedback notebook by @piotrm0 in #746

Bug Fixes

• Fixed the Langchain provider by @piotrm0 in #759
• Added missing Langchain provider docs by @joshreini1 in #760
• Converted structures to strings in feedback results table by @piotrm0 in #758
• Improve instrumentation of langchain apps by @piotrm0 in #757 and #755
• Fix subscripted generics typechecking for python < 3.10 by @coreyhu in #754
• Make Llama-Index completely optional by @piotrm0 in #745
• Bugfixes for WithClassInfo by @piotrm0 in #741
• Fixes for AzureOpenAI serialization by @piotrm0 in #735
• Pass prompt instead of messages to Bedrock by @joshreini1 in #732

TruLens Eval 0.20.0

What's Changed

• Added aliases for selectors for main method args and main method return by @piotrm0 in #696
• Exposed AzureOpenAI provider by @epinzur in #698
• Allow debug timeout to be adjusted by @piotrm0 in #713
• Import llama only if needed by @joshreini1 in #714
• Add shortcut to select_context() by @joshreini1 in #706

Bugfixes

• fix dashboard starts for colab by @piotrm0 in #721, #720
• fix(timeline): Clicking first call (second row) in timeline does not display that call's information by @walnutdust in #719
• Fix colab link by @joshreini1 in #699
• Fixed correctness prompt by @shayaks in #725
• Pydantic2 deprecation fix to model config by @piotrm0 in #724

Examples

• Gemini Example by @joshreini1 in #697
• ollama quickstart by @joshreini1 in #703

New Contributors

• @epinzur made their first contribution in #698

Full Changelog: trulens-eval-0.19.2...trulens-eval-0.20.0