Skip to content

docs: Add link to eval blog #1180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

mikemckiernan
Copy link
Member

Description

Aditi's blog supplements the eval information we have in the docs for input and output rails.

Related Issue(s)

Checklist

  • I've read the CONTRIBUTING guidelines.
  • I've updated the documentation if applicable.
  • I've added tests if applicable.
  • @mentions of the person or team responsible for reviewing proposed changes.

Signed-off-by: Mike McKiernan <[email protected]>
@mikemckiernan mikemckiernan added this to the v0.14.0 milestone May 8, 2025
@mikemckiernan mikemckiernan self-assigned this May 8, 2025
@mikemckiernan mikemckiernan added the documentation Improvements or additions to documentation label May 8, 2025
Copy link

github-actions bot commented May 8, 2025

Documentation preview

https://nvidia.github.io/NeMo-Guardrails/review/pr-1180

@mikemckiernan mikemckiernan requested a review from trebedea May 8, 2025 13:15
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.24%. Comparing base (b65cf0e) to head (3dfbbc4).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1180   +/-   ##
========================================
  Coverage    68.24%   68.24%           
========================================
  Files          161      161           
  Lines        15938    15938           
========================================
  Hits         10877    10877           
  Misses        5061     5061           
Flag Coverage Δ
python 68.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trebedea
Copy link
Collaborator

trebedea commented May 8, 2025

Hi @mikemckiernan ,

Actually the evaluation documentation probably needs a bit of changes. Right now we have two different type of evaluation tools and corresponding documentation :

  1. An eval tool and docs mainly intended for researchers and it shows how to evaluate individual rails (e.g. dialogue, content moderation, fact-checking). This is the documentation you are changing in the current commit.
  2. An end-to-end evaluation for guardrail configs that support different rails, this is also called policy-based evaluation. The docs for this eval tools are here: docs/user-guides/eval/methodology.md

The mention to the blog post should be done in the docs for #2, no?
Any restructuring ideas for these two eval tools and documentation would also be useful.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants