feat(weave): Add Llamaguard scorer 2/N #3413
Conversation
- Introduced `LlamaGuardScorer` class for evaluating the safety of model outputs using LlamaGuard models. - Implemented methods for tokenization, scoring, and postprocessing of outputs to determine safety and categorize violations. - Added unit tests for `LlamaGuardScorer`, covering postprocessing and scoring functionalities, including safe and unsafe content scenarios. - Enhanced evaluation capabilities with asynchronous testing for scoring and evaluation of datasets. - Ensured comprehensive test coverage for the new scorer's functionality.
tcapelle
changed the title
|
New dependencies detected. Learn more about Socket for GitHub ↗︎
|
|
🚨 Potential security issues detected. Learn more about Socket for GitHub ↗︎ To accept the risk, merge this PR and you will not be notified again.
Next stepsWhat is a license policy violation?This package is not allowed per your license policy. Review the package's license to ensure compliance. Find a package that does not violate your license policy or adjust your policy to allow this package's license. Take a deeper look at the dependencyTake a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev. Remove the packageIf you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency. Mark a package as acceptable riskTo ignore an alert, reply with a comment starting with
|
|
Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=ec57abc8f235d410bcf051fb8a882510a9dbacfd |
Add the Llamaguard scorer, requires #3412