Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

Open-Source Evaluation for LLM Application Pipelines

The LLM Evaluation Framework

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

A framework for few-shot evaluation of language models.

Python client for Kolena's machine learning testing platform

Evaluation framework for oncology foundation models (FMs)

The official evaluation suite and dynamic data release for MixEval.

Evaluate custom and HuggingFace text-to-image/zero-shot-image-classification models like CLIP, SigLIP, DFN5B, and EVA-CLIP. Metrics include Zero-shot accuracy, Linear Probe, Image retrieval, and KNN accuracy.

Python SDK for running evaluations on LLM generated responses

🐢 Open-Source Evaluation & Testing for LLMs and ML models

Evaluating LLMs' Cognitive Behavioral Reasoning for Cybersecurity

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models

Python package with Deep Learning utilities for Computer Vision

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

"Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning" by Chongyu Fan*, Jiancheng Liu*, Alfred Hero, Sijia Liu

The official implementation of ECCV'24 paper "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now". This work introduces one fast and effective attack method to evaluate the harmful-content generation ability of safety-driven unlearned diffusion models.

Red-teaming LLM applications.