Skip to content

Commit

Permalink
Update site link
Browse files Browse the repository at this point in the history
  • Loading branch information
mattpocock committed Dec 16, 2024
1 parent 2fe9bd7 commit e704092
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 74 deletions.
2 changes: 1 addition & 1 deletion packages/evalite/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"version": "0.6.2",
"type": "module",
"description": "Test your LLM-powered apps with a TypeScript-native, Vitest-based eval runner. No API key required.",
"homepage": "https://github.com/mattpocock/evalite",
"homepage": "https://evalite.dev",
"keywords": [
"ai",
"evals",
Expand Down
75 changes: 3 additions & 72 deletions packages/evalite/readme.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
![Evalite: the TypeScript-native, local-first tool for testing LLM-powered apps.](https://raw.githubusercontent.com/mattpocock/evalite/refs/heads/main/repo-card.jpg)

- [Join the Discord](https://www.mattpocock.com/ai-discord)

## What Is Evalite?

- Fully open source: **No API Key required**.
Expand All @@ -10,77 +8,10 @@
- Terminal UI for quick prototyping.
- Supports tracing and custom scorers.

## Quickstart

### 1. Install `evalite` and `autoevals`:

Install `evalite`, `vitest`, and a scoring library like `autoevals`:

```bash
pnpm add -D evalite vitest autoevals
```

### 2. Add an `eval:dev` script:

Add an `eval` script to your package.json:
## How Do I Learn More?

```json
{
"scripts": {
"eval:dev": "evalite watch"
}
}
```

### 3. Create your first eval:

Create `my-eval.eval.ts`:

```ts
// my-eval.eval.ts

import { evalite } from "evalite";
import { Levenshtein } from "autoevals";

evalite("My Eval", {
// A function that returns an array of test data
// - TODO: Replace with your test data
data: async () => {
return [{ input: "Hello", expected: "Hello World!" }];
},
// The task to perform
// - TODO: Replace with your LLM call
task: async (input) => {
return input + " World!";
},
// The scoring methods for the eval
scorers: [Levenshtein],
});
```

> [!NOTE]
>
> `.eval.ts` is the extension Evalite looks for when scanning for evals.
### 4. Run Your Eval

Run `pnpm run eval:dev`.

This runs `evalite`, which runs the evals:

- Runs the `data` function to get the test data
- Runs the `task` function on each test data
- Scores the output of the `task` function using the `scorers`
- Saves the results to a sqlite database in `node_modules/.evalite`

It then:

- Shows a UI for viewing the traces, scores, inputs and outputs at http://localhost:3006.
- If you only ran one eval, it also shows a table summarizing the eval in the terminal.

### 5. View Your Eval

Open http://localhost:3006 in your browser to view the results of the eval.
- [Read the Docs](https://www.evalite.dev/)
- [Join the Discord](https://www.mattpocock.com/ai-discord)

## Evalite Is Experimental

Expand Down
2 changes: 1 addition & 1 deletion readme.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
![Evalite: the TypeScript-native, local-first tool for testing LLM-powered apps.](https://raw.githubusercontent.com/mattpocock/evalite/refs/heads/main/repo-card.jpg)

- [View the docs](./packages/evalite/readme.md)
- [View the docs](https://www.evalite.dev/)
- [Join the Discord](https://www.mattpocock.com/ai-discord)

## Contributing
Expand Down

0 comments on commit e704092

Please sign in to comment.