[WIP] Proposal for new Evaluation repo

Signed-off-by: Nathan Weinberg <[email protected]>

.spellcheck-en-custom.txt

@@ -34,6 +34,7 @@ Dropdown
 env
 EP
 Eval
+eval
 Excalidraw
 exfiltrate
 exfiltrating
@@ -63,8 +64,8 @@ LLM
 llms
 LLVM
 lora
-md
 Markdownlint
+md
 Mergify
 Merlinite
 mimimum
@@ -74,8 +75,8 @@ MLX
 mlx
 NVidia
 Nvidia
-ollama
 Ollama
+ollama
 orchestrator
 ots
 Pareja
@@ -104,12 +105,12 @@ RX
 safetensors
 Salawu
 SDG
-Sigstore
 sdg
 sexualized
 SHA
 Shivchander
 Signoff
+Sigstore
 Srivastava
 subdirectory
 Sudalairaj

docs/backend/eval-repo.md

@@ -0,0 +1,35 @@
+# New Repository Proposal: eval
+
+## Summary
+
+This document proposes a new repository under the `instructlab` GitHub organization:
+
+- `instructlab/eval`
+
+## Background
+
+The `instructlab/instructlab` repository currently includes no implementation of
+Evaluation as described by the [LAB paper](https://arxiv.org/abs/2403.01081).
+
+We desire to build out an implementation of the described evaluation. We propose a
+new repository to house this code that publishes a new Python library called
+`instructlab-eval`. The reasoning for a new repository and library includes:
+
+- We expect multiple consumers of this code. The `ilab` CLI is one, but we also envision building a REST API around it to help support scaling out this functionality on a cluster.
+- We expect there is broader community interest in an open-source library and service for evaluation. We envision this library could support other evaluation techniques over time.
+
+## Maintainers
+
+The initial team of maintainers for this repository will be a copy of the
+`Backend Maintainers` GitHub team.
+
+## Alternatives Considered
+
+### Add to `instructlab/instructlab`
+
+We could add this code to the existing `instructlab/instructlab` repository.
+
+The primary argument against this approach is that we expect the scope of an
+`instructlab-eval` library to expand beyond the scope of what would be run by the
+`ilab` CLI. We instead envision a different community of contributors organizing
+around Evaluation specifically.