From ec0937d2dbbca776423d41bd5cefa18bf9326584 Mon Sep 17 00:00:00 2001
From: Nathan Weinberg <nweinber@redhat.com>
Date: Thu, 6 Jun 2024 13:17:39 -0400
Subject: [PATCH] [WIP] Proposal for new Evaluation repo

Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
---
 .spellcheck-en-custom.txt      |  7 ++++---
 docs/backend/eval-repo.md      | 35 ++++++++++++++++++++++++++++++++++
 docs/{ => backend}/sdg-repo.md |  0
 3 files changed, 39 insertions(+), 3 deletions(-)
 create mode 100644 docs/backend/eval-repo.md
 rename docs/{ => backend}/sdg-repo.md (100%)

diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt
index 4ebd275a..7baeb004 100644
--- a/.spellcheck-en-custom.txt
+++ b/.spellcheck-en-custom.txt
@@ -34,6 +34,7 @@ Dropdown
 env
 EP
 Eval
+eval
 Excalidraw
 exfiltrate
 exfiltrating
@@ -63,8 +64,8 @@ LLM
 llms
 LLVM
 lora
-md
 Markdownlint
+md
 Mergify
 Merlinite
 mimimum
@@ -74,8 +75,8 @@ MLX
 mlx
 NVidia
 Nvidia
-ollama
 Ollama
+ollama
 orchestrator
 ots
 Pareja
@@ -104,12 +105,12 @@ RX
 safetensors
 Salawu
 SDG
-Sigstore
 sdg
 sexualized
 SHA
 Shivchander
 Signoff
+Sigstore
 Srivastava
 subdirectory
 Sudalairaj
diff --git a/docs/backend/eval-repo.md b/docs/backend/eval-repo.md
new file mode 100644
index 00000000..cbf3bcf8
--- /dev/null
+++ b/docs/backend/eval-repo.md
@@ -0,0 +1,35 @@
+# New Repository Proposal: eval
+
+## Summary
+
+This document proposes a new repository under the `instructlab` GitHub organization:
+
+- `instructlab/eval`
+
+## Background
+
+The `instructlab/instructlab` repository currently includes no implementation of
+Evaluation as described by the [LAB paper](https://arxiv.org/abs/2403.01081).
+
+We desire to build out an implementation of the described evaluation. We propose a
+new repository to house this code that publishes a new Python library called
+`instructlab-eval`. The reasoning for a new repository and library includes:
+
+- We expect multiple consumers of this code. The `ilab` CLI is one, but we also envision building a REST API around it to help support scaling out this functionality on a cluster.
+- We expect there is broader community interest in an open-source library and service for evaluation. We envision this library could support other evaluation techniques over time.
+
+## Maintainers
+
+The initial team of maintainers for this repository will be a copy of the
+`Backend Maintainers` GitHub team.
+
+## Alternatives Considered
+
+### Add to `instructlab/instructlab`
+
+We could add this code to the existing `instructlab/instructlab` repository.
+
+The primary argument against this approach is that we expect the scope of an
+`instructlab-eval` library to expand beyond the scope of what would be run by the
+`ilab` CLI. We instead envision a different community of contributors organizing
+around Evaluation specifically.
diff --git a/docs/sdg-repo.md b/docs/backend/sdg-repo.md
similarity index 100%
rename from docs/sdg-repo.md
rename to docs/backend/sdg-repo.md