Conduct Benchmark Test of Candidate LLMs

[grayJiaaoLi]

User story

1. As a Software Developer
2. I want to make quantitative comparisons of candidate LLMs
3. So that we know how well each model performs relative to the others

Acceptance criteria

• Incorporating model training and obtained data
• choose relevant benchmarks for CNCF-focused tasks, for example:
  • Natural Language Questions (ARC or HellaSwag)
  • Explanatory Tasks (MMLU)
• Log the result of each LLM
• The benchmark can refer to Eleuther AI Language Model Evaluation Harness

Definition of done (DoD)

• At least three candidate LLMs are tested
• The results of tests are logged and organised
• The results provide quantitative comparisons between different models

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release