[feat] Teach the LLM to Speak in Haiku

[rishimo]

:)

Changing prompt to rhyme,
LLM now speaks in haiku,
Poetry through code.

[rishimo]

/evaluate

[github-actions]

Starting evaluation! Check the Actions tab for progress, or wait for a comment with the results.

[github-actions]

Evaluation results

metric	stat	baseline	pr33
gpt_groundedness	mean_rating	5.0	3.2
↑	pass_rate	1.0	0.55
gpt_relevance	mean_rating	4.95	2.75
↑	pass_rate	1.0	0.2
f1_score	mean	0.43	0.12
answer_length	mean	609.1	81.45
latency	mean	2.67	1.27
citations_matched	mean	0.63	0.18

Check the workflow run for more details.

[pamelafox]

Beautiful, now you just need a is-haiku-evaluator!

For your reference, here are my slides:
https://speakerdeck.com/pamelafox/github-universe-evaluating-rag-apps-in-github-actions

And the main repo this was based on:
https://github.com/pamelafox/rag-postgres-openai-python/
with evaluation guide here:
https://github.com/Azure-Samples/rag-postgres-openai-python/blob/main/docs/evaluation.md

To evaluate on Azure,
Azure AI Eval SDK docs are here:
https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/evaluate-sdk
And if you're interested in the Azure AI CI/CD private preview, sign up here:
https://aka.ms/genAI-CI-CD-private-preview