[Obs AI Assistant] Improve LLM evaluation framework #204574

viduni94 · 2024-12-17T13:20:30Z

Closes #203122

Summary

Problem

The Obs AI Assistant LLM evaluation framework cannot successfully run in the current state in the main branch and has missing scenarios.

Problems identified:

Unable to run the evaluation with a local Elasticsearch instance
Alerts and APM results are skipped entirely when reporting the final result on the terminal (due to consistent failures in the tests)
State contaminations between runs makes the script throw errors when run multiple times.
Authentication issues when calling /internal APIs

Solution

As a part of spacetime, worked on fixing the current issues in the LLM evaluation framework and working on improving and enhancing the framework.

Fixes

Problem	RC (Root Cause)	Fixed?
Running with a local Elasticsearch instance	Service URLs were not picking up the correct auth because of the format specified in `kibana.dev.yml`	✅
Alerts and APM results skipped in final result	Most (if not all) tests are failing in the alerts and APM suites, hence no final results are reported.	✅ (all test scenarios fixed)
State contaminations between runs	Some `after` hooks were not running successfully because of an error in the `callKibana` method	✅
Authentication issues when calling `/internal` APIs	The required headers are not present in the request	✅

Enhancements / Improvements

What was added	How does it enhance the framework
Added new KB retrieval test to the KB scenario	More scenarios covered
Added new scenario for the `retrieve_elastic_doc` function	Cover missing newly added functions
Enhance how scope is used for each scenario and apply correct scope	The scope determines the wording of the system message. Certain scenarios need to be scoped to observability (e.g.: `alerts`) to produce the best result. At present all scenarios use the scope `all` which is not ideal and doesn't align with the actual functionality of the AI Assistant
Avoid throwing unnecessary errors on the console (This was fixed by adding guard rails, e.g.: not creating a dataview if it exists)	Makes it easier to navigate through the results printed on the terminal
Improved readme	Easier to configure and use the framework while identifying all possible options
Improved logging	Easier to navigate through the terminal output

Checklist

The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

elasticmachine · 2024-12-23T13:50:15Z

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

…rch credentials are provided

…tests

…ging

github-actions · 2024-12-23T13:50:35Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…fix'

…-fix'

dgieselaar

Looks great @viduni94! just a few nits

dgieselaar · 2024-12-24T09:36:08Z

.../solutions/observability/plugins/observability_ai_assistant_app/scripts/evaluation/README.md


-#### Connector
+- Run Elasticsearch locally: `yarn es snapshot --license trial`


--license trial is unnecessary no?

Without --license trial, I can't call the Obs AI Assistant endpoints.

.../solutions/observability/plugins/observability_ai_assistant_app/scripts/evaluation/README.md

.../observability/plugins/observability_ai_assistant_app/scripts/evaluation/get_service_urls.ts

dgieselaar · 2024-12-24T09:42:56Z

...ons/observability/plugins/observability_ai_assistant_app/scripts/evaluation/kibana_client.ts

@@ -124,13 +124,13 @@ export class KibanaClient {
    return this.axios<T>({
      method,
      url,
-      data: data || {},
+      ...(method.toLowerCase() !== 'delete' ? { data: data || {} } : {}),


why is this needed?

Without this condition, deleting ruleIds fails here - https://github.com/elastic/kibana/pull/204574/files#diff-23cc9139c91a064a3ca574552ad823023c579cc2c68ff7f277c392102a0d526aL139

Because the DELETE method doesn't allow an undefined or empty body.

...ons/observability/plugins/observability_ai_assistant_app/scripts/evaluation/kibana_client.ts

...gins/observability_ai_assistant_app/scripts/evaluation/scenarios/elasticsearch/index.spec.ts

…-fix'

viduni94 · 2024-12-24T15:50:28Z

Results after the updates:

-------------------------------------------
Model azure-gpt4 scored 112.6 out of 123
-------------------------------------------
-------------------------------------------
Model azure-gpt4 Scores per Category
-------------------------
Category: Alert function - Scored 10 out of 10
-------------------------
Category: APM - Scored 11 out of 17
-------------------------
Category: Retrieve documentation function - Scored 14 out of 14
-------------------------
Category: Elasticsearch functions - Scored 19 out of 19
-------------------------
Category: ES|QL query generation - Scored 43.6 out of 48
-------------------------
Category: Knowledge base - Scored 15 out of 15
-------------------------------------------

elasticmachine · 2024-12-24T17:00:37Z

💚 Build Succeeded

Buildkite Build
Commit: 40c6445
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-204574-40c64459b0ba

Metrics [docs]

✅ unchanged

History

💚 Build #262943 succeeded 541fa80
💚 Build #262791 succeeded 112383c

cc @viduni94

viduni94 added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting v9.0.0 Team:Obs AI Assistant Observability AI Assistant labels Dec 17, 2024

viduni94 self-assigned this Dec 17, 2024

viduni94 force-pushed the improve-llm-evaluation-framework branch 6 times, most recently from 57b281e to e82e4e6 Compare December 20, 2024 15:52

viduni94 marked this pull request as ready for review December 23, 2024 13:50

viduni94 requested a review from a team as a code owner December 23, 2024 13:50

viduni94 added 16 commits December 23, 2024 08:50

[Obs AI Assistant] Update evaluation framework readme

c2b5270

[Obs AI Assistant] Fix auth for the kibana url when custom elasticsea…

7641ecb

…rch credentials are provided

[Obs AI Assistant] Create dataview if it doesn't exist

315ba30

[Obs AI Assistant] Logs for service urls

18f2b21

[Obs AI Assistant] Temp skip for scenarios except alerts

7519314

[Obs AI Assistant] Add header to enable accessing internal APIs

124028b

[Obs AI Assistant] Fix apm afterAll hook

e23ebf1

[Obs AI Assistant] Update error handling

51589f9

[Obs AI Assistant] Update calls to internal urls

545ab41

[Obs AI Assistant] Improve data view creation

b5dbce8

[Obs AI Assistant] Change internal origin to Kibana

84972ea

[Obs AI Assistant] Improve scopes handling in the chat client

3b7d770

[Obs AI Assistant] Update elasticsearch and es|ql scope before/after …

91d80f5

…tests

[Obs AI Assistant] Fix eslint issues

221abdf

[Obs AI Assistant] Fix eslint issues

34da5d5

[Obs AI Assistant] Add new scenario/test for KB retrieval

1b57832

viduni94 added 2 commits December 23, 2024 08:50

[Obs AI Assistant] Add new scenario for documentation and improve log…

6c0e59b

…ging

[Obs AI Assistant] Improve readme

5d7fe68

viduni94 force-pushed the improve-llm-evaluation-framework branch from b60786e to 5d7fe68 Compare December 23, 2024 13:50

botelastic bot added the ci:project-deploy-observability Create an Observability project label Dec 23, 2024

kibanamachine added 2 commits December 23, 2024 14:00

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

c2d1d65

…fix'

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

112383c

…-fix'

dgieselaar reviewed Dec 24, 2024

View reviewed changes

viduni94 and others added 4 commits December 24, 2024 08:39

[Obs AI Assistant] Address PR comments

8da301b

[Obs AI Assistant] Revert auth change as it's not necessary

541fa80

[Obs AI Assistant] Make scope a part of the complete function

bc3480e

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

8c9911e

…-fix'

viduni94 requested a review from dgieselaar December 24, 2024 15:49

[Obs AI Assistant] remove comment

40c6445

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Obs AI Assistant] Improve LLM evaluation framework #204574

[Obs AI Assistant] Improve LLM evaluation framework #204574

viduni94 commented Dec 17, 2024 •

edited

Loading

elasticmachine commented Dec 23, 2024

github-actions bot commented Dec 23, 2024

dgieselaar left a comment

dgieselaar Dec 24, 2024

viduni94 Dec 24, 2024

dgieselaar Dec 24, 2024

viduni94 Dec 24, 2024 •

edited

Loading

viduni94 commented Dec 24, 2024

elasticmachine commented Dec 24, 2024 •

edited

Loading


		#### Connector
		- Run Elasticsearch locally: `yarn es snapshot --license trial`

[Obs AI Assistant] Improve LLM evaluation framework #204574

Are you sure you want to change the base?

[Obs AI Assistant] Improve LLM evaluation framework #204574

Conversation

viduni94 commented Dec 17, 2024 • edited Loading

Summary

Problem

Solution

Fixes

Enhancements / Improvements

Checklist

elasticmachine commented Dec 23, 2024

github-actions bot commented Dec 23, 2024

🤖 GitHub comments

dgieselaar left a comment

Choose a reason for hiding this comment

dgieselaar Dec 24, 2024

Choose a reason for hiding this comment

viduni94 Dec 24, 2024

Choose a reason for hiding this comment

dgieselaar Dec 24, 2024

Choose a reason for hiding this comment

viduni94 Dec 24, 2024 • edited Loading

Choose a reason for hiding this comment

viduni94 commented Dec 24, 2024

elasticmachine commented Dec 24, 2024 • edited Loading

💚 Build Succeeded

Metrics [docs]

History

viduni94 commented Dec 17, 2024 •

edited

Loading

viduni94 Dec 24, 2024 •

edited

Loading

elasticmachine commented Dec 24, 2024 •

edited

Loading