Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Obs AI Assistant] fix knowledge base tests failing in MKI #205779

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[Obs AI Assistant] fix knowledge base tests failing in MKI #205779

wants to merge 1 commit into from
+35 −28

Conversation

neptunian
Copy link
Contributor

Resolves #205677
Resolves #205581

Brings the serverless tests in line with changes made in the stateful tests. Tested they were passing in MKI.

@neptunian neptunian added backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:Obs AI Assistant Observability AI Assistant labels Jan 7, 2025
@neptunian neptunian requested a review from a team as a code owner January 7, 2025 15:58
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

@neptunian neptunian added the release_note:skip Skip the PR/issue when compiling release notes label Jan 7, 2025
viduni94
viduni94 reviewed Jan 7, 2025
View reviewed changes
...tegration/test_suites/observability/ai_assistant/tests/knowledge_base/knowledge_base.spec.ts
// Fails on MKI: https://github.com/elastic/kibana/issues/205581
this.tags(['failsOnMKI']);

describe('Knowledge base', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @neptunian
The below tests are failing for me in MKI

1)    Observability AI Assistant API tests
   │       knowledge_base/knowledge_base.spec.ts
   │         Knowledge base
   │           when managing a single entry
   │             returns 200 on create:
   │
   │      Error: expected 200 "OK", got 500 "Internal Server Error"
   │       at Context.apply (knowledge_base.spec.ts:58:12)
   │       at Object.apply (wrap_function.js:74:30)
   │   ----
   │       at Test._assertStatus (node_modules/supertest/lib/test.js:252:14)
   │       at /Users/viduni/Workspace/Elastic/kibana/node_modules/supertest/lib/test.js:308:13
   │       at Test._assertFunction (node_modules/supertest/lib/test.js:285:13)
   │       at Test.assert (node_modules/supertest/lib/test.js:164:23)
   │       at localAssert (node_modules/supertest/lib/test.js:120:14)
   │       at fn (node_modules/supertest/lib/test.js:125:7)
   │       at Test.callback (node_modules/supertest/node_modules/superagent/src/node/index.js:899:3)
   │       at fn (node_modules/supertest/node_modules/superagent/src/node/index.js:1165:18)
   │       at IncomingMessage.<anonymous> (node_modules/supertest/node_modules/superagent/src/node/parsers/json.js:19:7)
   │       at IncomingMessage.emit (node:events:531:35)
   │       at endReadableNT (node:internal/streams/readable:1696:12)
   │       at processTicksAndRejections (node:internal/process/task_queues:82:21)
   │ 
   │ 
   │2)    Observability AI Assistant API tests
   │       knowledge_base/knowledge_base.spec.ts
   │         Knowledge base
   │           when managing a single entry
   │             returns 200 on get entries and entry exists:
   │
   │      TypeError: Cannot read properties of undefined (reading 'id')
   │       at Context.<anonymous> (knowledge_base.spec.ts:89:22)
   │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
   │       at Object.apply (wrap_function.js:74:16)
   │ 
   │ 
   │3)    Observability AI Assistant API tests
   │       knowledge_base/knowledge_base.spec.ts
   │         Knowledge base
   │           when managing a single entry
   │             returns 200 on delete:
   │
   │      Error: expected 200 "OK", got 500 "Internal Server Error"
   │       at Context.apply (knowledge_base.spec.ts:103:12)
   │       at Object.apply (wrap_function.js:74:30)
   │   ----
   │       at Test._assertStatus (node_modules/supertest/lib/test.js:252:14)
   │       at /Users/viduni/Workspace/Elastic/kibana/node_modules/supertest/lib/test.js:308:13
   │       at Test._assertFunction (node_modules/supertest/lib/test.js:285:13)
   │       at Test.assert (node_modules/supertest/lib/test.js:164:23)
   │       at localAssert (node_modules/supertest/lib/test.js:120:14)
   │       at fn (node_modules/supertest/lib/test.js:125:7)
   │       at Test.callback (node_modules/supertest/node_modules/superagent/src/node/index.js:899:3)
   │       at fn (node_modules/supertest/node_modules/superagent/src/node/index.js:1165:18)
   │       at IncomingMessage.<anonymous> (node_modules/supertest/node_modules/superagent/src/node/parsers/json.js:19:7)
   │       at IncomingMessage.emit (node:events:531:35)
   │       at endReadableNT (node:internal/streams/readable:1696:12)
   │       at processTicksAndRejections (node:internal/process/task_queues:82:21)
   │ 
   │ 
   │4)    Observability AI Assistant API tests
   │       knowledge_base/knowledge_base.spec.ts
   │         Knowledge base
   │           when managing multiple entries
   │             "before each" hook for "returns 200 on create":
   │
   │      Error: expected 200 "OK", got 500 "Internal Server Error"
   │       at Context.<anonymous> (knowledge_base.spec.ts:181:12)
   │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
   │       at Object.apply (wrap_function.js:74:16)
   │   ----
   │       at Test._assertStatus (node_modules/supertest/lib/test.js:252:14)
   │       at /Users/viduni/Workspace/Elastic/kibana/node_modules/supertest/lib/test.js:308:13
   │       at Test._assertFunction (node_modules/supertest/lib/test.js:285:13)
   │       at Test.assert (node_modules/supertest/lib/test.js:164:23)
   │       at localAssert (node_modules/supertest/lib/test.js:120:14)
   │       at fn (node_modules/supertest/lib/test.js:125:7)
   │       at Test.callback (node_modules/supertest/node_modules/superagent/src/node/index.js:899:3)
   │       at fn (node_modules/supertest/node_modules/superagent/src/node/index.js:1165:18)
   │       at IncomingMessage.<anonymous> (node_modules/supertest/node_modules/superagent/src/node/parsers/json.js:19:7)
   │       at IncomingMessage.emit (node:events:531:35)
   │       at endReadableNT (node:internal/streams/readable:1696:12)
   │       at processTicksAndRejections (node:internal/process/task_queues:82:21)
   │ 
   │ 
   │5)    Observability AI Assistant API tests
   │       knowledge_base/knowledge_base_status.spec.ts
   │         /internal/observability_ai_assistant/kb/status
   │           returns correct status after knowledge base is setup:
   │
   │      Error: expected false to equal true
   │       at Assertion.assert (expect.js:100:11)
   │       at Assertion.apply (expect.js:227:8)
   │       at Assertion.be (expect.js:69:22)
   │       at Context.<anonymous> (knowledge_base_status.spec.ts:49:33)
   │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
   │       at Object.apply (wrap_function.js:74:16)

Am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Thanks for testing. I initially was able to reproduce that, but after updating the other knowledge base tests in this PR, I no longer got them. Did you run all the tests together?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I used node --no-warnings scripts/functional_test_runner --exclude-tag skipMKI --config x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts

Here's the complete log:
MKI-test-run.log

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just ran them again and am now getting the errors... Guess this is inconsistent on my end for some reason.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did another run and knowledge_base_status.spec.ts passed this time. Only tests in knowledge_base.spec.ts failed.

@neptunian neptunian marked this pull request as draft January 7, 2025 17:05
@elasticmachine
Copy link
Contributor

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #69 / Observability AI Assistant API tests knowledge_base/knowledge_base_user_instructions.spec.ts Knowledge base user instructions when a user instruction exists and a conversation is created does not add the instruction conversation for other users

Metrics [docs]

✅ unchanged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@viduni94 viduni94 viduni94 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant
Projects
None yet
Milestone
No milestone
3 participants
@neptunian @elasticmachine @viduni94