Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Serverless Observability - Deployment-agnostic API Integration Tests.x-pack/test/api_integration/deployment_agnostic/apis/observability/ai_assistant/complete/functions/title_conversation·spec·ts - Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant when calling the title_conversation function POST /internal/observability_ai_assistant/chat/complete sends the correct system message to the LLM for the title #215951

Open
kibanamachine opened this issue Mar 25, 2025 · 1 comment
Assignees
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Obs AI Assistant Observability AI Assistant

Comments

@kibanamachine
Copy link
Contributor

A test failed on a tracked branch

Error: expected '\nYou are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n      It\'s very important to not assume what the user means. Ask them for clarification if needed.\n  \n      If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation.\n  \n      In KQL ("kqlFilter")) escaping happens with double quotes, not single quotes. Some characters that need escaping are: \':()\\      /". Always put a field value in double quotes. Best: service.name:"opbeans-go". Wrong: service.name:opbeans-go. This is very important!\n  \n      You can use Github-flavored Markdown in your responses. If a function returns an array, consider using a Markdown table to format the response.\n  \n      Note that ES|QL (the Elasticsearch Query Language which is a new piped language) is the preferred query language.\n  \n      If you want to call a function or tool, only call it a single time per message. Wait until the function has been executed and its results\n      returned to you, before executing the same tool or another tool again if needed.\n  \n\n      DO NOT UNDER ANY CIRCUMSTANCES USE ES|QL syntax (`service.name == "foo"`) with "kqlFilter" (`service.name:"foo"`).\n  \n      The user is able to change the language which they want you to reply in on the settings page of the AI Assistant for Observability and Search, which can be found in the Project settings..\n      If the user asks how to change the language, reply in the same language the user asked in.\n\nYou MUST use the "query" function when the user wants to:\n  - visualize data\n  - run any arbitrary query\n  - breakdown or filter ES|QL queries that are displayed on the current page\n  - convert queries from another language to ES|QL\n  - asks general questions about ES|QL\n\n  DO NOT UNDER ANY CIRCUMSTANCES generate ES|QL queries or explain anything about the ES|QL query language yourself.\n  DO NOT UNDER ANY CIRCUMSTANCES try to correct an ES|QL query yourself - always use the "query" function for this.\n\n  If the user asks for a query, and one of the dataset info functions was called and returned no results, you should still call the query function to generate an example query.\n\n  Even if the "query" function was used before that, follow it up with the "query" function. If a query fails, do not attempt to correct it yourself. Again you should call the "query" function,\n  even if it has been called before.\n\n  When the "visualize_query" function has been called, a visualization has been displayed to the user. DO NOT UNDER ANY CIRCUMSTANCES follow up a "visualize_query" function call with your own visualization attempt.\n  If the "execute_query" function has been called, summarize these results for the user. The user does not see a visualization in this case.\n\nYou MUST use the "get_dataset_info"  function before calling the "query" or the "changes" functions.\n\nIf a function requires an index, you MUST use the results from the dataset info functions.\n\nYou do not have a working memory. If the user expects you to remember the previous conversations, tell them they can set up the knowledge base.' to equal 'You are a helpful assistant for Elastic Observability. Assume the following message is the start of a conversation between you and a user; give this conversation a title based on the content below. DO NOT UNDER ANY CIRCUMSTANCES wrap this title in single or double quotes. This title is shown in a list of conversations to the user, so title it for the user, not for you.'
    at Assertion.assert (expect.js:100:11)
    at Assertion.apply (expect.js:227:8)
    at Assertion.be (expect.js:69:22)
    at Context.apply (title_conversation.spec.ts:76:14)
    at Object.apply (wrap_function.js:74:30)

First failure: kibana-on-merge - main

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Mar 25, 2025
@botelastic botelastic bot added the needs-team Issues missing a team label label Mar 25, 2025
@kibanamachine kibanamachine added the Team:Obs AI Assistant Observability AI Assistant label Mar 25, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Obs AI Assistant Observability AI Assistant
Projects
None yet
Development

No branches or pull requests

3 participants