Sustainable Kibana Architecture: Move modules owned by `@elastic/security-generative-ai` #202848

gsoldevila · 2024-12-04T00:05:59Z

Summary

This PR aims at relocating some of the Kibana modules (plugins and packages) into a new folder structure, according to the Sustainable Kibana Architecture initiative.

Important

We kindly ask you to:
- Manually fix the errors in the error section below (if there are any).
- Search for the packages[\/\\] and plugins[\/\\] patterns in the source code (Babel and Eslint config files), and update them appropriately.
- Manually review .buildkite/scripts/pipelines/pull_request/pipeline.ts to ensure that any CI pipeline customizations continue to be correctly applied after the changed path names
- Review all of the updated files, specially the .ts and .js files listed in the sections below, as some of them contain relative paths that have been updated.
- Think of potential impact of the move, including tooling and configuration files that can be pointing to the relocated modules. E.g.:
  - customised eslint rules
  - docs pointing to source code

Note

This PR has been auto-generated.
Any manual contributions will be lost if the 'relocate' script is re-run.
Try to obtain the missing reviews / approvals before applying manual fixes, and/or keep your changes in a .patch / git stash.
Please use #sustainable_kibana_architecture Slack channel for feedback.

1 plugin(s) are going to be relocated:

Id	Target folder
`@kbn/elastic-assistant-plugin`	`x-pack/solutions/security/plugins/elastic_assistant`

3 packages(s) are going to be relocated:

Id	Target folder
`@kbn/elastic-assistant`	`x-pack/platform/packages/shared/kbn-elastic-assistant`
`@kbn/elastic-assistant-common`	`x-pack/platform/packages/shared/kbn-elastic-assistant-common`
`@kbn/langchain`	`x-pack/platform/packages/shared/kbn-langchain`

Updated references

./.buildkite/scripts/steps/code_generation/elastic_assistant_codegen.sh
./.buildkite/scripts/steps/openapi_bundling/security_solution_openapi_bundling.sh
./.eslintrc.js
./.github/codeql/codeql-config.yml
./docs/developer/plugin-list.asciidoc
./oas_docs/scripts/merge_ess_oas.js
./oas_docs/scripts/merge_serverless_oas.js
./package.json
./packages/kbn-repo-packages/package-map.json
./packages/kbn-ts-projects/config-paths.json
./tsconfig.base.json
./x-pack/.i18nrc.json
./x-pack/packages/kbn-elastic-assistant-common/README.md
./x-pack/platform/packages/shared/kbn-elastic-assistant-common/README.md
./x-pack/platform/packages/shared/kbn-elastic-assistant-common/env/README.md
./x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/capabilities/README.md
./x-pack/platform/packages/shared/kbn-elastic-assistant-common/jest.config.js
./x-pack/platform/packages/shared/kbn-elastic-assistant/README.md
./x-pack/platform/packages/shared/kbn-elastic-assistant/jest.config.js
./x-pack/platform/packages/shared/kbn-langchain/jest.config.js
./x-pack/plugins/elastic_assistant/README.md
./x-pack/plugins/security_solution/docs/openapi/README.md
./x-pack/solutions/security/plugins/elastic_assistant/README.md
./x-pack/solutions/security/plugins/elastic_assistant/jest.config.js
./x-pack/solutions/security/plugins/elastic_assistant/scripts/create_conversations_script.ts
./x-pack/solutions/security/plugins/elastic_assistant/server/__mocks__/docs_from_directory_loader.ts
./x-pack/solutions/security/plugins/elastic_assistant/server/knowledge_base/security_labs/embedding_security_in_llm_workflows.md
./x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/index.ts
./yarn.lock

Updated relative paths

x-pack/platform/packages/shared/kbn-elastic-assistant-common/jest.config.js:21
x-pack/platform/packages/shared/kbn-elastic-assistant-common/scripts/openapi/bundle.js:8
x-pack/platform/packages/shared/kbn-elastic-assistant-common/scripts/openapi/generate.js:8
x-pack/platform/packages/shared/kbn-elastic-assistant-common/tsconfig.json:2
x-pack/platform/packages/shared/kbn-elastic-assistant/jest.config.js:21
x-pack/platform/packages/shared/kbn-elastic-assistant/tsconfig.json:15
x-pack/platform/packages/shared/kbn-elastic-assistant/tsconfig.json:2
x-pack/platform/packages/shared/kbn-langchain/jest.config.js:20
x-pack/platform/packages/shared/kbn-langchain/tsconfig.json:2
x-pack/solutions/security/plugins/elastic_assistant/jest.config.js:14
x-pack/solutions/security/plugins/elastic_assistant/scripts/create_conversations.js:8
x-pack/solutions/security/plugins/elastic_assistant/scripts/draw_graph.js:8
x-pack/solutions/security/plugins/elastic_assistant/scripts/model_evaluator.js:8
x-pack/solutions/security/plugins/elastic_assistant/tsconfig.json:13
x-pack/solutions/security/plugins/elastic_assistant/tsconfig.json:2

github-actions · 2024-12-04T00:06:14Z

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

elasticmachine · 2024-12-04T08:23:37Z

Pinging @elastic/kibana-core (Team:Core)

KDKHD · 2024-12-09T17:22:33Z

...elastic_assistant/server/knowledge_base/security_labs/embedding_security_in_llm_workflows.md


-Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.
+Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.


Seems like this hyperlink was wrong originally already.

Suggested change

Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.

Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/platform/packages/shared/kbn-langchain/server/tracers/README.mdx) for more details.

Thanks! I'm re-running the script on this PR (against latest main).
The logic has ben enhanced since the previous run, so it'll now take into account references in .buildkite/scripts/steps/code_generation/elastic_assistant_codegen.sh (the culprit of CI failure).

Can you please update the broken path on a separate PR AFTER this one is merged? TIA

Mistake

KDKHD

Looks good. Tested locally with no issues.

kc13greiner

Changes to .github/codeql/codeql-config.yml LGTM

KDKHD · 2024-12-13T09:49:15Z

There is a missing update in the code owners here: https://github.com/elastic/kibana/blob/794fe0dc2d3484d93f046f5b3bf760602cd95504/.github/CODEOWNERS#L2457C1-L2457C50. Will this be updated in another team's PR?

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -

…e-ai

…thub.com:gsoldevila/kibana into kbn-team-1309-move-kibana-security-generative-ai

elasticmachine · 2024-12-13T15:35:46Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 913d7c2

Failed CI Steps

Post-Build

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [d89153d]

History

💚 Build #260059 succeeded 794fe0d
💔 Build #259739 failed c08f2f1
💔 Build #258964 failed afcd19d
💔 Build #258875 failed 2ff10da
💔 Build #257542 failed 6e1b13c

kibanamachine · 2024-12-13T15:39:44Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12318618998

kibanamachine · 2024-12-13T15:45:06Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 202848

Questions ?

Please refer to the Backport tool documentation

…rity-generative-ai` (elastic#202848) ## Summary This PR aims at relocating some of the Kibana modules (plugins and packages) into a new folder structure, according to the _Sustainable Kibana Architecture_ initiative. > [!IMPORTANT] > * We kindly ask you to: > * Manually fix the errors in the error section below (if there are any). > * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the source code (Babel and Eslint config files), and update them appropriately. > * Manually review `.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that any CI pipeline customizations continue to be correctly applied after the changed path names > * Review all of the updated files, specially the `.ts` and `.js` files listed in the sections below, as some of them contain relative paths that have been updated. > * Think of potential impact of the move, including tooling and configuration files that can be pointing to the relocated modules. E.g.: > * customised eslint rules > * docs pointing to source code > [!NOTE] > * This PR has been auto-generated. > * Any manual contributions will be lost if the 'relocate' script is re-run. > * Try to obtain the missing reviews / approvals before applying manual fixes, and/or keep your changes in a .patch / git stash. > * Please use [#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E) Slack channel for feedback. #### 1 plugin(s) are going to be relocated: | Id | Target folder | | -- | ------------- | | `@kbn/elastic-assistant-plugin` | `x-pack/solutions/security/plugins/elastic_assistant` | #### 3 packages(s) are going to be relocated: | Id | Target folder | | -- | ------------- | | `@kbn/elastic-assistant` | `x-pack/platform/packages/shared/kbn-elastic-assistant` | | `@kbn/elastic-assistant-common` | `x-pack/platform/packages/shared/kbn-elastic-assistant-common` | | `@kbn/langchain` | `x-pack/platform/packages/shared/kbn-langchain` | <details > <summary>Updated references</summary> ``` ./.buildkite/scripts/steps/code_generation/elastic_assistant_codegen.sh ./.buildkite/scripts/steps/openapi_bundling/security_solution_openapi_bundling.sh ./.eslintrc.js ./.github/codeql/codeql-config.yml ./docs/developer/plugin-list.asciidoc ./oas_docs/scripts/merge_ess_oas.js ./oas_docs/scripts/merge_serverless_oas.js ./package.json ./packages/kbn-repo-packages/package-map.json ./packages/kbn-ts-projects/config-paths.json ./tsconfig.base.json ./tsconfig.base.type_check.json ./tsconfig.refs.json ./x-pack/.i18nrc.json ./x-pack/packages/kbn-elastic-assistant-common/README.md ./x-pack/platform/packages/shared/kbn-elastic-assistant-common/README.md ./x-pack/platform/packages/shared/kbn-elastic-assistant-common/env/README.md ./x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/capabilities/README.md ./x-pack/platform/packages/shared/kbn-elastic-assistant-common/jest.config.js ./x-pack/platform/packages/shared/kbn-elastic-assistant/README.md ./x-pack/platform/packages/shared/kbn-elastic-assistant/jest.config.js ./x-pack/platform/packages/shared/kbn-langchain/jest.config.js ./x-pack/plugins/elastic_assistant/README.md ./x-pack/plugins/security_solution/docs/openapi/README.md ./x-pack/solutions/security/plugins/elastic_assistant/README.md ./x-pack/solutions/security/plugins/elastic_assistant/jest.config.js ./x-pack/solutions/security/plugins/elastic_assistant/scripts/create_conversations_script.ts ./x-pack/solutions/security/plugins/elastic_assistant/server/__mocks__/docs_from_directory_loader.ts ./x-pack/solutions/security/plugins/elastic_assistant/server/knowledge_base/security_labs/embedding_security_in_llm_workflows.md ./x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/index.ts ./yarn.lock ``` </details><details > <summary>Updated relative paths</summary> ``` x-pack/platform/packages/shared/kbn-elastic-assistant-common/jest.config.js:21 x-pack/platform/packages/shared/kbn-elastic-assistant-common/scripts/openapi/bundle.js:8 x-pack/platform/packages/shared/kbn-elastic-assistant-common/scripts/openapi/generate.js:8 x-pack/platform/packages/shared/kbn-elastic-assistant-common/tsconfig.json:2 x-pack/platform/packages/shared/kbn-elastic-assistant/jest.config.js:21 x-pack/platform/packages/shared/kbn-elastic-assistant/tsconfig.json:15 x-pack/platform/packages/shared/kbn-elastic-assistant/tsconfig.json:2 x-pack/platform/packages/shared/kbn-langchain/jest.config.js:20 x-pack/platform/packages/shared/kbn-langchain/tsconfig.json:2 x-pack/solutions/security/plugins/elastic_assistant/jest.config.js:14 x-pack/solutions/security/plugins/elastic_assistant/scripts/create_conversations.js:8 x-pack/solutions/security/plugins/elastic_assistant/scripts/draw_graph.js:8 x-pack/solutions/security/plugins/elastic_assistant/scripts/model_evaluator.js:8 x-pack/solutions/security/plugins/elastic_assistant/tsconfig.json:13 x-pack/solutions/security/plugins/elastic_assistant/tsconfig.json:2 ``` </details> --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 6de695a) # Conflicts: # .github/CODEOWNERS # oas_docs/scripts/merge_serverless_oas.js # x-pack/.i18nrc.json # x-pack/platform/packages/shared/kbn-elastic-assistant/tsconfig.json

gsoldevila · 2024-12-15T20:43:45Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-12-16T15:49:03Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

stephmilovic · 2024-12-16T18:47:16Z

backport: #204468

…c/security-generative-ai` (#202848) (#204468) ## Summary Summarize your PR. If it involves visual changes include a screenshot or gif. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Gerard Soldevila <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

kibanamachine · 2024-12-17T15:49:43Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 202848 locally

kibanamachine · 2024-12-19T10:48:56Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 202848 locally

gsoldevila marked this pull request as ready for review December 4, 2024 08:23

gsoldevila requested review from a team as code owners December 4, 2024 08:23

gsoldevila requested review from pzl and szwarckonrad December 4, 2024 08:23

pzl requested review from ashokaditya and removed request for pzl December 4, 2024 13:38

gsoldevila force-pushed the kbn-team-1309-move-kibana-security-generative-ai branch from 0ad279d to 5b3a95b Compare December 5, 2024 15:45

KDKHD reviewed Dec 9, 2024

View reviewed changes

KDKHD previously approved these changes Dec 9, 2024

View reviewed changes

KDKHD approved these changes Dec 9, 2024

View reviewed changes

rudolf approved these changes Dec 9, 2024

View reviewed changes

gsoldevila force-pushed the kbn-team-1309-move-kibana-security-generative-ai branch from 6e1b13c to b9f322e Compare December 10, 2024 15:54

gsoldevila requested a review from a team as a code owner December 10, 2024 15:54

kc13greiner self-requested a review December 10, 2024 19:42

kc13greiner approved these changes Dec 10, 2024

View reviewed changes

Ikuni17 approved these changes Dec 11, 2024

View reviewed changes

joeypoon approved these changes Dec 11, 2024

View reviewed changes

szwarckonrad approved these changes Dec 12, 2024

View reviewed changes

gsoldevila force-pushed the kbn-team-1309-move-kibana-security-generative-ai branch from afcd19d to 6c886e4 Compare December 12, 2024 11:21

KDKHD added 7 commits December 13, 2024 13:28

Relocating module @kbn/elastic-assistant

859b1cb

Relocating module @kbn/elastic-assistant-common

bbfeb90

Relocating module @kbn/elastic-assistant-plugin

d571826

Relocating module @kbn/langchain

e193a49

adjust styled_components_files.js to include kbn-elastic-assistant

a2fc85d

Fix broken link

39d8e2f

@kibanamachine

bb416ea

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -

KDKHD force-pushed the kbn-team-1309-move-kibana-security-generative-ai branch from 794fe0d to bb416ea Compare December 13, 2024 13:48

KDKHD and others added 3 commits December 13, 2024 13:49

Merge branch 'main' into kbn-team-1309-move-kibana-security-generativ…

5daf411

…e-ai

fix path in codeowners

372081d

Merge branch 'kbn-team-1309-move-kibana-security-generative-ai' of gi…

913d7c2

…thub.com:gsoldevila/kibana into kbn-team-1309-move-kibana-security-generative-ai

gsoldevila merged commit 6de695a into elastic:main Dec 13, 2024
8 checks passed

kibanamachine added the v9.0.0 label Dec 13, 2024

gsoldevila mentioned this pull request Dec 15, 2024

[8.x] Sustainable Kibana Architecture: Move modules owned by @elastic/security-generative-ai (#202848) #204326

Closed

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 16, 2024

rudolf added the plugin groups Relocate all plugins and packages under their respective platform/solution groups label Dec 18, 2024

marshallmain added backport:skip This commit does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Dec 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sustainable Kibana Architecture: Move modules owned by `@elastic/security-generative-ai` #202848

Sustainable Kibana Architecture: Move modules owned by `@elastic/security-generative-ai` #202848

gsoldevila commented Dec 4, 2024 •

edited by kibanamachine

Loading

github-actions bot commented Dec 4, 2024

elasticmachine commented Dec 4, 2024

KDKHD Dec 9, 2024 •

edited

Loading

gsoldevila Dec 10, 2024

KDKHD left a comment

kc13greiner left a comment

KDKHD commented Dec 13, 2024 •

edited

Loading

elasticmachine commented Dec 13, 2024 •

edited

Loading

kibanamachine commented Dec 13, 2024

kibanamachine commented Dec 13, 2024

gsoldevila commented Dec 15, 2024

kibanamachine commented Dec 16, 2024

stephmilovic commented Dec 16, 2024

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 19, 2024


		Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.
		Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.

Sustainable Kibana Architecture: Move modules owned by @elastic/security-generative-ai #202848

Sustainable Kibana Architecture: Move modules owned by @elastic/security-generative-ai #202848

Conversation

gsoldevila commented Dec 4, 2024 • edited by kibanamachine Loading

Summary

1 plugin(s) are going to be relocated:

3 packages(s) are going to be relocated:

github-actions bot commented Dec 4, 2024

elasticmachine commented Dec 4, 2024

KDKHD Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

gsoldevila Dec 10, 2024

Choose a reason for hiding this comment

KDKHD left a comment

Choose a reason for hiding this comment

kc13greiner left a comment

Choose a reason for hiding this comment

KDKHD commented Dec 13, 2024 • edited Loading

elasticmachine commented Dec 13, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

kibanamachine commented Dec 13, 2024

kibanamachine commented Dec 13, 2024

💔 All backports failed

Manual backport

Questions ?

gsoldevila commented Dec 15, 2024

💚 All backports created successfully

Questions ?

kibanamachine commented Dec 16, 2024

stephmilovic commented Dec 16, 2024

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 19, 2024

Sustainable Kibana Architecture: Move modules owned by `@elastic/security-generative-ai` #202848

Sustainable Kibana Architecture: Move modules owned by `@elastic/security-generative-ai` #202848

gsoldevila commented Dec 4, 2024 •

edited by kibanamachine

Loading

KDKHD Dec 9, 2024 •

edited

Loading

KDKHD commented Dec 13, 2024 •

edited

Loading

elasticmachine commented Dec 13, 2024 •

edited

Loading