Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Telemetry][Security Solution] Index metadata collector #194004

Merged
merged 62 commits into from
Dec 13, 2024
Merged

[Telemetry][Security Solution] Index metadata collector #194004

merged 62 commits into from
Dec 13, 2024
+3,087 −136

Conversation

szaffarano
Copy link
Contributor

@szaffarano szaffarano commented Sep 25, 2024 •
edited by kibanamachine
Loading

Summary

Implements a security_solution task scheduled to run once a day to collect the following information:

  1. Datastreams stats
  2. Indices stats
  3. ILMs stats
  4. ILM configs

The task allows a runtime configuration to limit the number of indices and data streams to analyze or event to disable the feature entirely.

Once the data is gathered, the task sends it as EBT events.

@szaffarano szaffarano force-pushed the szaffarano/datastreams-metadata-poc branch 3 times, most recently from 4d7a50f to f0bf108 Compare September 25, 2024 13:45
@szaffarano szaffarano added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Sep 25, 2024
@szaffarano szaffarano self-assigned this Sep 25, 2024
@szaffarano szaffarano force-pushed the szaffarano/datastreams-metadata-poc branch 2 times, most recently from 2412982 to 5c1a6da Compare September 26, 2024 16:46
@szaffarano
Copy link
Contributor Author

@elasticmachine merge upstream

@szaffarano szaffarano marked this pull request as ready for review October 1, 2024 17:59
@szaffarano szaffarano requested review from a team as code owners October 1, 2024 17:59
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@szaffarano szaffarano added the backport:all-open Backport to all branches that could still receive a release label Oct 1, 2024
pmuellr
pmuellr approved these changes Oct 1, 2024
View reviewed changes
Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResponseOps changes LGTM, made some drive-by comments

@szaffarano szaffarano added ci:project-deploy-observability Create an Observability project and removed ci:project-deploy-observability Create an Observability project labels Oct 2, 2024
@szaffarano
Copy link
Contributor Author

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 539 542 +3

Total ESLint disabled count

id before after diff
securitySolution 624 627 +3

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @szaffarano

@donaherc donaherc changed the title [Telemetry][Security Solution] Index metadata [Telemetry][POC][Security Solution] Index metadata Oct 8, 2024
@szaffarano
Copy link
Contributor Author

@elasticmachine merge upstream

@szaffarano szaffarano enabled auto-merge (squash) December 13, 2024 16:54
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 13, 2024 •
edited
Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #51 / aiops log pattern analysis attaches log pattern analysis table to a case

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 569 572 +3

Total ESLint disabled count

id before after diff
securitySolution 653 656 +3

History

cc @szaffarano

@szaffarano szaffarano merged commit 36b344a into main Dec 13, 2024
8 checks passed
@szaffarano szaffarano deleted the szaffarano/datastreams-metadata-poc branch December 13, 2024 18:31
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 7.17, 8.15, 8.16, 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/12321079152

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 7.17 Backport failed because of merge conflicts
❌ 8.15 Backport failed because of merge conflicts
❌ 8.16 Backport failed because of merge conflicts
❌ 8.17 Backport failed because of merge conflicts
❌ 8.x Backport failed because of merge conflicts

You might need to backport the following PRs to 8.x:
- [Security Solution][Endpoint] Add FTR API tests that validates creation of DOT indices (#197899)

Manual backport

To create the backport manually run:

node scripts/backport --pr 194004

Questions ?

Please refer to the Backport tool documentation

@szaffarano szaffarano mentioned this pull request Dec 14, 2024
Merged
@szaffarano
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@kibanamachine kibanamachine mentioned this pull request Dec 16, 2024
Merged
szaffarano added a commit that referenced this pull request Dec 16, 2024
@szaffarano @elasticmachine
[8.x] [Telemetry][Security Solution] Index metadata collector (#194004)…
7a3de72 
… (#204311)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Telemetry][Security Solution] Index metadata collector
(#194004)](#194004)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Sebastián
Zaffarano","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-13T18:31:03Z","message":"[Telemetry][Security
Solution] Index metadata collector (#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:
SecuritySolution","backport:all-open","ci:cloud-deploy"],"number":194004,"url":"https://github.com/elastic/kibana/pull/194004","mergeCommit":{"message":"[Telemetry][Security
Solution] Index metadata collector (#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194004","number":194004,"mergeCommit":{"message":"[Telemetry][Security
Solution] Index metadata collector (#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d"}}]}]
BACKPORT-->

Co-authored-by: Elastic Machine <[email protected]>
This was referenced Dec 17, 2024
Open
Open
szaffarano added a commit to szaffarano/kibana that referenced this pull request Dec 17, 2024
@szaffarano
[8.x] [Telemetry][Security Solution] Index metadata collector (elasti…
91115c6 
…c#194004) (elastic#204311)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Telemetry][Security Solution] Index metadata collector
(elastic#194004)](elastic#194004)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Sebastián
Zaffarano","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-13T18:31:03Z","message":"[Telemetry][Security
Solution] Index metadata collector (elastic#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:
SecuritySolution","backport:all-open","ci:cloud-deploy"],"number":194004,"url":"https://github.com/elastic/kibana/pull/194004","mergeCommit":{"message":"[Telemetry][Security
Solution] Index metadata collector (elastic#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194004","number":194004,"mergeCommit":{"message":"[Telemetry][Security
Solution] Index metadata collector (elastic#194004)\n\n##
Summary\r\n\r\nImplements a security_solution task scheduled to run once
a day to\r\ncollect the following information:\r\n\r\n1. Datastreams
stats\r\n2. Indices stats\r\n3. ILMs stats\r\n4. ILM configs\r\n\r\nThe
task allows a runtime configuration to limit the number of
indices\r\nand data streams to analyze or event to disable the feature
entirely.\r\n\r\nOnce the data is gathered, the task sends it as EBT
events.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"36b344a4c58a3d78a892288e0eef71e9ff163b9d"}}]}]
BACKPORT-->

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 7a3de72)
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 18, 2024
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

10 similar comments
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JDKurma JDKurma JDKurma left review comments

@KeerthyND KeerthyND KeerthyND approved these changes

@marshallmain marshallmain marshallmain approved these changes

@angorayc angorayc angorayc approved these changes

@afharo afharo afharo approved these changes

@jbudz jbudz jbudz approved these changes

@rudolf rudolf rudolf approved these changes

@pmuellr pmuellr pmuellr approved these changes

Assignees

@szaffarano szaffarano

Labels
backport:all-open Backport to all branches that could still receive a release backport missing Added to PRs automatically when the are determined to be missing a backport. ci:cloud-deploy Create or update a Cloud deployment release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.