Skip to content

Commit

Permalink
Sustainable Kibana Architecture: Move modules owned by `@elastic/stac…
Browse files Browse the repository at this point in the history
…k-monitoring` (#202860)

## Summary

This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.

> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
>     * customised eslint rules
>     * docs pointing to source code

> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.

Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).

#### 2 plugin(s) are going to be relocated:

| Id | Target folder |
| -- | ------------- |
| `@kbn/monitoring-collection-plugin` |
`x-pack/platform/plugins/private/monitoring_collection` |
| `@kbn/monitoring-plugin` |
`x-pack/platform/plugins/private/monitoring` |


<details >
<summary>Updated relative paths</summary>

```
x-pack/platform/plugins/private/monitoring/dev_docs/how_to/testing.md:7
x-pack/platform/plugins/private/monitoring/dev_docs/how_to/testing.md:8
x-pack/platform/plugins/private/monitoring/jest.config.js:10
x-pack/platform/plugins/private/monitoring/tsconfig.json:2
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:100
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:103
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:106
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:109
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:115
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:121
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:127
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:130
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:133
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:136
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:139
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:142
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:145
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:19
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:2
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:22
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:25
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:28
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:31
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:34
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:37
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:64
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:67
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:70
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:73
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:76
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:79
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:82
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:85
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:88
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:91
x-pack/platform/plugins/private/monitoring/tsconfig.type_check.json:97
x-pack/platform/plugins/private/monitoring_collection/jest.config.js:10
x-pack/platform/plugins/private/monitoring_collection/tsconfig.json:2
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:16
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:19
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:2
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:22
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:25
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:28
x-pack/platform/plugins/private/monitoring_collection/tsconfig.type_check.json:31
```

</details>

---------

Co-authored-by: pgayvallet <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
  • Loading branch information
3 people authored Dec 24, 2024
1 parent 681d40e commit 47f675c
Show file tree
Hide file tree
Showing 1,105 changed files with 394 additions and 143 deletions.
6 changes: 3 additions & 3 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,8 @@ x-pack/platform/plugins/private/grokdebugger @elastic/kibana-management
x-pack/platform/plugins/private/index_lifecycle_management @elastic/kibana-management
x-pack/platform/plugins/private/license_api_guard @elastic/kibana-management
x-pack/platform/plugins/private/logstash @elastic/logstash
x-pack/platform/plugins/private/monitoring @elastic/stack-monitoring
x-pack/platform/plugins/private/monitoring_collection @elastic/stack-monitoring
x-pack/platform/plugins/private/painless_lab @elastic/kibana-management
x-pack/platform/plugins/private/remote_clusters @elastic/kibana-management
x-pack/platform/plugins/private/rollup @elastic/kibana-management
Expand Down Expand Up @@ -900,8 +902,6 @@ x-pack/plugins/global_search_providers @elastic/appex-sharedux
x-pack/plugins/graph @elastic/kibana-visualizations
x-pack/plugins/lens @elastic/kibana-visualizations
x-pack/plugins/licensing @elastic/kibana-core
x-pack/plugins/monitoring @elastic/stack-monitoring
x-pack/plugins/monitoring_collection @elastic/stack-monitoring
x-pack/plugins/notifications @elastic/appex-sharedux
x-pack/plugins/observability_solution/apm @elastic/obs-ux-infra_services-team
x-pack/plugins/observability_solution/apm_data_access @elastic/obs-ux-infra_services-team
Expand Down Expand Up @@ -2544,7 +2544,7 @@ x-pack/solutions/security/plugins/security_solution/server/lib/security_integrat

# Observability design
/x-pack/plugins/fleet/**/*.scss @elastic/observability-design
/x-pack/plugins/monitoring/**/*.scss @elastic/observability-design
/x-pack/platform/plugins/private/monitoring/**/*.scss @elastic/observability-design

# Ent. Search design
/x-pack/plugins/enterprise_search/**/*.scss @elastic/search-design
Expand Down
4 changes: 2 additions & 2 deletions docs/developer/plugin-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -728,11 +728,11 @@ using the CURL scripts in the scripts folder.
Elastic.
|{kib-repo}blob/{branch}/x-pack/plugins/monitoring/readme.md[monitoring]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/private/monitoring/readme.md[monitoring]
|This plugin provides the Stack Monitoring kibana application.
|{kib-repo}blob/{branch}/x-pack/plugins/monitoring_collection/README.md[monitoringCollection]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/private/monitoring_collection/README.md[monitoringCollection]
|This plugin allows for other plugins to add data to Kibana stack monitoring documents.
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -682,8 +682,8 @@
"@kbn/ml-url-state": "link:x-pack/platform/packages/private/ml/url_state",
"@kbn/ml-validators": "link:x-pack/platform/packages/private/ml/validators",
"@kbn/monaco": "link:packages/kbn-monaco",
"@kbn/monitoring-collection-plugin": "link:x-pack/plugins/monitoring_collection",
"@kbn/monitoring-plugin": "link:x-pack/plugins/monitoring",
"@kbn/monitoring-collection-plugin": "link:x-pack/platform/plugins/private/monitoring_collection",
"@kbn/monitoring-plugin": "link:x-pack/platform/plugins/private/monitoring",
"@kbn/navigation-plugin": "link:src/plugins/navigation",
"@kbn/newsfeed-plugin": "link:src/plugins/newsfeed",
"@kbn/newsfeed-test-plugin": "link:test/common/plugins/newsfeed",
Expand Down
10 changes: 5 additions & 5 deletions src/dev/precommit_hook/casing_check_config.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export const IGNORE_FILE_GLOBS = [
'x-pack/platform/plugins/private/canvas/canvas_plugin_src/**/*',
'x-pack/platform/plugins/private/canvas/server/templates/assets/*.{png,jpg,svg}',
'x-pack/plugins/cases/docs/**/*',
'x-pack/plugins/monitoring/public/lib/jquery_flot/**/*',
'x-pack/platform/plugins/private/monitoring/public/lib/jquery_flot/**/*',
'x-pack/plugins/fleet/cypress/packages/*.zip',
'**/apm-diagnostics-*.json',
'**/.*',
Expand Down Expand Up @@ -178,10 +178,10 @@ export const TEMPORARILY_IGNORED_PATHS = [
'test/functional/apps/management/exports/_import_objects-conflicts.json',
'x-pack/legacy/platform/plugins/shared/index_management/public/lib/editSettings.js',
'x-pack/legacy/platform/plugins/shared/license_management/public/store/reducers/licenseManagement.js',
'x-pack/plugins/monitoring/public/icons/health-gray.svg',
'x-pack/plugins/monitoring/public/icons/health-green.svg',
'x-pack/plugins/monitoring/public/icons/health-red.svg',
'x-pack/plugins/monitoring/public/icons/health-yellow.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-gray.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-green.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-red.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-yellow.svg',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/noto/NotoSansCJKtc-Medium.ttf',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/noto/NotoSansCJKtc-Regular.ttf',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/roboto/Roboto-Italic.ttf',
Expand Down
8 changes: 4 additions & 4 deletions tsconfig.base.json
Original file line number Diff line number Diff line change
Expand Up @@ -1298,10 +1298,10 @@
"@kbn/mock-idp-utils/*": ["packages/kbn-mock-idp-utils/*"],
"@kbn/monaco": ["packages/kbn-monaco"],
"@kbn/monaco/*": ["packages/kbn-monaco/*"],
"@kbn/monitoring-collection-plugin": ["x-pack/plugins/monitoring_collection"],
"@kbn/monitoring-collection-plugin/*": ["x-pack/plugins/monitoring_collection/*"],
"@kbn/monitoring-plugin": ["x-pack/plugins/monitoring"],
"@kbn/monitoring-plugin/*": ["x-pack/plugins/monitoring/*"],
"@kbn/monitoring-collection-plugin": ["x-pack/platform/plugins/private/monitoring_collection"],
"@kbn/monitoring-collection-plugin/*": ["x-pack/platform/plugins/private/monitoring_collection/*"],
"@kbn/monitoring-plugin": ["x-pack/platform/plugins/private/monitoring"],
"@kbn/monitoring-plugin/*": ["x-pack/platform/plugins/private/monitoring/*"],
"@kbn/navigation-plugin": ["src/plugins/navigation"],
"@kbn/navigation-plugin/*": ["src/plugins/navigation/*"],
"@kbn/newsfeed-plugin": ["src/plugins/newsfeed"],
Expand Down
2 changes: 1 addition & 1 deletion x-pack/.i18nrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@
"platform/plugins/shared/ml"
],
"xpack.monitoring": [
"plugins/monitoring"
"platform/plugins/private/monitoring"
],
"xpack.observability": "solutions/observability/plugins/observability",
"xpack.observabilityAiAssistant": [
Expand Down
4 changes: 2 additions & 2 deletions x-pack/.telemetryrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
"output": "plugins/telemetry_collection_xpack/schema/xpack_plugins.json",
"root": "plugins/",
"exclude": [
"plugins/monitoring/server/telemetry_collection/"
"platform/plugins/private/monitoring/server/telemetry_collection/"
]
},
{
"output": "plugins/telemetry_collection_xpack/schema/xpack_monitoring.json",
"root": "plugins/monitoring/server/telemetry_collection/",
"root": "platform/plugins/private/monitoring/server/telemetry_collection/",
"exclude": []
},
{
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
When diagnosing issues with UI or alert errors, it can be useful to know the exact query being sent by kibana to elasticsearch.

Rules should have a `fetchData` setup [like this one](/x-pack/plugins/monitoring/server/alerts/nodes_changed_rule.ts#L96). Tracing down will get you to `fetchNodesFromClusterStats`.
Rules should have a `fetchData` setup [like this one](/x-pack/platform/plugins/private/monitoring/server/alerts/nodes_changed_rule.ts#L96). Tracing down will get you to `fetchNodesFromClusterStats`.

Just before the `esClient.search` call you can log out the query parameters using something like:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ docker run --name metricbeat \
--pull always --rm \
--hostname=metricbeat \
--publish=5066:5066 \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/metricbeat.yarn.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/metricbeat.yarn.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
docker.elastic.co/beats/metricbeat:master-SNAPSHOT
```

Expand All @@ -65,7 +65,7 @@ docker run --name filebeat \
--hostname=filebeat \
--publish=5067:5067 \
--volume="$(pwd)/.es:/es:ro" \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/filebeat.yarn.yml:/usr/share/filebeat/filebeat.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/filebeat.yarn.yml:/usr/share/filebeat/filebeat.yml:ro" \
docker.elastic.co/beats/filebeat:master-SNAPSHOT
```

Expand All @@ -80,8 +80,8 @@ docker run --name logstash \
--pull always --rm \
--hostname=logstash \
--publish=9600:9600 \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \
docker.elastic.co/logstash/logstash:master-SNAPSHOT
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ See the [Kibana Testing guide](https://www.elastic.co/guide/en/kibana/master/dev

We mainly use:
1. Jest unit tests - located in sibling files to the source code
2. [api_integration tests](../../../../test/api_integration/apis/monitoring)
3. [functional tests](../../../../test/functional/apps/monitoring)
2. [api_integration tests](../../../../../../test/api_integration/apis/monitoring)
3. [functional tests](../../../../../../test/functional/apps/monitoring)

The functional and api integration tests are both under a 'Monitoring' description, so you can use `--grep Monitoring` to run only our tests.
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ subgraph Kibana
click Collectors "https://github.com/elastic/kibana/tree/main/src/core/server/metrics/collectors"
click OpsMetricsObservable "https://github.com/elastic/kibana/blob/92a8636f0ff63ab072527574e96e6616327b2ea4/src/core/server/metrics/metrics_service.ts#L32"
click BulkUploader "https://github.com/elastic/kibana/blob/main/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.ts"
click BulkUploader "https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/private/monitoring/server/kibana_monitoring/bulk_uploader.ts"
end
BulkUploader-->|/_monitoring/bulk|ProdElasticsearch
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ When trying to debug why a CPU metric doesn't look the way you expect it to in a

At the time of writing, the code path to get from a system level CPU metric to a utilization percentage looks like this:

1. `node_cpu_metric` set to `node_cgroup_quota_as_cpu_utilization` when cgroup is enabled: [node_detail.js](/x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/node_detail.js#L61-65)
1. `node_cgroup_quota_as_cpu_utilization` defined as a `QuotaMetric` against `cpu.cfs_quota_micros`: [metrics.ts](/x-pack/plugins/monitoring/server/lib/metrics/elasticsearch/metrics.ts#L798-801)
1. `QuotaMetric` tries to produce a ratio of usage to quota, but returns null when quota isn't a positive number: [quota_metric.ts](/x-pack/plugins/monitoring/server/lib/metrics/classes/quota_metric.ts#L79-80)
1. `node_cpu_metric` set to `node_cgroup_quota_as_cpu_utilization` when cgroup is enabled: [node_detail.js](/x-pack/platform/plugins/private/monitoring/server/routes/api/v1/elasticsearch/node_detail.js#L61-65)
1. `node_cgroup_quota_as_cpu_utilization` defined as a `QuotaMetric` against `cpu.cfs_quota_micros`: [metrics.ts](/x-pack/platform/plugins/private/monitoring/server/lib/metrics/elasticsearch/metrics.ts#L798-801)
1. `QuotaMetric` tries to produce a ratio of usage to quota, but returns null when quota isn't a positive number: [quota_metric.ts](/x-pack/platform/plugins/private/monitoring/server/lib/metrics/classes/quota_metric.ts#L79-80)

So it's important to be aware of the `monitoring.ui.container.elasticsearch.enabled` setting, which defaults to `true` on cloud.elastic.co.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
If the stack monitoring UI isn't showing data for any cluster or missing key metrics, it may first be useful to survey the available data.

If troubleshooting a cluster with a version >= 8.3.1, the [Stack Monitoring health API](https://github.com/elastic/kibana/tree/main/x-pack/plugins/monitoring/server/routes/api/v1/_health) is the recommended way to get an overview of the available data and possible metrics collection issues.
If troubleshooting a cluster with a version >= 8.3.1, the [Stack Monitoring health API](https://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/private/monitoring/server/routes/api/v1/_health) is the recommended way to get an overview of the available data and possible metrics collection issues.
The API is included in the [support-diagnostics utility](https://github.com/elastic/support-diagnostics) so if a bundle is provided in the issue you're working on, the API response would already be available in `kibana_stack_monitoring_health.json`. Otherwise, one can ask for the API response instead of the raw queries.

If troubleshooting an older version, the following queries would be good starters:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@

module.exports = {
preset: '@kbn/test',
rootDir: '../../..',
roots: ['<rootDir>/x-pack/plugins/monitoring'],
coverageDirectory: '<rootDir>/target/kibana-coverage/jest/x-pack/plugins/monitoring',
rootDir: '../../../../..',
roots: ['<rootDir>/x-pack/platform/plugins/private/monitoring'],
coverageDirectory:
'<rootDir>/target/kibana-coverage/jest/x-pack/platform/plugins/private/monitoring',
coverageReporters: ['text', 'html'],
collectCoverageFrom: [
'<rootDir>/x-pack/plugins/monitoring/{common,public,server}/**/*.{js,ts,tsx}',
'<rootDir>/x-pack/platform/plugins/private/monitoring/{common,public,server}/**/*.{js,ts,tsx}',
],
};
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* 2.0.
*/

// From x-pack/plugins/monitoring/public/lib/elasticsearch_settings/enabler.js
// From x-pack/platform/plugins/private/monitoring/public/lib/elasticsearch_settings/enabler.js
export class Enabler {
http: any;
updateModel: any;
Expand Down
Loading

0 comments on commit 47f675c

Please sign in to comment.