Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x] Sustainable Kibana Architecture: Move modules owned by @elastic/stack-monitoring (#202860) #205139

Merged
merged 4 commits into from
Dec 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 3 additions & 3 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -651,8 +651,8 @@ x-pack/platform/packages/private/ml/validators @elastic/ml-ui
packages/kbn-mock-idp-plugin @elastic/kibana-security
packages/kbn-mock-idp-utils @elastic/kibana-security
packages/kbn-monaco @elastic/appex-sharedux
x-pack/plugins/monitoring_collection @elastic/stack-monitoring
x-pack/plugins/monitoring @elastic/stack-monitoring
x-pack/platform/plugins/private/monitoring_collection @elastic/stack-monitoring
x-pack/platform/plugins/private/monitoring @elastic/stack-monitoring
src/plugins/navigation @elastic/appex-sharedux
src/plugins/newsfeed @elastic/kibana-core
test/common/plugins/newsfeed @elastic/kibana-core
Expand Down Expand Up @@ -1926,7 +1926,7 @@ x-pack/solutions/security/plugins/security_solution/server/lib/security_integrat

# Observability design
/x-pack/plugins/fleet/**/*.scss @elastic/observability-design
/x-pack/plugins/monitoring/**/*.scss @elastic/observability-design
/x-pack/platform/plugins/private/monitoring/**/*.scss @elastic/observability-design

# Ent. Search design
/x-pack/plugins/enterprise_search/**/*.scss @elastic/search-design
Expand Down
4 changes: 2 additions & 2 deletions docs/developer/plugin-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -728,11 +728,11 @@ using the CURL scripts in the scripts folder.
Elastic.


|{kib-repo}blob/{branch}/x-pack/plugins/monitoring/readme.md[monitoring]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/private/monitoring/readme.md[monitoring]
|This plugin provides the Stack Monitoring kibana application.


|{kib-repo}blob/{branch}/x-pack/plugins/monitoring_collection/README.md[monitoringCollection]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/private/monitoring_collection/README.md[monitoringCollection]
|This plugin allows for other plugins to add data to Kibana stack monitoring documents.


Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -681,8 +681,8 @@
"@kbn/ml-url-state": "link:x-pack/platform/packages/private/ml/url_state",
"@kbn/ml-validators": "link:x-pack/platform/packages/private/ml/validators",
"@kbn/monaco": "link:packages/kbn-monaco",
"@kbn/monitoring-collection-plugin": "link:x-pack/plugins/monitoring_collection",
"@kbn/monitoring-plugin": "link:x-pack/plugins/monitoring",
"@kbn/monitoring-collection-plugin": "link:x-pack/platform/plugins/private/monitoring_collection",
"@kbn/monitoring-plugin": "link:x-pack/platform/plugins/private/monitoring",
"@kbn/navigation-plugin": "link:src/plugins/navigation",
"@kbn/newsfeed-plugin": "link:src/plugins/newsfeed",
"@kbn/newsfeed-test-plugin": "link:test/common/plugins/newsfeed",
Expand Down
10 changes: 5 additions & 5 deletions src/dev/precommit_hook/casing_check_config.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export const IGNORE_FILE_GLOBS = [
'x-pack/platform/plugins/private/canvas/canvas_plugin_src/**/*',
'x-pack/platform/plugins/private/canvas/server/templates/assets/*.{png,jpg,svg}',
'x-pack/plugins/cases/docs/**/*',
'x-pack/plugins/monitoring/public/lib/jquery_flot/**/*',
'x-pack/platform/plugins/private/monitoring/public/lib/jquery_flot/**/*',
'x-pack/plugins/fleet/cypress/packages/*.zip',
'**/apm-diagnostics-*.json',
'**/.*',
Expand Down Expand Up @@ -174,10 +174,10 @@ export const TEMPORARILY_IGNORED_PATHS = [
'test/functional/apps/management/exports/_import_objects-conflicts.json',
'x-pack/legacy/platform/plugins/shared/index_management/public/lib/editSettings.js',
'x-pack/legacy/platform/plugins/shared/license_management/public/store/reducers/licenseManagement.js',
'x-pack/plugins/monitoring/public/icons/health-gray.svg',
'x-pack/plugins/monitoring/public/icons/health-green.svg',
'x-pack/plugins/monitoring/public/icons/health-red.svg',
'x-pack/plugins/monitoring/public/icons/health-yellow.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-gray.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-green.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-red.svg',
'x-pack/platform/plugins/private/monitoring/public/icons/health-yellow.svg',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/noto/NotoSansCJKtc-Medium.ttf',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/noto/NotoSansCJKtc-Regular.ttf',
'x-pack/platform/plugins/shared/screenshotting/server/assets/fonts/roboto/Roboto-Italic.ttf',
Expand Down
8 changes: 4 additions & 4 deletions tsconfig.base.json
Original file line number Diff line number Diff line change
Expand Up @@ -1296,10 +1296,10 @@
"@kbn/mock-idp-utils/*": ["packages/kbn-mock-idp-utils/*"],
"@kbn/monaco": ["packages/kbn-monaco"],
"@kbn/monaco/*": ["packages/kbn-monaco/*"],
"@kbn/monitoring-collection-plugin": ["x-pack/plugins/monitoring_collection"],
"@kbn/monitoring-collection-plugin/*": ["x-pack/plugins/monitoring_collection/*"],
"@kbn/monitoring-plugin": ["x-pack/plugins/monitoring"],
"@kbn/monitoring-plugin/*": ["x-pack/plugins/monitoring/*"],
"@kbn/monitoring-collection-plugin": ["x-pack/platform/plugins/private/monitoring_collection"],
"@kbn/monitoring-collection-plugin/*": ["x-pack/platform/plugins/private/monitoring_collection/*"],
"@kbn/monitoring-plugin": ["x-pack/platform/plugins/private/monitoring"],
"@kbn/monitoring-plugin/*": ["x-pack/platform/plugins/private/monitoring/*"],
"@kbn/navigation-plugin": ["src/plugins/navigation"],
"@kbn/navigation-plugin/*": ["src/plugins/navigation/*"],
"@kbn/newsfeed-plugin": ["src/plugins/newsfeed"],
Expand Down
4 changes: 3 additions & 1 deletion x-pack/.i18nrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,9 @@
"platform/packages/private/ml/ui_actions",
"platform/plugins/shared/ml"
],
"xpack.monitoring": ["plugins/monitoring"],
"xpack.monitoring": [
"platform/plugins/private/monitoring"
],
"xpack.observability": "solutions/observability/plugins/observability",
"xpack.observabilityAiAssistant": [
"platform/plugins/shared/observability_solution/observability_ai_assistant",
Expand Down
4 changes: 2 additions & 2 deletions x-pack/.telemetryrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
"output": "plugins/telemetry_collection_xpack/schema/xpack_plugins.json",
"root": "plugins/",
"exclude": [
"plugins/monitoring/server/telemetry_collection/"
"platform/plugins/private/monitoring/server/telemetry_collection/"
]
},
{
"output": "plugins/telemetry_collection_xpack/schema/xpack_monitoring.json",
"root": "plugins/monitoring/server/telemetry_collection/",
"root": "platform/plugins/private/monitoring/server/telemetry_collection/",
"exclude": []
},
{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
When diagnosing issues with UI or alert errors, it can be useful to know the exact query being sent by kibana to elasticsearch.

Rules should have a `fetchData` setup [like this one](/x-pack/plugins/monitoring/server/alerts/nodes_changed_rule.ts#L96). Tracing down will get you to `fetchNodesFromClusterStats`.
Rules should have a `fetchData` setup [like this one](/x-pack/platform/plugins/private/monitoring/server/alerts/nodes_changed_rule.ts#L96). Tracing down will get you to `fetchNodesFromClusterStats`.

Just before the `esClient.search` call you can log out the query parameters using something like:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ docker run --name metricbeat \
--pull always --rm \
--hostname=metricbeat \
--publish=5066:5066 \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/metricbeat.yarn.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/metricbeat.yarn.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
docker.elastic.co/beats/metricbeat:master-SNAPSHOT
```

Expand All @@ -65,7 +65,7 @@ docker run --name filebeat \
--hostname=filebeat \
--publish=5067:5067 \
--volume="$(pwd)/.es:/es:ro" \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/filebeat.yarn.yml:/usr/share/filebeat/filebeat.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/filebeat.yarn.yml:/usr/share/filebeat/filebeat.yml:ro" \
docker.elastic.co/beats/filebeat:master-SNAPSHOT
```

Expand All @@ -80,8 +80,8 @@ docker run --name logstash \
--pull always --rm \
--hostname=logstash \
--publish=9600:9600 \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \
--volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \
--volume="$(pwd)/x-pack/platform/plugins/private/monitoring/dev_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \
docker.elastic.co/logstash/logstash:master-SNAPSHOT
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ See the [Kibana Testing guide](https://www.elastic.co/guide/en/kibana/master/dev

We mainly use:
1. Jest unit tests - located in sibling files to the source code
2. [api_integration tests](../../../../test/api_integration/apis/monitoring)
3. [functional tests](../../../../test/functional/apps/monitoring)
2. [api_integration tests](../../../../../../test/api_integration/apis/monitoring)
3. [functional tests](../../../../../../test/functional/apps/monitoring)

The functional and api integration tests are both under a 'Monitoring' description, so you can use `--grep Monitoring` to run only our tests.
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ subgraph Kibana

click Collectors "https://github.com/elastic/kibana/tree/main/src/core/server/metrics/collectors"
click OpsMetricsObservable "https://github.com/elastic/kibana/blob/92a8636f0ff63ab072527574e96e6616327b2ea4/src/core/server/metrics/metrics_service.ts#L32"
click BulkUploader "https://github.com/elastic/kibana/blob/main/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.ts"
click BulkUploader "https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/private/monitoring/server/kibana_monitoring/bulk_uploader.ts"
end

BulkUploader-->|/_monitoring/bulk|ProdElasticsearch
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ When trying to debug why a CPU metric doesn't look the way you expect it to in a

At the time of writing, the code path to get from a system level CPU metric to a utilization percentage looks like this:

1. `node_cpu_metric` set to `node_cgroup_quota_as_cpu_utilization` when cgroup is enabled: [node_detail.js](/x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/node_detail.js#L61-65)
1. `node_cgroup_quota_as_cpu_utilization` defined as a `QuotaMetric` against `cpu.cfs_quota_micros`: [metrics.ts](/x-pack/plugins/monitoring/server/lib/metrics/elasticsearch/metrics.ts#L798-801)
1. `QuotaMetric` tries to produce a ratio of usage to quota, but returns null when quota isn't a positive number: [quota_metric.ts](/x-pack/plugins/monitoring/server/lib/metrics/classes/quota_metric.ts#L79-80)
1. `node_cpu_metric` set to `node_cgroup_quota_as_cpu_utilization` when cgroup is enabled: [node_detail.js](/x-pack/platform/plugins/private/monitoring/server/routes/api/v1/elasticsearch/node_detail.js#L61-65)
1. `node_cgroup_quota_as_cpu_utilization` defined as a `QuotaMetric` against `cpu.cfs_quota_micros`: [metrics.ts](/x-pack/platform/plugins/private/monitoring/server/lib/metrics/elasticsearch/metrics.ts#L798-801)
1. `QuotaMetric` tries to produce a ratio of usage to quota, but returns null when quota isn't a positive number: [quota_metric.ts](/x-pack/platform/plugins/private/monitoring/server/lib/metrics/classes/quota_metric.ts#L79-80)

So it's important to be aware of the `monitoring.ui.container.elasticsearch.enabled` setting, which defaults to `true` on cloud.elastic.co.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
If the stack monitoring UI isn't showing data for any cluster or missing key metrics, it may first be useful to survey the available data.

If troubleshooting a cluster with a version >= 8.3.1, the [Stack Monitoring health API](https://github.com/elastic/kibana/tree/main/x-pack/plugins/monitoring/server/routes/api/v1/_health) is the recommended way to get an overview of the available data and possible metrics collection issues.
If troubleshooting a cluster with a version >= 8.3.1, the [Stack Monitoring health API](https://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/private/monitoring/server/routes/api/v1/_health) is the recommended way to get an overview of the available data and possible metrics collection issues.
The API is included in the [support-diagnostics utility](https://github.com/elastic/support-diagnostics) so if a bundle is provided in the issue you're working on, the API response would already be available in `kibana_stack_monitoring_health.json`. Otherwise, one can ask for the API response instead of the raw queries.

If troubleshooting an older version, the following queries would be good starters:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@

module.exports = {
preset: '@kbn/test',
rootDir: '../../..',
roots: ['<rootDir>/x-pack/plugins/monitoring'],
coverageDirectory: '<rootDir>/target/kibana-coverage/jest/x-pack/plugins/monitoring',
rootDir: '../../../../..',
roots: ['<rootDir>/x-pack/platform/plugins/private/monitoring'],
coverageDirectory:
'<rootDir>/target/kibana-coverage/jest/x-pack/platform/plugins/private/monitoring',
coverageReporters: ['text', 'html'],
collectCoverageFrom: [
'<rootDir>/x-pack/plugins/monitoring/{common,public,server}/**/*.{js,ts,tsx}',
'<rootDir>/x-pack/platform/plugins/private/monitoring/{common,public,server}/**/*.{js,ts,tsx}',
],
};
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* 2.0.
*/

// From x-pack/plugins/monitoring/public/lib/elasticsearch_settings/enabler.js
// From x-pack/platform/plugins/private/monitoring/public/lib/elasticsearch_settings/enabler.js
export class Enabler {
http: any;
updateModel: any;
Expand Down
Loading
Loading