[8.19] Backport data stream settings #129213

joegallo · 2025-06-10T17:21:15Z

This is a single PR to backport the whole feature all at once. That is, it's backporting #126947, #127282, #127417, #127515, #127858, #128269, #128375, #128535, and #128594.

Warning

I did not do the transport version dance, the version that I introduced on this branch is bogus and can't be used as is.

Copilot

Pull Request Overview

This PR backports the entire data streams settings and mappings feature to the 8.19 branch, introducing new APIs and ensuring compatibility in tests.

Adds GET/PUT _data_stream/{name}/_settings endpoints and their transport implementations to view and update data stream settings.
Integrates settings support into existing data stream actions (e.g., GetDataStreamAction) and bumps the transport version.
Covers the new feature with unit and integration tests.

Reviewed Changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/src/main/java/org/elasticsearch/action/datastreams/UpdateDataStreamSettingsAction.java	New action/request/response classes for updating data stream settings
modules/data-streams/src/main/java/org/elasticsearch/datastreams/action/TransportUpdateDataStreamSettingsAction.java	Master-node logic to apply settings to data streams and backing indices
modules/data-streams/src/main/java/org/elasticsearch/datastreams/rest/RestUpdateDataStreamSettingsAction.java / RestGetDataStreamSettingsAction.java	REST handlers for the new settings API
modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamsPlugin.java	Registers the new actions and REST handlers
modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamSettingsIT.java	Integration tests for the settings API
server/src/main/java/org/elasticsearch/TransportVersions.java	Adds a new transport version constant for this feature

Copilot · 2025-06-11T09:31:05Z

.../main/java/org/elasticsearch/datastreams/action/TransportUpdateDataStreamSettingsAction.java

+    UpdateDataStreamSettingsAction.Request,
+    UpdateDataStreamSettingsAction.Response> {
+    private static final Logger logger = LogManager.getLogger(TransportUpdateDataStreamSettingsAction.class);
+    private static final Set<String> APPLY_TO_BACKING_INDICES = Set.of("index.lifecycle.name", IndexSettings.PREFER_ILM);


APPLY_TO_BACKING_INDICES mixes a String with a Setting instance. Use IndexSettings.PREFER_ILM.getKey() so that settingName comparisons work correctly.

Suggested change

private static final Set<String> APPLY_TO_BACKING_INDICES = Set.of("index.lifecycle.name", IndexSettings.PREFER_ILM);

private static final Set<String> APPLY_TO_BACKING_INDICES = Set.of("index.lifecycle.name", IndexSettings.PREFER_ILM.getKey());

I don't know what Copilot is talking about -- Index>settings.PREFER_ILM is a string.

Copilot · 2025-06-11T09:31:05Z

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -238,6 +238,8 @@ static TransportVersion def(int id) {
    public static final TransportVersion ESQL_QUERY_PLANNING_DURATION_8_19 = def(8_841_0_45);
    public static final TransportVersion SEARCH_SOURCE_EXCLUDE_VECTORS_PARAM_8_19 = def(8_841_0_46);
    public static final TransportVersion ML_INFERENCE_MISTRAL_CHAT_COMPLETION_ADDED_8_19 = def(8_841_0_47);
+    public static final TransportVersion SETTINGS_IN_DATA_STREAMS_8_19 = def(8_841_0_48); // TODO NO NO NO NO NO


[nitpick] Remove or refine the placeholder comment 'TODO NO NO NO NO NO' to maintain professional code quality and clarity.

Suggested change

public static final TransportVersion SETTINGS_IN_DATA_STREAMS_8_19 = def(8_841_0_48); // TODO NO NO NO NO NO

public static final TransportVersion SETTINGS_IN_DATA_STREAMS_8_19 = def(8_841_0_48); // TODO: Verify the implementation and ensure compatibility with data stream settings.

Copilot · 2025-06-11T09:31:06Z

...streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamSettingsIT.java

+                .getDataStreamSettingsResponses();
+            assertThat(dataStreamSettingsResponses.size(), equalTo(testDataStreamNames.size()));
+            for (int i = 0; i < testDataStreamNames.size(); i++) {
+                UpdateDataStreamSettingsAction.DataStreamSettingsResponse dataStreamSettingsResponse = dataStreamSettingsResponses.get(0);


Inside the loop over testDataStreamNames, you always fetch index 0. It should be dataStreamSettingsResponses.get(i) to validate each response correctly.

Suggested change

UpdateDataStreamSettingsAction.DataStreamSettingsResponse dataStreamSettingsResponse = dataStreamSettingsResponses.get(0);

UpdateDataStreamSettingsAction.DataStreamSettingsResponse dataStreamSettingsResponse = dataStreamSettingsResponses.get(i);

Oops, I think it's right.

#127417)

)

…ticsearch-specification repository (#128535)

…8.19

joegallo · 2025-06-11T18:17:58Z

b3c559e could be its own PR targeting 8.19. It makes it so that the splash damage from a single test failure (in that yaml file) doesn't result in like 60+ spurious test failures. The remaining docs test failures are real, and I think they're pretty trivial to fix.

docs/reference/data-streams/lifecycle/tutorial-migrate-data-stream-from-ilm-to-dsl.asciidoc

…8.19

…ateAction

…8.19

…thub.com:elastic/elasticsearch into datastreams-settings-and-mappings-overrides-8.19

joegallo added WIP backport :Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team v8.19.0 labels Jun 10, 2025

joegallo assigned masseyke Jun 10, 2025

This comment was marked as resolved.

Sign in to view

lukewhiting requested a review from Copilot June 11, 2025 09:28

Copilot AI reviewed Jun 11, 2025

View reviewed changes

masseyke and others added 10 commits June 11, 2025 10:00

Adding settings to data streams (#126947)

a3fe59f

Using DataStream::getEffectiveSettings (#127282)

ad77f8b

Adding transport actions for getting and updating data stream settings (

8bc8337

#127417)

Fixing DataStream::getEffectiveSettings for component templates (#127515

daa75f5

)

Adding rest actions to get and set data stream settings (#127858)

521dce6

Adding dry_run mode for setting data stream settings (#128269)

a92a55d

Adding prefer_ilm as a whitelisted data stream setting (#128375)

5d8c61e

Making the data stream settngs rest-api-spec consistent with the elas…

a0dae8e

…ticsearch-specification repository (#128535)

Removing the data stream settings feature flag (#128594)

229b943

Collapse the transport versions (and rename)

8708721

joegallo force-pushed the datastreams-settings-and-mappings-overrides-8.19 branch from a2d0422 to 8708721 Compare June 11, 2025 14:00

joegallo added 2 commits June 11, 2025 12:28

Promote this snippet into being a proper TEARDOWN

b3c559e

Merge branch '8.19' into datastreams-settings-and-mappings-overrides-…

2199d54

…8.19

Add "settings": {} where needed to the docs

9872b79

bmorelli25 reviewed Jun 11, 2025

View reviewed changes

docs/reference/data-streams/lifecycle/tutorial-migrate-data-stream-from-ilm-to-dsl.asciidoc Outdated Show resolved Hide resolved

joegallo added 2 commits June 11, 2025 17:32

Remove this line of slashes

3e1948f

Merge branch '8.19' into datastreams-settings-and-mappings-overrides-…

2a788ea

…8.19

joegallo marked this pull request as ready for review June 12, 2025 15:07

masseyke changed the title ~~[8.19] Backport datastreams settings and mappings~~ [8.19] Backport data stream settings Jun 16, 2025

merging 8.19

213e26e

masseyke mentioned this pull request Jun 17, 2025

Claiming a transport version for the data stream settings backport #129560

Merged

masseyke added 6 commits June 17, 2025 10:20

making TransportGetDataStreamSettingsAction a TransportLocalClusterSt…

eaa0160

…ateAction

removing transport version comment due to #129560

06aa0e2

applying DataStreamSettingsIT fix from #129564

a15da87

removing unneccessary use of MasterNodeReadRequest

b847cc8

Merge branch '8.19' into datastreams-settings-and-mappings-overrides-…

d7b0216

…8.19

Merge branch 'datastreams-settings-and-mappings-overrides-8.19' of gi…

8af5789

…thub.com:elastic/elasticsearch into datastreams-settings-and-mappings-overrides-8.19

masseyke approved these changes Jun 17, 2025

View reviewed changes

masseyke merged commit 5b51ddc into 8.19 Jun 17, 2025
23 checks passed

masseyke deleted the datastreams-settings-and-mappings-overrides-8.19 branch June 17, 2025 20:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[8.19] Backport data stream settings #129213

[8.19] Backport data stream settings #129213

joegallo commented Jun 10, 2025

Uh oh!

This comment was marked as resolved.

Copilot AI left a comment

Uh oh!

Copilot AI Jun 11, 2025

Uh oh!

masseyke Jun 17, 2025

Uh oh!

Copilot AI Jun 11, 2025

Uh oh!

Copilot AI Jun 11, 2025

Uh oh!

masseyke Jun 17, 2025

Uh oh!

joegallo commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	private static final Set<String> APPLY_TO_BACKING_INDICES = Set.of("index.lifecycle.name", IndexSettings.PREFER_ILM);
	private static final Set<String> APPLY_TO_BACKING_INDICES = Set.of("index.lifecycle.name", IndexSettings.PREFER_ILM.getKey());

	public static final TransportVersion SETTINGS_IN_DATA_STREAMS_8_19 = def(8_841_0_48); // TODO NO NO NO NO NO
	public static final TransportVersion SETTINGS_IN_DATA_STREAMS_8_19 = def(8_841_0_48); // TODO: Verify the implementation and ensure compatibility with data stream settings.

	UpdateDataStreamSettingsAction.DataStreamSettingsResponse dataStreamSettingsResponse = dataStreamSettingsResponses.get(0);
	UpdateDataStreamSettingsAction.DataStreamSettingsResponse dataStreamSettingsResponse = dataStreamSettingsResponses.get(i);

[8.19] Backport data stream settings #129213

[8.19] Backport data stream settings #129213

Conversation

joegallo commented Jun 10, 2025

Uh oh!

This comment was marked as resolved.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

joegallo commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!