Allow a dynamic target index in ISM Rollup

[mark-meyer]

I would like to apply a policy to daily log indices that performs a rollup in one of the states before deleting the index. However is seem the target_index requires a string and there is no option to interpolate something like {{ctx.index}} or similar. This makes it impossible to apply this policy to more than one index unless you are okay with rollup overwriting the previous one, which you probably aren't.

Describe the solution you'd like
I would like to specify the target_index field of an ism rollup dynamically based on the name (or some other metadata) of the index being rolled up such as "target_index": "rollup_{{ctx.index}}"

[dbbaughe]

I believe @thalurur just did something similar for snapshot name too.

That being said, correct me if I'm wrong Ravi, but if @mark-meyer applies this policy to daily log indices even if they roll up into the same target index they will not overwrite each other. We do support multiple rollups in the same index. Each index that is managed by the policy would have a rollup job created for itself when the rollup action is created which will use its own rollup ID as a namespace in the target index to handle conflicts. And the rollup ID seems to consist of a hash of the job configuration and the source index which will be unique per daily log index.

Edit: Do we delete these temporary rollup jobs after they finish @thalurur?

[thalurur]

At the moment ISM action will not delete the jobs after they are finished. Rest of the behavior described by Drew is accurate - at the moment the same policy will be creating multiple jobs (based on the source index the policy is being executed on) and all of these jobs writing data to the same target index. And data from each job cannot be touched by other jobs as their namespace includes the rollup job id.

Rollup implementation have some constraints imposed at the moment:

1. If multiple rollup jobs have written data to same target index then data from only one job is picked when searching the rollup index, there is no ability to make the search check through all the jobs at the moment.
2. If search requests consists of multiple rollup indices then the search on rollup indices is failed.

The above makes the only way to rollup data for daily log indices is to create one job that writes to one target index. So the rolled up data can be searched together.

What happens with current policy:
In this case for each new daily log index a new rollup job is created, but all the jobs write to the same target index.
Though the documents from different rollup jobs inside target index are shielded from overwriting, fetching documents from this index is constrained by constraint "1". During search on the target index a single job is picked from multiple jobs and this will prevent ever fetching all the data or user desired job data from target index.

What happens with proposed policy:
If we update the rollup ISM policy to take in a scripted target index, then for each new daily log index a new rollup job is created, writing to a new target index.
In this case since only one job is writing to target index during search there is no issue of which job is created and user can pick which index they want to search from. But, if user want to search across target indices or all target indices its not possible.

We have few options if we need to support either:

• Adjust the constraint "2" - multiple rollup indices cannot be searched together. Adjusting this constraint and making the ISM rollup policy target index scriptable like requested will make it possible to search across all ISM policy created rollup jobs as long as all jobs wrote to different index.
• Adjust the constraint "1" - make it possible to search multiple rollup job data on the same target index, this makes the current policy work
• A combination of both - make it possible to search multiple rollup jobs on same target index and allow multiple rollup indices to be searched in same search request

@mark-meyer Couple of questions:

1. Is there an use case where you would need to search all the data from different source indices that is rolled up through ISM policy together? (I am assuming you do cause since rollup definition is same in policy for all jobs, the data is just spread out temporally)
2. If 1 is yes, do you need the data to be rolled up to different indices or having it single index works as well?

[garlicsauce]

+1

I was trying to setup rollup policy with target index containing date math expression "target_index": "<test-{now/d}>",. Index was created - test-2021.04.28 however rollup job fails with an error saying

Status
Failed: Failed to update mappings of target index [<test-{now/d}>] with rollup job

Without such possibility I guess I need to create some script running every day that would create rollup jobs itself which is much uglier solution. To sum up - it would be nice to be able to setup a rollup job/rollup policy with dynamic target index.