[Feature]: support index_template

[jgournet]

Requirement

As a jaeger admin, we're using Opensearch as the storage.
We have also migrated all our old "template" to "index_template" ( https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html ).

Problem

Problem is that when template AND index_template are mixed together, they do not behave well. One will take over and the other just does not work at all

Proposal

Move to index_template

Open questions

See discussion there on the bug, the error message and the resolution:
https://discuss.elastic.co/t/connecting-from-jaeger-tracing-to-elasticsearch/365713

[madmecodes]

is the issue still open, right?

[madmecodes]

Hi @jgournet @mahadzaryab1

Reconfirming the Problem: When template AND index_template are mixed together, they do not behave well. One will take over and the other just does not work at all.

Reproduce the issue:
Current Situation:

1. Both template types exist simultaneously:
   • Legacy template (_template) configuring 3 shards:

   curl http://localhost:9200/_template/jaeger-service?pretty
   {
     "settings": { "index": { "number_of_shards": "3" } }
   }

Index template (_index_template) configuring 5 shards:

curl http://localhost:9200/_index_template/jaeger-service?pretty
{
  "template": { "settings": { "index": { "number_of_shards": "5" } } }
}

When checking actual indices:

curl http://localhost:9200/jaeger-service-*/_settings?pretty
{
  "number_of_shards": "5"  // index_template wins
}

This confirms that when both templates exist:

According to Elasticsearch's documentation and behavior:

• Template Precedence Order:
• When multiple templates match an index, Elasticsearch follows this order:

i.e

• Index templates take precedence over legacy templates
• Templates with higher order values take precedence

The index_template takes precedence
The legacy template is ignored
Could cause confusion for users managing their Elasticsearch setup

Proposed Solution:

Deprecate support for legacy templates?
Add migration documentation?

Anything you want to share s this intended behavior to have support for both the shards?
(Also how shall i tag the maintainers in this repo?)

[jgournet]

To me, ideally, I'd say that when jaeger starts up, it should check the ES version before creating those templates; and create the "right" ones depending on the version.
Otherwise, jaeger could attempt to create _index_template first; and:

• if that fails, create _template
• if that succeeds, delete any jaeger _template