-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support elasticsearch rollover #68
Comments
I just ran into this issues after successfully configuring all of Jaeger to use ES rollover. Using rollover (and also ILM) is just much more sensible when dealing with the ever changing amounts of data that tracing might produce (or might not). |
Considering there also is the Flink based implementation, but which currently lacks ES support altogether (jaegertracing/jaeger-analytics-flink#7). What is the intended way forward around creating the dependencies data? |
There hasn't been much work on the flink job. There are no plans to support ES in the flink project. The whole project hasn't been "productized". It makes sense to add support for rollover here, however it will need some changes how the data are loaded since we cannot read all the data from the read alias. Maybe we could use timestamps from spans instead. |
@pavolloffay while peeking into the code, I expected the only change required is to directly address the read and write alias in the run method (i.e. when env ES_USE_ALIASES is set instead of calling the indexDate method (see: https://github.com/jaegertracing/spark-dependencies/blob/master/jaeger-spark-dependencies-elasticsearch/src/main/java/io/jaegertracing/spark/dependencies/elastic/ElasticsearchDependenciesJob.java#L203) |
Read alias might point to multiple indices for extended period of time e.g. week or two weeks. We cannot load that much data into memory. Also the dependencies reader in Jaeger expects derived dependencies for the current day (or previous). |
@pavolloffay I see, my bad. But thank you for taking the time to think about this issue. But in any case a limit on the number of docs requested from ES makes sense. Even a single day index could contain up to 2 billion docs (ES / Lucene limit) How about simply applying a filter via startTimeMillis when querying jaeger-span-read? Potentially using the terms query build into ES could speed things up even more (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html#query-dsl-terms-lookup) -- this will need to be done in chunks of 65k terms though.... |
This is exactly what I have already proposed. Use the span timestamp for query and then just store the data into dependency daily index like we do now. It does not require the changes on the Jaeger side. |
Yeah - that sounds like simplest approach. Only downside about having a daily dependency index is that is is quite wasteful on the number shards ... but potentially not really an issue when talking about a few days worth of indices. |
@frittentheke I know this is an old issue now but I tried setting
It still seems to be attempting to read an index with a date suffix The container image ID being used seems to be the latest on dockerhub Any help understanding what's going on would be much appreciated, thanks. |
Sorry @lgenasi-gocity for never responding. |
Support elasticearch rollover indices - we need to read data from
jaeger-span-read
and not daily indices.The write could be done to daily indices - we do not support rollover aliases for dependency index at the moment.
The text was updated successfully, but these errors were encountered: