Add ordinal range encode for tsid

[dnhatn]

When a keyword is the primary sort field, we store the starting document of each ordinal instead of blocks of ordinals. By default, this is not enabled if the average number of documents per ordinal is less than 512, as storing block values may be more efficient and safer. Reading a large range of documents—a common pattern in ES|QL—can be more efficient with this approach.

[martijnvg]

I took a first look and this is a great idea to speedup reading the _tsid (and other doc values fields that are primary sort field) without having to enable a doc value skipper.

[martijnvg]

I think in TsdbDocValueBwcTests we should also add a test method that tests ES819TSDBDocValuesFormat without ordinal range encoding to with ordinal range encoding for bwc purposes?

[dnhatn]

I added a test in 6bdfd69

[martijnvg]

I executed the tsdb track with this change and seeing the following results:

|                                                  50th percentile latency | esql-memory-pagefault-rate-by-container-hour |  4307.26        |   3646.82        |   -660.443       |     ms |  -15.33% |
|                                                  50th percentile latency |  esql-memory-pagefault-rate-by-container-day |  3585.47        |   3154.48        |   -430.99        |     ms |  -12.02% |
|                                                  50th percentile latency |               esql-memory-pagefault-rate-day |  3139.99        |   2847.56        |   -292.424       |     ms |   -9.31% |
|                                                  50th percentile latency |              esql-memory-pagefault-rate-hour |  3739.36        |   3480.29        |   -259.076       |     ms |   -6.93% |
|                                                  50th percentile latency |           esql-memory-usage-by-container-day |  1350.7         |   1394.87        |     44.1711      |     ms |   +3.27% |
|                                                  50th percentile latency |           esql-memory-usage-by-container-day |  1350.7         |   1394.87        |     44.1711      |     ms |   +3.27% |
|                                                  50th percentile latency |          esql-memory-usage-by-container-hour |  1637.53        |   1675.15        |     37.6272      |     ms |   +2.30% |
|                                                  50th percentile latency |                        esql-memory-usage-day |   698.187       |    739.367       |     41.1804      |     ms |   +5.90% |
|                                                  50th percentile latency |                       esql-memory-usage-hour |   984.767       |    996.926       |     12.159       |     ms |   +1.23% |

The p50 latency of rate based queries nicely improved. I think the difference of p50 of other queries is within noise range.

Other details:

|                                                        Median Throughput |                                        index | 63901.2         |  64452.7         |    551.492       | docs/s |   +0.86% |
|                                                               Store size |                                              |     4.04692     |      4.05151     |      0.00459     |     GB |   +0.11% |
|                                                    tsdb _tsid doc values |                                              |     9.84666     |      3.2097      |     -6.63696     |     MB |  -67.40% |

Indexing throughput is similar, while tsid disk usage got much smaller (compared to total store size this is neglectable).

Full rally compare:
comparison-2.txt

[martijnvg]

Thanks Nhat! LGTM

[kkrik-es]

Is it possible that we've already inserted maxDoc in the loop above?

[dnhatn]

We should not, because startDocs should be at most maxDoc - 1.

[kkrik-es]

Nit: toIntExact not needed?

[dnhatn]

thanks - I removed it in 8dc9641.

[kkrik-es]

Thanks Nhat!

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Hi @dnhatn, I've created a changelog YAML for you.