[META] Search streams using Apache Arrow and Flight

[rishabhmaurya]

Please describe the end goal of this project

• In-memory columnar representation of any intermediate results from search
• • Data adjacency for sequential access (scans)
• • O(1) (constant-time) random access
• • SIMD and vectorization-friendly
• • Relocatable without “pointer swizzling”, allowing for true zero-copy access in shared memory.
• Interoperable representation of columnar data to be used across different engines like sharing between opensearch and datafusion, which is a rust based engine.
• RPC using bidirectional streams: making use of GRPC bidirectional streams handling backpressure from the client in realtime and producing batches of records on demand. Used both for internode communication (between data nodes and cordinator) as well as communication with end client.

Use cases

• Optimize memory overhead, cpu utilization and performance for -
• • Search pagination API
• • Aggregation (more details to follow) .
• • JOINs in Opensearch.

Apache Arrow will serve as a library for in-memory columnar representation on any transient results used for retrieval in these use cases. Arrow Flight to be used for stream RPC.

Supporting References

JOINs RFC making use of this integration - #15185

Issues

• Library changes containing POJOs and Arrow vector APIs - Library changes for Apache Arrow integration #16691

• OpenSearch basic Arrow Flight server and client implementation as a separate module. [Feature Request] Arrow Flight server and client bootstrap logic #16963

• • Integration with Tasks API for all stream producers. [Feature Request] Arrow Flight server and client bootstrap logic #16963
• • Integration with server module via plugin interface [Feature Request] Arrow Flight server and client bootstrap logic #16963
• • TLS integration for Flight server and client - [Feature Request] Handle certificate hot reload for Flight server #16965
• • Security plugin changes to expose parameters to build SslContext using SecureTransportSettingsProvider [FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin security#5011
• • Revisit defaults for server, client and allocator configs after benchmarks [Feature Request] Tune up server config for Flight server #16966

• General purpose FlightProducer supporting basic getStream() and getFlightInfo() APIs. [Feature Request] Base Flight producer for FlightServer #17065

• • ProxyStreamProducer acting as a proxy stream connecting the right data node holding the stream for a given ticket to the client.

• Stream cancellation, error handling and renewal.

• Metrics & Troubleshooting

• Admission control

• Integration tests

• Benchmark - compare against performance of scroll API.

• Documentation

Related component

Search

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[jngz-es]

This feature is awesome! We have one more use case from OpenSearch ML opensearch-project/ml-commons#3328 to address streaming model/agent prediction especially for LLM calls.

[rishabhmaurya]

@jngz-es https://huggingface.co/docs/datasets/en/about_arrow this is in context of LLM datasets and how arrow format can help.