Library changes for Apache Arrow integration #16691

rishabhmaurya · 2024-11-19T21:32:31Z

Description

Library changes for Apache Arrow integration. Lib changes just exposes POJOs for creation of StreamProducer and registering them with StreamManager. StreamProducer in turn exposes BatchedJob which are based on creation and filling Arrow Vectors in a batched manner handling client backpressure. So the arrow APIs exposed are kept minimal, limited to -

  api "org.apache.arrow:arrow-vector:${versions.arrow}"
  api "org.apache.arrow:arrow-format:${versions.arrow}"
  api "org.apache.arrow:arrow-memory-core:${versions.arrow}"

server module will depend on libs:arrow to create and register StreamProducer for search results by populating vectors with a well defined schema for search results.

Future PR will contain a module modules:arrow-flight-rpc, which will actually expose the Arrow Flight server, client and actual logic to create FlightStream. It will be a bulky modules in terms of all its direct and transitive dependencies.

Related Issues

Resolves #16679

Check List

Functionality includes testing.
~~- [ ] API changes companion pull request created, if applicable.~~
~~- [ ] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-11-19T21:36:21Z

Hello!
We have added a performance benchmark workflow that runs by adding a comment on the PR.
Please refer https://github.com/opensearch-project/OpenSearch/blob/main/PERFORMANCE_BENCHMARKS.md on how to run benchmarks on pull requests.

github-actions · 2024-11-19T21:36:37Z

❌ Gradle check result for c4d0735: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-19T22:14:00Z

❌ Gradle check result for fe262f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2024-11-19T23:45:10Z

❌ Gradle check result for 3295caf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2024-11-25T18:11:46Z

❕ Gradle check result for dca6d1f: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

libs/arrow-spi/build.gradle

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicket.java

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2024-11-26T00:35:55Z

✅ Gradle check result for e472e7e: SUCCESS

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2024-11-26T20:46:59Z

✅ Gradle check result for 49b701e: SUCCESS

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicket.java

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicketFactory.java

reta · 2024-11-28T07:48:48Z

A few nits , but LGTM otherwise @rishabhmaurya ! I think the SPI is looking good to start with (certainly will be evolving but this is fine).

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2024-11-28T19:02:59Z

✅ Gradle check result for 7ef6dc3: SUCCESS

rishabhmaurya · 2024-11-28T19:13:25Z

@reta I agree! StreamReader is the one which will evolve most, its very minimal right now. I appreciate your feedbacks in improving the APIs, looks a lot better now. I'm looking forward to your reviews for the next PR related to server bootstrap logic.

reta

@andrross do you want to take a look? thanks!

andrross · 2024-11-29T17:38:54Z

@rishabhmaurya Just curious, how are you going to handle the Guava dependency that Arrow has? The code currently forbids the server module from taking a Guava dependency (presumably to prevent dependency conflicts with plugins that may require a different version?). Will you be able to isolate that dependency to this future arrow-flight-rpc module?

andrross · 2024-11-29T17:46:16Z

@rishabhmaurya Should we backport to 2.x?

* Library changes for arrow integration Signed-off-by: Rishabh Maurya <[email protected]> * Bump guava 32->33 Signed-off-by: Rishabh Maurya <[email protected]> * add support for onCancel and Cancellable for BatchedJob in lib:arrow module Signed-off-by: Rishabh Maurya <[email protected]> * address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * Move StreamTicket to an interface Signed-off-by: Rishabh Maurya <[email protected]> * remove jackson dependencies Signed-off-by: Rishabh Maurya <[email protected]> * make sl4j runtime only Signed-off-by: Rishabh Maurya <[email protected]> * introduce factory for stream ticket Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> --------- Signed-off-by: Rishabh Maurya <[email protected]> (cherry picked from commit 6d3fd37) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

rishabhmaurya · 2024-11-29T19:37:28Z

how are you going to handle the Guava dependency that Arrow has? The code currently forbids the server module from taking a Guava dependency (presumably to prevent dependency conflicts with plugins that may require a different version?). Will you be able to isolate that dependency to this future arrow-flight-rpc module?

@andrross yes, i might be wrong but its mostly arrow flight modules and not arrow vector and memory which needs guava dependencies and thus it will just be isolated to the arrow-flight-rpc module part of my feature branch . I ended up bumping up the version of guava early as part of this PR though.

Should we backport to 2.x?

yes, just added the label.

andrross · 2024-11-29T20:15:00Z

yes, i might be wrong but its mostly arrow flight modules and not arrow vector and memory which needs guava dependencies

@rishabhmaurya I'm guessing Guava comes transitively via the grpc dependencies, which are probably isolated to the flight modules.

* Library changes for arrow integration * Bump guava 32->33 * add support for onCancel and Cancellable for BatchedJob in lib:arrow module * address PR comments * Move StreamTicket to an interface * remove jackson dependencies * make sl4j runtime only * introduce factory for stream ticket * Address PR comments --------- (cherry picked from commit 6d3fd37) Signed-off-by: Rishabh Maurya <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

rishabhmaurya changed the title ~~Library changes for arrow integration~~ Library changes for Apache Arrow integration Nov 19, 2024

rishabhmaurya mentioned this pull request Nov 19, 2024

[META] Search streams using Apache Arrow and Flight #16679

Open

14 tasks

rishabhmaurya self-assigned this Nov 19, 2024

rishabhmaurya added the Search:Performance label Nov 19, 2024

github-actions bot added the Meta Meta issue, not directly linked to a PR label Nov 19, 2024

rishabhmaurya added skip-changelog and removed Meta Meta issue, not directly linked to a PR labels Nov 19, 2024

github-actions bot added the Meta Meta issue, not directly linked to a PR label Nov 19, 2024

rishabhmaurya force-pushed the arrow-integration-lib branch 3 times, most recently from 59c0acd to fe262f2 Compare November 19, 2024 22:05

Library changes for arrow integration

3295caf

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya force-pushed the arrow-integration-lib branch from fe262f2 to 3295caf Compare November 19, 2024 23:38

Bump guava 32->33

d053e4a

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya marked this pull request as ready for review November 20, 2024 00:31

rishabhmaurya requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, jainankitk, kotwanikunal and linuxpi as code owners November 20, 2024 00:31

remove jackson dependencies

dca6d1f

Signed-off-by: Rishabh Maurya <[email protected]>

reta reviewed Nov 25, 2024

View reviewed changes

libs/arrow-spi/build.gradle Outdated Show resolved Hide resolved

reta reviewed Nov 25, 2024

View reviewed changes

libs/arrow-spi/build.gradle Outdated Show resolved Hide resolved

reta reviewed Nov 25, 2024

View reviewed changes

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicket.java Outdated Show resolved Hide resolved

opensearch-ci-bot mentioned this pull request Nov 25, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreStatsIT #14310

Open

make sl4j runtime only

e472e7e

Signed-off-by: Rishabh Maurya <[email protected]>

introduce factory for stream ticket

49b701e

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya force-pushed the arrow-integration-lib branch from cccfa05 to 49b701e Compare November 26, 2024 19:42

reta reviewed Nov 28, 2024

View reviewed changes

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicket.java Outdated Show resolved Hide resolved

reta reviewed Nov 28, 2024

View reviewed changes

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicket.java Outdated Show resolved Hide resolved

reta reviewed Nov 28, 2024

View reviewed changes

libs/arrow-spi/src/main/java/org/opensearch/arrow/spi/StreamTicketFactory.java Outdated Show resolved Hide resolved

Address PR comments

7ef6dc3

Signed-off-by: Rishabh Maurya <[email protected]>

reta approved these changes Nov 28, 2024

View reviewed changes

andrross approved these changes Nov 29, 2024

View reviewed changes

andrross merged commit 6d3fd37 into opensearch-project:main Nov 29, 2024
39 of 42 checks passed

rishabhmaurya added the backport 2.x Backport to 2.x branch label Nov 29, 2024

opensearch-trigger-bot bot mentioned this pull request Nov 29, 2024

[Backport 2.x] Library changes for Apache Arrow integration #16747

Merged

opensearch-ci-bot mentioned this pull request Dec 17, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SegmentReplicationAllocationIT #14327

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Library changes for Apache Arrow integration #16691

Library changes for Apache Arrow integration #16691

rishabhmaurya commented Nov 19, 2024 •

edited

Loading

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 25, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

reta commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

rishabhmaurya commented Nov 28, 2024

reta left a comment

andrross commented Nov 29, 2024

andrross commented Nov 29, 2024

rishabhmaurya commented Nov 29, 2024 •

edited

Loading

andrross commented Nov 29, 2024

Library changes for Apache Arrow integration #16691

Library changes for Apache Arrow integration #16691

Conversation

rishabhmaurya commented Nov 19, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 25, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

reta commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

rishabhmaurya commented Nov 28, 2024

reta left a comment

Choose a reason for hiding this comment

andrross commented Nov 29, 2024

andrross commented Nov 29, 2024

rishabhmaurya commented Nov 29, 2024 • edited Loading

andrross commented Nov 29, 2024

rishabhmaurya commented Nov 19, 2024 •

edited

Loading

rishabhmaurya commented Nov 29, 2024 •

edited

Loading