Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flightsql SQL driver #2619

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

mkysel
Copy link
Contributor

@mkysel mkysel commented May 30, 2024

This is a small step in the Apache Arrow / FlightSQL direction. It is the minimal working improvement that is both easy and useful.

FlightSQL Driver: https://pkg.go.dev/github.com/apache/arrow/go/v14/arrow/flight/flightsql/driver#section-readme

I tested it against the new InfluxDBv3 which is flightsql/arrow compatible.

Docs for Influx here: https://www.influxdata.com/blog/two-methods-connecting-influxdb-3-0/

Example config:

input:
  sql_select:
    driver: flightsql
    dsn: flightsql://us-east-1-1.aws.cloud2.influxdata.com:443?bucket-name=get-started&tls=enabled&token=[token]
    table: census
    columns: [ '*' ]

I haven't yet figured out variable binding, but here are some breadcrumbs apache/arrow#40311

@zeroshade
Copy link

There might be benefit for a wider scope and potential if you target ADBC and then just use the FlightSQL driver for it.

That way it can be easily adapted to also allow using any of the ADBC drivers, such as snowflake, with minimal future changes.

Just an idea

@mkysel
Copy link
Contributor Author

mkysel commented Jun 5, 2024

One of the hiccups with this implementation is that not all FlightSQL endpoints support variable/parameter bindings. Those that do, use different dialects that either use $ or ?.

There are no integration tests since I tested this against influxv3 which does not have a docker image yet. I suspect @zeroshade might have ideas on what the easiest integration test might be against.

@zeroshade
Copy link

The open source Dremio docker image is likely the easiest integration test I can think of. Have a look at https://github.com/apache/arrow-adbc/blob/715fe2459a836527602383c50ecd40b8d520bb41/.github/workflows/integration.yml#L106 which is the CI workflow used for the flightsql integration testing for ADBC for examples of how to set it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants