Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix ibis az problems on linux #2135

Merged
merged 4 commits into from
Dec 16, 2024
Merged

fix ibis az problems on linux #2135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b71eb57
try to fix ibis az problems on linux
sh-rp Dec 11, 2024
476487a
remove duckdb certs fix
sh-rp Dec 11, 2024
0a666d1
test explicitely setting transport options
sh-rp Dec 12, 2024
da975d6
sets the ssl curl on a correct connection clone
rudolfix Dec 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/test_destinations.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,6 @@ jobs:
- name: Install dependencies
run: poetry install --no-interaction -E redshift -E postgis -E postgres -E gs -E s3 -E az -E parquet -E duckdb -E cli -E filesystem --with sentry-sdk --with pipeline,ibis -E deltalake -E pyiceberg

- name: enable certificates for azure and duckdb
run: sudo mkdir -p /etc/pki/tls/certs && sudo ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt

- name: Upgrade sqlalchemy
run: poetry run pip install sqlalchemy==2.0.18 # minimum version required by `pyiceberg`

Expand Down
22 changes: 13 additions & 9 deletions dlt/helpers/ibis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import sqlglot
from ibis import BaseBackend, Expr
except ModuleNotFoundError:
raise MissingDependencyException("dlt ibis Helpers", ["ibis"])
raise MissingDependencyException("dlt ibis helpers", ["ibis-framework"])


SUPPORTED_DESTINATIONS = [
Expand Down Expand Up @@ -123,18 +123,22 @@ def create_ibis_backend(
)
from dlt.destinations.impl.duckdb.factory import DuckDbCredentials

# we create an in memory duckdb and create all tables on there
duck = duckdb.connect(":memory:")
# we create an in memory duckdb and create the ibis backend from it
fs_client = cast(FilesystemClient, client)
creds = DuckDbCredentials(duck)
sql_client = FilesystemSqlClient(
fs_client, dataset_name=fs_client.dataset_name, credentials=creds
fs_client,
dataset_name=fs_client.dataset_name,
credentials=DuckDbCredentials(duckdb.connect()),
)

# do not use context manager to not return and close the cloned connection
duckdb_conn = sql_client.open_connection()
# make all tables available here
# NOTE: we should probably have the option for the user to only select a subset of tables here
with sql_client as _:
sql_client.create_views_for_all_tables()
con = ibis.duckdb.from_connection(duck)
sql_client.create_views_for_all_tables()
# why this works now: whenever a clone of connection is made, all SET commands
# apply only to it. old code was setting `curl` on the internal clone of sql_client
# now we export this clone directly to ibis to it works
con = ibis.duckdb.from_connection(duckdb_conn)

return con

Expand Down
Loading