Code

Clickhouse #1097

Code

Merged

sh-rp merged 165 commits intodeveldlt-hub/dlt:develfrom 1055-implement-clickhouse-destinationdlt-hub/dlt:1055-implement-clickhouse-destinationCopy head branch name to clipboardApr 26, 2024

+2,213-94

Commits

Commits on Mar 6, 2024

• Merge branch 'devel' of github.com:dlt-hub/dlt into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  2d72134

Commits on Mar 8, 2024

• Add clickhouse driver dependency #1055

  Show description for 1f28952
  Pipboyguy

  committed
  1f28952

Commits on Mar 9, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  dc5d4b0

• Preliminary wireframe #1055

  Show description for d736dee
  Pipboyguy

  committed
  d736dee

Commits on Mar 11, 2024

• Update preliminary Clickhouse configurations #1055

  Show description for 0407ab8
  Pipboyguy

  committed
  0407ab8

• Format #1055

  Show description for d503c14
  Pipboyguy

  committed
  d503c14

Commits on Mar 12, 2024

• Merge

  Show description for 9b3ef87
  Pipboyguy

  committed
  9b3ef87

• Finalize wireframing #1055

  Show description for ed218e5
  Pipboyguy

  committed
  ed218e5

• Wireframe ClickhouseSqlClient #1055

  Show description for 81091c7
  Pipboyguy

  committed
  81091c7

• Refactor Clickhouse SqlClient wireframing and update capabilities #1055

  Show description for 7b7edff
  Pipboyguy

  committed
  7b7edff

Commits on Mar 13, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  94ca2a9

• Update error messages and transaction capability in Clickhouse

  Show description for 5f55bab
  Pipboyguy

  committed
  5f55bab

• Update Clickhouse configuration and factory

  Show description for c9fb02e
  Pipboyguy

  committed
  c9fb02e

Commits on Mar 14, 2024

• Update

  Show description for f53ca52
  Pipboyguy

  committed
  f53ca52

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  0df32f5

• Update identifier escaping logic #1055

  Show description for c1f106b
  Pipboyguy

  committed
  c1f106b

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  29ed8f6

Commits on Mar 15, 2024

• Refactor ClickhouseSqlClient for better error handling #1055

  Show description for 6fc750f
  Pipboyguy

  committed
  6fc750f

Commits on Mar 16, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  14d39e4

• Refine clickhouse destination and basic adapter #1055

  Show description for dbad9b1
  Pipboyguy

  committed
  dbad9b1

• Finish ClickhouseClient

  Show description for 16df851
  Pipboyguy

  committed
  16df851

Commits on Mar 17, 2024

• Add escape_clickhouse_literal function for Clickhouse

  Show description for c134cad
  Pipboyguy

  committed
  c134cad

• Add "insert_values" to supported loader file formats

  Show description for 2fa644c
  Pipboyguy

  committed
  2fa644c

• Add wei to "from_db_type"

  Show description for 0af45b9
  Pipboyguy

  committed
  0af45b9

• Improve Clickhouse loader code and update comments #1055

  Show description for de66efa
  Pipboyguy

  committed
  de66efa

Commits on Mar 18, 2024

• Preliminary CH tests and utility module

  Show description for 7e0d508
  Pipboyguy

  committed
  7e0d508

• Refactor Clickhouse utilities and update tests

  Show description for 57ceeee
  Pipboyguy

  committed
  57ceeee

• Refactor URL conversion and staging for Clickhouse

  Show description for 6cdf086
  Pipboyguy

  committed
  6cdf086

• Tests don't pass, but they are there #1055

  Show description for 6606f5e
  Pipboyguy

  committed
  6606f5e

Commits on Mar 19, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Show description for 383290d
  Pipboyguy

  committed
  383290d

• Fix poetry collision #1055

  Show description for 45bafa7
  Pipboyguy

  committed
  45bafa7

• Swap engine/primary key clause ordering #1055

  Show description for 29b5a07
  Pipboyguy

  committed
  29b5a07

• Pass basic tests #1055

  Show description for c9b1ae4
  Pipboyguy

  committed
  c9b1ae4

• Add Clickhouse tests and improve its configuration handling

  Show description for 1e96a78
  Pipboyguy

  committed
  1e96a78

Commits on Mar 20, 2024

• Resove merge conflict

  Pipboyguy

  committed
  2087b37

• Format and Lint

  Show description for ab50287
  Pipboyguy

  committed
  ab50287

Commits on Mar 22, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  5d768fa

Commits on Mar 23, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  9d905ec

• Improve DRYness #1055

  Show description for 86685fd
  Pipboyguy

  committed
  86685fd

• Remove old comment #1055

  Show description for e702d8a
  Pipboyguy

  committed
  e702d8a

• Update pyproject.toml #1055

  Show description for 83b3812
  Pipboyguy

  committed
  83b3812

Commits on Mar 25, 2024

• Fix secure connection settings #1055

  Show description for d26183b
  Pipboyguy

  committed
  d26183b

• Minor config parsing amendments #1055

  Show description for e596ae3
  Pipboyguy

  committed
  e596ae3

• Fix ssl connection (correct port) #1055

  Show description for e970a31
  Pipboyguy

  committed
  e970a31

Commits on Mar 26, 2024

• Reuse Athena destination pyformat converter #1055

  Show description for 69a672c
  Pipboyguy

  committed
  69a672c

• Filesystem Auth issues #1055

  Show description for 0cbb657
  Pipboyguy

  committed
  0cbb657

Commits on Mar 27, 2024

• Fix incorrect arguments in render_object_storage_table_function #1055

  Show description for bccab86
  Pipboyguy

  committed
  bccab86

Commits on Mar 30, 2024

• Pass all providers append tests #1055

  Pipboyguy

  committed
  c56f1a4

• Add merge test #1055

  Pipboyguy

  committed
  74e5a6d

Commits on Apr 1, 2024

• Resolved driver parameter substitution issues #1055

  Show description for ff4e214
  Pipboyguy

  committed
  ff4e214

Commits on Apr 2, 2024

• Merge Disposition #1055

  Show description for 5a31ca4
  Pipboyguy

  committed
  5a31ca4

• Fall back to append disposition for merge #1055

  Show description for 75a61eb
  Pipboyguy

  committed
  75a61eb

• Clickhouse CI #1055

  Show description for a4ccd2b
  Pipboyguy

  committed
  a4ccd2b

• CI and merge #1055

  Show description for 200dce5
  Pipboyguy

  committed
  200dce5

• Update active destinations to ClickHouse

  Show description for 7f730c1
  Pipboyguy

  committed
  7f730c1

• Expand Clickhouse dependencies in pyproject.toml

  Show description for dc47c12
  Pipboyguy

  committed
  dc47c12

• Update lock file #1055

  Show description for 34646ad
  Pipboyguy

  committed
  34646ad

Commits on Apr 3, 2024

• Revert back to merge implementation #1055

  Show description for 8bc3c21
  Pipboyguy

  committed
  8bc3c21

• Add default sql test #1055

  Show description for 99e82ff
  Pipboyguy

  committed
  99e82ff

• Support jsonlines

  Show description for 6bded9a
  Pipboyguy

  committed
  6bded9a

• Revert non-applicable changes #1055

  Show description for eca4d2d
  Pipboyguy

  committed
  eca4d2d

Commits on Apr 4, 2024

• Fix 'from_db_type' #1055

  Show description for 092a524
  Pipboyguy

  committed
  092a524

• Remove unused tests #1055

  Show description for 79d9b80
  Pipboyguy

  committed
  79d9b80

• No staging test case #1055

  Show description for c5d8709
  Pipboyguy

  committed
  c5d8709

• Minor changes

  Show description for 594accc
  Pipboyguy

  committed
  594accc

• Refactor Clickhouse loader

  Show description for 5d8a228
  Pipboyguy

  committed
  5d8a228

• WIP

  Show description for d454d8f
  Pipboyguy

  committed
  d454d8f

Commits on Apr 5, 2024

• Remove from standard sql tests

  Show description for fdd052a
  Pipboyguy

  committed
  fdd052a

• Remove unnecessary compression #1055

  Show description for 939bd35
  Pipboyguy

  committed
  939bd35

• Dataset prefix and dataset-table seperator #1055

  Show description for b554693
  Pipboyguy

  committed
  b554693

• Remove DATASET_PREFIX from sql_client.py

  Show description for 8215f44
  Pipboyguy

  committed
  8215f44

• Add clickhouse connect as local fallback #1055

  Show description for 881a0b9
  Pipboyguy

  committed
  881a0b9

• Set settings on local #1055

  Show description for a4904a8
  Pipboyguy

  committed
  a4904a8

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Show description for 89bab97
  Pipboyguy

  committed
  89bab97

• Update lock

  Show description for b44bea3
  Pipboyguy

  committed
  b44bea3

Commits on Apr 6, 2024

• Revert some files back to devel

  Show description for d6309b3
  Pipboyguy

  committed
  d6309b3

• Remove redundant merge logic #1055

  Show description for dfc274d
  Pipboyguy

  committed
  dfc274d

• Spelling fix

  Show description for 9d0e1ba
  Pipboyguy

  committed
  9d0e1ba

• Don't synthesise CH credentials __init__. #1055

  Show description for 9f41edd
  Pipboyguy

  committed
  9f41edd

• Optimize import and type hinting in Clickhouse factory #1055

  Show description for 584ebe7
  Pipboyguy

  committed
  584ebe7

Commits on Apr 8, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  853c353

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  1c69a26

• Revert back to temp table

  Show description for 32d3f62
  Pipboyguy

  committed
  32d3f62

• Refactor Clickhouse to ClickHouse for consistency

  Show description for 6cb4ee8
  Pipboyguy

  committed
  6cb4ee8

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Show description for 5004b5a
  Pipboyguy

  committed
  5004b5a

• Support compression codec for azure and local #1055

  Show description for a646118
  Pipboyguy

  committed
  a646118

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  350cae3

• Merge

  Show description for 6fee65e
  Pipboyguy

  committed
  6fee65e

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  95caba7

Commits on Apr 9, 2024

• Fix table-name separator config resolution #1055

  Show description for 7c0ac80
  Pipboyguy

  committed
  7c0ac80

• Format

  Show description for 411ff77
  Pipboyguy

  committed
  411ff77

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  7fe0a91

• Set compression parameter for local #1055

  Show description for e30ce6d
  Pipboyguy

  committed
  e30ce6d

• Set compression method to 'auto' for s3 table function #1055

  Show description for 3c89b3b
  Pipboyguy

  committed
  3c89b3b

• Typing

  Show description for df7fcf2
  Pipboyguy

  committed
  df7fcf2

• Typing

  Show description for 743cc05
  Pipboyguy

  committed
  743cc05

Commits on Apr 10, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  76a6bcb

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  5d327f3

• Initial draft doc

  Show description for 3f20381
  Pipboyguy

  committed
  3f20381

• auto compression for parquet, detects compression of local files

  rudolfix

  committed
  463ca1d

• fixes has_dataset, recognizes more exceptions

  rudolfix

  committed
  e7e5925

• fixes some tests

  rudolfix

  committed
  871aa4a

• aligns clickhouse config with dataclasses

  rudolfix

  committed
  8ed4919

Commits on Apr 11, 2024

• Merge remote-tracking branch 'origin/1055-implement-clickhouse-destination' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  b13942b

• Remove empty dataset default #1055

  Show description for dce1967
  Pipboyguy

  committed
  dce1967

Commits on Apr 12, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  68ac747

• Update clickhouse configuration and docs sidebar

  Show description for d1c7cde
  Pipboyguy

  committed
  d1c7cde

Commits on Apr 13, 2024

• Clickhouse docs #1055

  Show description for 66fbc0e
  Pipboyguy

  committed
  66fbc0e

Commits on Apr 14, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Show description for 1ab55a1
  Pipboyguy

  committed
  1ab55a1

Commits on Apr 15, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  5ff769c

• Don't use Jinja #1055

  Show description for 077455d
  Pipboyguy

  committed
  077455d

Commits on Apr 16, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Show description for 8d5bb3e
  Pipboyguy

  committed
  8d5bb3e

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Show description for 767def4
  sh-rp

  committed
  767def4

• udpate clickhouse workflow file

  sh-rp

  committed
  65bf250

• add missing secrets to clickhouse workflow

  sh-rp

  committed
  e61f681

• Merge remote-tracking branch 'origin/1055-implement-clickhouse-destination' into 1055-implement-clickhouse-destination

  Show description for 974d150
  Pipboyguy

  committed
  974d150

• Add test for clickhouse config settings #1055

  Show description for 027d52a
  Pipboyguy

  committed
  027d52a

• Set experimental session in DSN #1055

  Show description for ecfe173
  Pipboyguy

  committed
  ecfe173

Commits on Apr 17, 2024

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Show description for 9cd4f7f
  Pipboyguy

  committed
  9cd4f7f

• Update data mapping and test for ClickHouse

  Show description for 6ca11c8
  Pipboyguy

  committed
  6ca11c8

• Revert previous

  Show description for 0cf5f63
  Pipboyguy

  committed
  0cf5f63

• Fix table aliasing issue

  Show description for e5500ce
  Pipboyguy

  committed
  e5500ce

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  4edeec8

Commits on Apr 18, 2024

• remove additional clickhouse destinations from test setup

  sh-rp

  committed
  8830f3e

• fix lockfile

  sh-rp

  committed
  91c257d

• fix merging

  Show description for 9bda629
  sh-rp

  committed
  9bda629

• slightly clean up clickhouse load job

  sh-rp

  committed
  f9e2920

• Merge branch 'refs/heads/devel' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  23800b8

• fix merge job a bit more

  sh-rp

  committed
  ebe0289

• Refactor key table clause generation in ClickHouse

  Show description for 6fd243e
  Pipboyguy

  committed
  6fd243e

• Merge remote-tracking branch 'origin/1055-implement-clickhouse-destination' into 1055-implement-clickhouse-destination

  Show description for dcb6655
  Pipboyguy

  committed
  dcb6655

Commits on Apr 19, 2024

• fixes a bunch of tests

  sh-rp

  committed
  766ecd2

• simplify clickhouse load job a bit

  sh-rp

  committed
  dc3cb76

• add sentinel table for dataset existence check

  sh-rp

  committed
  9ae6979

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Show description for 0a26798
  sh-rp

  committed
  0a26798

• post merge lockfile update

  sh-rp

  committed
  56c2b9f

• add support for scd2

  sh-rp

  committed
  1525e73

• add correct high_ts for clickhouse

  sh-rp

  committed
  15855d7

Commits on Apr 22, 2024

• remove corelated query from scd2 implementation

  sh-rp

  committed
  544a0a4

• fix merge sql for clickhouse

  sh-rp

  committed
  cb9f35b

• fix merge tests

  sh-rp

  committed
  4929bd1

• some further fixes

  sh-rp

  committed
  6badbcc

• fix athena tests

  sh-rp

  committed
  7950642

• disable dbt for now

  sh-rp

  committed
  c98b8b2

Commits on Apr 23, 2024

• smaller changes

  sh-rp

  committed
  89f30d0

• add clickhouse adapter tests, update small part of docs and correct imports

  sh-rp

  committed
  02dfb08

• update scd2 sql based on jorrits suggestions

  sh-rp

  committed
  b06bd9b

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  sh-rp

  committed
  1208ccd

• change merge change test to make it pass

  sh-rp

  committed
  0ca6c36

• use text for json in clickhouse

  sh-rp

  committed
  290faa1

• remove some unrelated unneeded stuff

  sh-rp

  committed
  9c8a8b2

• update docs a bit

  sh-rp

  committed
  65c9cec

• fix json to string tests and implementation

  sh-rp

  committed
  04f357a

Commits on Apr 24, 2024

• move gcp access credentials into proper config

  sh-rp

  committed
  d2231ef

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Show description for 989d93c
  sh-rp

  committed
  989d93c

• Add GCS Clickhouse staging docs #1055

  Show description for 78e4c56
  Pipboyguy

  committed
  78e4c56

• Merge remote-tracking branch 'origin/1055-implement-clickhouse-destination' into 1055-implement-clickhouse-destination

  Pipboyguy

  committed
  e77a160

• Add http_port to docs #1055

  Show description for 1c8e4ef
  Pipboyguy

  committed
  1c8e4ef

Commits on Apr 25, 2024

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  sh-rp

  committed
  8e04b81

• fix import after merge

  sh-rp

  committed
  a8887c4

• small changes to the docs

  sh-rp

  committed
  2c21e19

• tolerate rounding errors when loading from jsonl

  sh-rp

  committed
  7627945

• Merge branch 'devel' into 1055-implement-clickhouse-destination

  Show description for 4d5db26
  sh-rp

  committed
  4d5db26

• post devel merge fix

  sh-rp

  committed
  13f4b1c

Commits on Apr 26, 2024

• remove unneeded stuff from scd2 merge

  sh-rp

  committed
  244ed77