Code

schema contract#594

Code

Merged

rudolfix merged 85 commits intodeveldlt-hub/dlt:develfrom d#/data_contractsdlt-hub/dlt:d#/data_contractsCopy head branch name to clipboardNov 21, 2023

+5,304-2,122

Commits

Commits on Aug 31, 2023

• basic schema freezing

  sh-rp

  committed
  78d6d71

  Copy full SHA for 78d6d71

• small changes

  sh-rp

  committed
  09d8c63

  Copy full SHA for 09d8c63

• temp

  sh-rp

  committed
  aec10b1

  Copy full SHA for aec10b1

• add new schema update mode

  sh-rp

  committed
  9441dcb

  Copy full SHA for 9441dcb

• fix linting errors and one bug

  sh-rp

  committed
  edad4ad

  Copy full SHA for edad4ad

• move freeze code to schema

  sh-rp

  committed
  bef7ea4

  Copy full SHA for bef7ea4

Commits on Sep 4, 2023

• some work on schema evolution modes

  sh-rp

  committed
  fc6f083

  Copy full SHA for fc6f083

• add tests

  sh-rp

  committed
  b36a74f

  Copy full SHA for b36a74f

• small tests change

  sh-rp

  committed
  5659827

  Copy full SHA for 5659827

• Merge branch 'devel' into d#/data_contracts

  Show description for 5343f4b
  sh-rp

  committed
  5343f4b

  Copy full SHA for 5343f4b

Commits on Sep 5, 2023

• small fix

  sh-rp

  committed
  894875f

  Copy full SHA for 894875f

• fix some tests

  sh-rp

  committed
  8bccfe5

  Copy full SHA for 8bccfe5

• add global override for schema evolution

  sh-rp

  committed
  cfd3f64

  Copy full SHA for cfd3f64

• finish implemention of global override

  sh-rp

  committed
  e05855c

  Copy full SHA for e05855c

Commits on Sep 6, 2023

• better tests

  sh-rp

  committed
  3f07127

  Copy full SHA for 3f07127

• carry over schema settings on update

  sh-rp

  committed
  6308369

  Copy full SHA for 6308369

• add tests for single values

  sh-rp

  committed
  ab0b8d7

  Copy full SHA for ab0b8d7

• small changes to tests and code

  sh-rp

  committed
  6e99ed9

  Copy full SHA for 6e99ed9

• fix small error

  sh-rp

  committed
  1a10ec4

  Copy full SHA for 1a10ec4

• add tests for data contract interaction

  sh-rp

  committed
  175bee5

  Copy full SHA for 175bee5

Commits on Sep 7, 2023

• fix tests

  sh-rp

  committed
  18b9341

  Copy full SHA for 18b9341

Commits on Sep 12, 2023

• Merge branch 'devel' into d#/data_contracts

  Show description for 6dcaa7d
  sh-rp

  committed
  6dcaa7d

  Copy full SHA for 6dcaa7d

• some PR work

  sh-rp

  committed
  881d79a

  Copy full SHA for 881d79a

Commits on Sep 13, 2023

• update schema management

  sh-rp

  committed
  e707580

  Copy full SHA for e707580

• fix schema related tests

  sh-rp

  committed
  b3dc41d

  Copy full SHA for b3dc41d

• add nice schema tests

  sh-rp

  committed
  ac8f766

  Copy full SHA for ac8f766

• add docs page

  sh-rp

  committed
  869f278

  Copy full SHA for 869f278

• small test fix

  sh-rp

  committed
  0f22ba0

  Copy full SHA for 0f22ba0

Commits on Sep 14, 2023

• smaller PR fixes

  sh-rp

  committed
  463c447

  Copy full SHA for 463c447

Commits on Sep 17, 2023

• more work

  sh-rp

  committed
  db3f447

  Copy full SHA for db3f447

Commits on Sep 18, 2023

• tests update

  rudolfix

  committed
  c17577a

  Copy full SHA for c17577a

Commits on Sep 25, 2023

• almost there

  sh-rp

  committed
  fb1d224

  Copy full SHA for fb1d224

Commits on Sep 26, 2023

• tmp

  sh-rp

  committed
  6a15fa2

  Copy full SHA for 6a15fa2

• fix freeze tests

  sh-rp

  committed
  d66c2e6

  Copy full SHA for d66c2e6

• Merge branch 'devel' into d#/data_contracts

  Show description for 5e238c5
  sh-rp

  committed
  5e238c5

  Copy full SHA for 5e238c5

• cleanup

  sh-rp

  committed
  bf9da7e

  Copy full SHA for bf9da7e

• Merge branch 'devel' into d#/data_contracts

  Show description for 9ba2496
  sh-rp

  committed
  9ba2496

  Copy full SHA for 9ba2496

• create data contracts page

  sh-rp

  committed
  038d03a

  Copy full SHA for 038d03a

• small cleanup

  sh-rp

  committed
  84384c3

  Copy full SHA for 84384c3

• add pydantic dep to destination tests

  sh-rp

  committed
  44dfb69

  Copy full SHA for 44dfb69

Commits on Sep 28, 2023

• Merge branch 'devel' into d#/data_contracts

  Show description for ece2bfc
  sh-rp

  committed
  ece2bfc

  Copy full SHA for ece2bfc

Commits on Sep 29, 2023

• rename contract settings

  sh-rp

  committed
  7b8f2d2

  Copy full SHA for 7b8f2d2

• rename schema contract dict keys

  sh-rp

  committed
  00c540b

  Copy full SHA for 00c540b

• some work

  sh-rp

  committed
  3ed4630

  Copy full SHA for 3ed4630

• more work...

  sh-rp

  committed
  333217c

  Copy full SHA for 333217c

Commits on Oct 1, 2023

• more work

  sh-rp

  committed
  d69e54d

  Copy full SHA for d69e54d

Commits on Oct 2, 2023

• move checking of new tables into extract function

  sh-rp

  committed
  041da6d

  Copy full SHA for 041da6d

• fix most tests

  sh-rp

  committed
  b72a1a9

  Copy full SHA for b72a1a9

• Merge branch 'devel' into d#/data_contracts

  Show description for f2abadf
  sh-rp

  committed
  f2abadf

  Copy full SHA for f2abadf

• fix linter after merge

  sh-rp

  committed
  2ae36e3

  Copy full SHA for 2ae36e3

• small cleanup

  sh-rp

  committed
  d85e04f

  Copy full SHA for d85e04f

Commits on Oct 10, 2023

• Merge branch 'devel' into d#/data_contracts

  rudolfix

  committed
  1d7be25

  Copy full SHA for 1d7be25

Commits on Oct 17, 2023

• Merge branch 'devel' into d#/data_contracts

  Show description for 96785c4
  sh-rp

  committed
  96785c4

  Copy full SHA for 96785c4

• post merge code updates

  sh-rp

  committed
  302d909

  Copy full SHA for 302d909

• small fixes

  sh-rp

  committed
  4a1fab0

  Copy full SHA for 4a1fab0

Commits on Oct 18, 2023

• some cleanup

  sh-rp

  committed
  903f000

  Copy full SHA for 903f000

• update docs

  sh-rp

  committed
  912dd8b

  Copy full SHA for 912dd8b

Commits on Oct 29, 2023

• Merge branch 'devel' into d#/data_contracts

  rudolfix

  committed
  ef1b10f

  Copy full SHA for ef1b10f

Commits on Nov 1, 2023

• Merge branch 'devel' into d#/data_contracts

  rudolfix

  committed
  256920e

  Copy full SHA for 256920e

• makes bumping version optional in Schema, preserves hashes on replace schema content

  rudolfix

  committed
  168f0da

  Copy full SHA for 168f0da

• extracts on single pipeline schema

  rudolfix

  committed
  760cc43

  Copy full SHA for 760cc43

Commits on Nov 12, 2023

• allows to control relational normalizer descend with send

  rudolfix

  committed
  b645927

  Copy full SHA for b645927

• refactors data contract apply to generate filters instead of actual filtering

  rudolfix

  committed
  8989a75

  Copy full SHA for 8989a75

• detects if bytes string possibly contains pue characters

  rudolfix

  committed
  57842cc

  Copy full SHA for 57842cc

• applies schema contracts in item normalizer, uses binary stream, detects pue to skip decoding

  rudolfix

  committed
  441299f

  Copy full SHA for 441299f

• methods to remove and rename arrow columns, need arrow 12+

  rudolfix

  committed
  fc0eb47

  Copy full SHA for fc0eb47

• implements contracts in extract, fixes issues in apply hints, arrow data filtering still missing

  rudolfix

  committed
  9977227

  Copy full SHA for 9977227

• always uses pipeline schema when extracting

  rudolfix

  committed
  d74242a

  Copy full SHA for d74242a

• returns new items count from buffered write

  rudolfix

  committed
  e9344ee

  Copy full SHA for e9344ee

• bumps pyarrow to 12, temporary removes snowflake extra

  rudolfix

  committed
  e980396

  Copy full SHA for e980396

• Merge branch 'devel' into d#/data_contracts

  rudolfix

  committed
  5e2d131

  Copy full SHA for 5e2d131

Commits on Nov 13, 2023

• fixes arrow imports and normalizer config

  rudolfix

  committed
  3dc4fa5

  Copy full SHA for 3dc4fa5

• fixes normalizer config tests and pipeline state serialization

  rudolfix

  committed
  c76788a

  Copy full SHA for c76788a

• normalizes arrow tables before saving

  rudolfix

  committed
  423a163

  Copy full SHA for 423a163

Commits on Nov 16, 2023

• adds validation and model synth for contracts to pydantic helper

  rudolfix

  committed
  c1c32d6

  Copy full SHA for c1c32d6

• splits extractor into files, improves pydantic validator

  rudolfix

  committed
  c24b643

  Copy full SHA for c24b643

• runs tests on ci with minimal dependencies

  rudolfix

  committed
  2b97a4f

  Copy full SHA for 2b97a4f

• fixes deps in ci workflows

  rudolfix

  committed
  c35ec2d

  Copy full SHA for c35ec2d

• re-adds snowflake connector

  rudolfix

  committed
  340ed3d

  Copy full SHA for 340ed3d

Commits on Nov 18, 2023

• Merge branch 'devel' into d#/data_contracts

  rudolfix

  committed
  35c10b7

  Copy full SHA for 35c10b7

Commits on Nov 19, 2023

• updates pydantic helper

  rudolfix

  committed
  2d260e8

  Copy full SHA for 2d260e8

• improves contract violation exception

  rudolfix

  committed
  aa990f5

  Copy full SHA for aa990f5

• splits source and resource in extract, adds more tests

  rudolfix

  committed
  a6a782b

  Copy full SHA for a6a782b

• temp disable pydantic 1 tests

  rudolfix

  committed
  f4d2cac

  Copy full SHA for f4d2cac

Commits on Nov 20, 2023

• fixes generic type parametrization on 3.8

  rudolfix

  committed
  f556584

  Copy full SHA for f556584