Code

LanceDB Destination#1375

Code

Merged

sh-rp merged 179 commits intodeveldlt-hub/dlt:develfrom 1370-lancedb-destinationdlt-hub/dlt:1370-lancedb-destinationCopy head branch name to clipboardJun 27, 2024

+2,121-208

Commits

Commits on May 16, 2024

• Added lancedb as an optional dependency

  Show description for 5c35057
  Pipboyguy

  committed
  5c35057

• Added lancedb to dependencies in test workflow

  Show description for f4450a7
  Pipboyguy

  committed
  f4450a7

Commits on May 17, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  0808bb2

• Add initial capabilities for LanceDB destination

  Show description for 5380892
  Pipboyguy

  committed
  5380892

• Added new lancedb_adapter

  Show description for 0f40ffa
  Pipboyguy

  committed
  0f40ffa

• Added LanceDB factory in destinations implementation

  Show description for d34ab6f
  Pipboyguy

  committed
  d34ab6f

• Added LanceDB client configuration with embedding details

  Show description for 0f3d57f
  Pipboyguy

  committed
  0f3d57f

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  69e1daa

Commits on May 20, 2024

• Added LanceDB Client with data load and schema management functionalities

  Show description for c51a9bc
  Pipboyguy

  committed
  c51a9bc

Commits on May 29, 2024

• Merge remote-tracking branch 'origin/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  c49ee2b

• Lockfile

  Show description for 389d06b
  Pipboyguy

  committed
  389d06b

Commits on May 30, 2024

• Wireframe LanceDB client implementation

  Show description for 1d9a072
  Pipboyguy

  committed
  1d9a072

• Add abstract methods

  Show description for 8c0de4c
  Pipboyguy

  committed
  8c0de4c

Commits on May 31, 2024

• Enhance LanceDB client with additional functionality

  Show description for d6d02b2
  Pipboyguy

  committed
  d6d02b2

Commits on Jun 1, 2024

• Add tests and GitHub workflow for LanceDB destination

  Show description for ada3ebe
  Pipboyguy

  committed
  ada3ebe

Commits on Jun 2, 2024

• Update Python version to 3.11.x in GitHub workflow

  Show description for 5d961b5
  Pipboyguy

  committed
  5d961b5

Commits on Jun 3, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  c3cad2f

• Refactor and cleanup LanceDBClient and LoadLanceDBJob classes

  Show description for f97256e
  Pipboyguy

  committed
  f97256e

• Refactor load tests in lancedb/utils.py and add test for LanceDB model inference

  Show description for 1b68391
  Pipboyguy

  committed
  1b68391

• Added functionality to infer LanceDB model from data and refactored name for reserved fields

  Show description for ae70f97
  Pipboyguy

  committed
  ae70f97

Commits on Jun 4, 2024

• Remove storage options

  Show description for 2222a88
  Pipboyguy

  committed
  2222a88

• Refactor test pipeline and implement lancedb_adapter in LanceDBClient

  Show description for f819c25
  Pipboyguy

  committed
  f819c25

• Add schema argument to LoadLanceDBJob function

  Show description for 8bbd515
  Pipboyguy

  committed
  8bbd515

• Format

  Show description for fb7565e
  Pipboyguy

  committed
  fb7565e

• Refactor LanceDB related code and increase type hint coverage

  Show description for 4c73541
  Pipboyguy

  committed
  4c73541

• Refactor LanceDB client and tests, enhance DB type mapping

  Show description for bfcc8bb
  Pipboyguy

  committed
  bfcc8bb

• Refactor code to improve readability by reducing line breaks

  Show description for 4827798
  Pipboyguy

  committed
  4827798

• Refactor LanceDB client code by adding schema_conversion and utils modules

  Show description for 53c0b0d
  Pipboyguy

  committed
  53c0b0d

• Remove redundant variables in lancedb_client.py

  Show description for 2cfbdb4
  Pipboyguy

  committed
  2cfbdb4

• Refactor code to improve readability and move environment variable set function to utils.py

  Show description for 092fcf0
  Pipboyguy

  committed
  092fcf0

• Refactor LanceDB client implementation and error handling

  Show description for b1783b2
  Pipboyguy

  committed
  b1783b2

• Refactor code for better readability and add type ignore comments

  Show description for 08dcea1
  Pipboyguy

  committed
  08dcea1

Commits on Jun 6, 2024

• Added lancedb as an optional dependency

  Show description for ad43996
  Pipboyguy

  committed
  ad43996

• Added lancedb to dependencies in test workflow

  Show description for c225916
  Pipboyguy

  committed
  c225916

• Add initial capabilities for LanceDB destination

  Show description for cfc4038
  Pipboyguy

  committed
  cfc4038

• Added new lancedb_adapter

  Show description for 7e1b651
  Pipboyguy

  committed
  7e1b651

• Added LanceDB factory in destinations implementation

  Show description for 892c4f8
  Pipboyguy

  committed
  892c4f8

• Added LanceDB client configuration with embedding details

  Show description for b4e8ad5
  Pipboyguy

  committed
  b4e8ad5

• Added LanceDB Client with data load and schema management functionalities

  Show description for 605ec89
  Pipboyguy

  committed
  605ec89

• Wireframe LanceDB client implementation

  Show description for 8cd3208
  Pipboyguy

  committed
  8cd3208

• Add abstract methods

  Show description for d4a9314
  Pipboyguy

  committed
  d4a9314

• Enhance LanceDB client with additional functionality

  Show description for 99fb712
  Pipboyguy

  committed
  99fb712

• Add tests and GitHub workflow for LanceDB destination

  Show description for e024925
  Pipboyguy

  committed
  e024925

• Update Python version to 3.11.x in GitHub workflow

  Show description for c1e84b5
  Pipboyguy

  committed
  c1e84b5

• Refactor and cleanup LanceDBClient and LoadLanceDBJob classes

  Show description for fe0cfbe
  Pipboyguy

  committed
  fe0cfbe

• Refactor load tests in lancedb/utils.py and add test for LanceDB model inference

  Show description for 32fd5c8
  Pipboyguy

  committed
  32fd5c8

• Added functionality to infer LanceDB model from data and refactored name for reserved fields

  Show description for 1eea62a
  Pipboyguy

  committed
  1eea62a

• Remove storage options

  Show description for eb05fcd
  Pipboyguy

  committed
  eb05fcd

• Refactor test pipeline and implement lancedb_adapter in LanceDBClient

  Show description for d560269
  Pipboyguy

  committed
  d560269

• Add schema argument to LoadLanceDBJob function

  Show description for 80eef96
  Pipboyguy

  committed
  80eef96

• Format

  Show description for 1414ded
  Pipboyguy

  committed
  1414ded

• Refactor LanceDB related code and increase type hint coverage

  Show description for 033c981
  Pipboyguy

  committed
  033c981

• Refactor LanceDB client and tests, enhance DB type mapping

  Show description for c655c8a
  Pipboyguy

  committed
  c655c8a

• Refactor code to improve readability by reducing line breaks

  Show description for 1815da4
  Pipboyguy

  committed
  1815da4

• Refactor LanceDB client code by adding schema_conversion and utils modules

  Show description for 32478ae
  Pipboyguy

  committed
  32478ae

• Remove redundant variables in lancedb_client.py

  Show description for c4bd4f2
  Pipboyguy

  committed
  c4bd4f2

• Refactor code to improve readability and move environment variable set function to utils.py

  Show description for 6c74cc5
  Pipboyguy

  committed
  6c74cc5

• Refactor LanceDB client implementation and error handling

  Show description for 9e5d6db
  Pipboyguy

  committed
  9e5d6db

• Refactor code for better readability and add type ignore comments

  Show description for fb6974d
  Pipboyguy

  committed
  fb6974d

• Dependency Versioning

  Show description for ee239e5
  Pipboyguy

  committed
  ee239e5

• Merge remote-tracking branch 'origin/1370-lancedb-destination' into 1370-lancedb-destination

  Show description for 42e32d9
  Pipboyguy

  committed
  42e32d9

• Remove unnecessary dependencies and update lancedb and pylance versions

  Show description for 827ebd3
  Pipboyguy

  committed
  827ebd3

• Silence mypy warnings

  Show description for 71e3579
  Pipboyguy

  committed
  71e3579

• Revert mypy ignores

  Show description for d5b8cda
  Pipboyguy

  committed
  d5b8cda

• Revert mypy ignores

  Show description for 3ea0a3a
  Pipboyguy

  committed
  3ea0a3a

• Fix versioning with 3.8

  Show description for 395e1b7
  Pipboyguy

  committed
  395e1b7

• Fix versioning

  Show description for ec2774d
  Pipboyguy

  committed
  ec2774d

• Update default URI and dataset separator in LanceDB configuration

  Show description for 1591a1f
  Pipboyguy

  committed
  1591a1f

Commits on Jun 7, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  7f3c772

Commits on Jun 8, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  a727cf3

• Refactor LanceDB typemapper with timestamp and decimal precision adjustments

  Show description for 66d1cc7
  Pipboyguy

  committed
  66d1cc7

• Updated method for retrieving sentinel table name

  Show description for 21bc285
  Pipboyguy

  committed
  21bc285

• Remove redundant table normalisation for version_table_name

  Show description for 7fadbe5
  Pipboyguy

  committed
  7fadbe5

• Refactor LanceDB functionalities and improve handling of optional embedding fields

  Show description for bda6123
  Pipboyguy

  committed
  bda6123

• Refactor LanceDBClient and update parameter defaults in schema.py

  Show description for 74ac9f3
  Pipboyguy

  committed
  74ac9f3

Commits on Jun 9, 2024

• Added lancedb to default vector configs and improved type annotations in tests.

  Show description for e4b3a8d
  Pipboyguy

  committed
  e4b3a8d

• Return self in enter context manager method

  Show description for 5e8718c
  Pipboyguy

  committed
  5e8718c

Commits on Jun 10, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  310eccf

• Handle FileNotFoundError

  Show description for 6a5db5c
  Pipboyguy

  committed
  6a5db5c

• Replace FileNotFoundError with DestinationUndefinedEntity in lancedb_client.py

  Show description for f4bcfe7
  Pipboyguy

  committed
  f4bcfe7

• Refactor LanceDB client for simplified table name handling

  Show description for b193f38
  Pipboyguy

  committed
  b193f38

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  6e61c7a

• Refactored LanceDB schema creation and storage update processes to pyarrow

  Show description for 4701547
  Pipboyguy

  committed
  4701547

• Remove LanceModels

  Show description for 74080a4
  Pipboyguy

  committed
  74080a4

Commits on Jun 11, 2024

• Ensure 'records' is a list in lancedb_client.py

  Show description for f3ac7e2
  Pipboyguy

  committed
  f3ac7e2

• Refactor code and add batch error handling in lancedb client

  Show description for 173be9e
  Pipboyguy

  committed
  173be9e

• Refactor LanceDB client and schema for improved embedding handling

  Show description for 9d6c6d2
  Pipboyguy

  committed
  9d6c6d2

• Improve error handling and retries in LanceDB client

  Show description for bb165a8
  Pipboyguy

  committed
  bb165a8

• Add error decorator to get_stored_state method in lancedb_client

  Show description for 2adeaea
  Pipboyguy

  committed
  2adeaea

• Change error handling from FileNotFoundError to IndexError

  Show description for ba02ed5
  Pipboyguy

  committed
  ba02ed5

• Refactor lancedb_client.py and add error decorators

  Show description for 432dd76
  Pipboyguy

  committed
  432dd76

• Add configurable read consistency to LanceDB client

  Show description for fa1202e
  Pipboyguy

  committed
  fa1202e

Commits on Jun 12, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Show description for 8963876
  Pipboyguy

  committed
  8963876

• Versioning

  Show description for 26f497e
  Pipboyguy

  committed
  26f497e

• Refactor code for readability and change return type in tests

  Show description for 30cc9a7
  Pipboyguy

  committed
  30cc9a7

• Update queries in lancedb_client to order by insertion date

  Show description for 175b6db
  Pipboyguy

  committed
  175b6db

• Refactor LanceDB client and schema for better table creation and management

  Show description for 719fcfb
  Pipboyguy

  committed
  719fcfb

• Combine "skip" and "append" write dispositions in batch upload

  Show description for aa683da
  Pipboyguy

  committed
  aa683da

• Add schema version hash check in LanceDB client write operations

  Show description for 6483152
  Pipboyguy

  committed
  6483152

• Remove testing code

  Show description for 65ede6f
  Pipboyguy

  committed
  65ede6f

• Refactor return statement in lancedb_client for successful state loads

  Show description for fbc558f
  Pipboyguy

  committed
  fbc558f

• Update lancedb_client.py to improve table handling and embedding fields

  Show description for 8fbae21
  Pipboyguy

  committed
  8fbae21

• Refactor LanceDB schema generation and handle metadata for embedding functions

  Show description for 88bc519
  Pipboyguy

  committed
  88bc519

Commits on Jun 13, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Pipboyguy

  committed
  aab21a6

• Refactor schema creation and remove unused code

  Show description for 3b85442
  Pipboyguy

  committed
  3b85442

• Add mapping for provider environment variables and update schema comment

  Show description for 31e1895
  Pipboyguy

  committed
  31e1895

• Update package versions in pyproject.toml and poetry.lock

  Show description for 9400e02
  Pipboyguy

  committed
  9400e02

• Refactor LanceDB utils and client, handle exception and remove unnecessary comment

  Show description for 10c41a8
  Pipboyguy

  committed
  10c41a8

• Refactor utility functions in lancedb tests

  Show description for 9ba97a8
  Pipboyguy

  committed
  9ba97a8

• Update 'replace' mode and improve table handling in lancedb client

  Show description for 97127a5
  Pipboyguy

  committed
  97127a5

• Refactor assert_unordered_list_equal to handle dictionaries

  Show description for b77b6f7
  Pipboyguy

  committed
  b77b6f7

• Refactor code for better readability and remove unnecessary blank lines

  Show description for bc8948e
  Pipboyguy

  committed
  bc8948e

• Refactor code for readability and remove redundant comments

  Show description for c9a1667
  Pipboyguy

  committed
  c9a1667

• Update sentinel table name in test_pipeline.py

  Show description for 84357d7
  Pipboyguy

  committed
  84357d7

• "Add order by clause to database query in lancedb_client"

  Show description for a334c98
  Pipboyguy

  committed
  a334c98

Commits on Jun 14, 2024

• Use super method to reduce redundancy

  Show description for d4b56da
  Pipboyguy

  committed
  d4b56da

• Syntax

  Show description for 8f89d38
  Pipboyguy

  committed
  8f89d38

• Remove bare except clauses

  Show description for 3b44631
  Pipboyguy

  committed
  3b44631

• Revert "Remove bare except clauses"

  Show description for ee6f525
  Pipboyguy

  committed
  ee6f525

• Remove bare except clause

  Show description for d8db465
  Pipboyguy

  committed
  d8db465

• Remove bare except clause

  Show description for 5aed9e5
  Pipboyguy

  committed
  5aed9e5

• Remove bare except clause

  Show description for cd68966
  Pipboyguy

  committed
  cd68966

• Remove bare except clause

  Show description for 11f562f
  Pipboyguy

  committed
  11f562f

• Refactor error handling in LanceDB client

  Show description for 06408b9
  Pipboyguy

  committed
  06408b9

• Add configurable sentinel table name in LanceDB client configuration

  Show description for cf41859
  Pipboyguy

  committed
  cf41859

• Update embedding model config and schema in LanceDB

  Show description for bbc00a5
  Pipboyguy

  committed
  bbc00a5

• Refactor lancedb_client.py, remove unused methods and imports

  Show description for 9322477
  Pipboyguy

  committed
  9322477

Commits on Jun 15, 2024

• Add support for adding multiple fields to LanceDB table in a single operation

  Show description for 9b4c519
  Pipboyguy

  committed
  9b4c519

• Only filter by successful loads

  Show description for 71ffaaa
  Pipboyguy

  committed
  71ffaaa

• Remove redundant exception handling in JSON extraction

  Show description for 9a7c0b5
  Pipboyguy

  committed
  9a7c0b5

• Refactor lancedb_client.py for better code readability

  Show description for a854649
  Pipboyguy

  committed
  a854649

• Merge remote-tracking branch 'origin/1370-lancedb-destination' into 1370-lancedb-destination

  Show description for 2751699
  Pipboyguy

  committed
  2751699

• Refactor lancedb_client.py for improved code readability

  Show description for 348825f
  Pipboyguy

  committed
  348825f

• Fix module docstring

  Show description for 4bfee6d
  Pipboyguy

  committed
  4bfee6d

• Remove embedding_fields from make_arrow_field_schema function

  Show description for 54ef1ac
  Pipboyguy

  committed
  54ef1ac

• Add merge key support

  Show description for a11f0e4
  Pipboyguy

  committed
  a11f0e4

• Refactor get_stored_state to perform join in memory

  Show description for aff0032
  Pipboyguy

  committed
  aff0032

• Packaging

  Show description for 7c693b8
  Pipboyguy

  committed
  7c693b8

• Format

  Show description for 78cb85d
  Pipboyguy

  committed
  78cb85d

• Update dependencies in GitHub workflow for testing lancedb

  Show description for de23786
  Pipboyguy

  committed
  de23786

• Add "cohere" to package dependencies in pyproject.toml

  Show description for 894585c
  Pipboyguy

  committed
  894585c

Commits on Jun 16, 2024

• Dependencies

  Show description for c228ea3
  Pipboyguy

  committed
  c228ea3

• Update dependencies installation in GitHub workflow

  Show description for d472e16
  Pipboyguy

  committed
  d472e16

• Dependencies

  Show description for ba6d8b8
  Pipboyguy

  committed
  ba6d8b8

• Update dependency in GitHub workflow

  Show description for 71b7fce
  Pipboyguy

  committed
  71b7fce

• Dependencies

  Show description for b7d1ebf
  Pipboyguy

  committed
  b7d1ebf

• Dependencies

  Show description for a4f355d
  Pipboyguy

  committed
  a4f355d

• Add documentation for LanceDB

  Show description for a7249b2
  Pipboyguy

  committed
  a7249b2

• Add limitations

  Show description for 1052757
  Pipboyguy

  committed
  1052757

• Offload ordering logic from LanceDB

  Show description for 5c7288b
  Pipboyguy

  committed
  5c7288b

• Update import statements in lancedb client and exceptions files

  Show description for 554c471
  Pipboyguy

  committed
  554c471

• Create _get_table_name getter

  Show description for 3750aae
  Pipboyguy

  committed
  3750aae

• Format

  Show description for 289d679
  Pipboyguy

  committed
  289d679

• Avoid race conditions by delegating all state management to dlt

  Show description for a8e4e62
  Pipboyguy

  committed
  a8e4e62

• Imports

  Show description for b977e5c
  Pipboyguy

  committed
  b977e5c

Commits on Jun 17, 2024

• small doc and test fixes

  sh-rp

  committed
  232a84f

• Fix OpenAI embedding handling of empty strings

  Show description for e08bb8a
  Pipboyguy

  committed
  e08bb8a

• Add 'embeddings' dependencies manually

  Show description for a862742
  Pipboyguy

  committed
  a862742

• Finally...

  Show description for c722979
  Pipboyguy

  committed
  c722979

• Dependencies

  Show description for eb16858
  Pipboyguy

  committed
  eb16858

• Dependencies

  Show description for d510af1
  Pipboyguy

  committed
  d510af1

• Docs

  Show description for 15a65c9
  Pipboyguy

  committed
  15a65c9

Commits on Jun 18, 2024

• Remove superfluous helper method.

  Show description for 5d42963
  Pipboyguy

  committed
  5d42963

Commits on Jun 21, 2024

• Merge branch 'refs/heads/devel' into 1370-lancedb-destination

  Show description for 9e4d8bd
  Pipboyguy

  committed
  9e4d8bd

• Lock File

  Show description for e07ab1f
  Pipboyguy

  committed
  e07ab1f

• Make api_key and embedding_model_provider_api_key optional

  Show description for ec78282
  Pipboyguy

  committed
  ec78282

• Clear environment for config test

  Show description for 9d3c57d
  Pipboyguy

  committed
  9d3c57d

• Minor test config

  Show description for 7f08f85
  Pipboyguy

  committed
  7f08f85

• test config

  Show description for db761e9
  Pipboyguy

  committed
  db761e9

• lancedb config

  Show description for e450f47
  Pipboyguy

  committed
  e450f47

• Config test

  Show description for 44405d7
  Pipboyguy

  committed
  44405d7

• Config

  Show description for 656a9b7
  Pipboyguy

  committed
  656a9b7

• config

  Show description for 107907a
  Pipboyguy

  committed
  107907a

• Import lancedb_adapter function instead of module in adapter collection module

  Show description for 114e74d
  Pipboyguy

  committed
  114e74d

Commits on Jun 25, 2024

• Merge branch 'devel' into 1370-lancedb-destination

  Show description for 433ce28
  Pipboyguy

  committed
  433ce28

• Clarify embedding facilities in LanceDB docs

  Show description for 38f9e11
  Pipboyguy

  committed
  38f9e11

Commits on Jun 27, 2024

• Merge branch 'devel' into 1370-lancedb-destination

  Show description for a318ccb
  sh-rp

  committed
  a318ccb

• update lancedb to support new naming setup (cleanup will follow)

  sh-rp

  committed
  19f583d

• update lockfile

  sh-rp

  committed
  db1e81d