Code

LanceDB - Remove Orphaned Chunks#1620

Code

Merged

rudolfix merged 149 commits intodeveldlt-hub/dlt:develfrom 1587-lancedb-support-efficient-update-strategy-for-chunked-documentsdlt-hub/dlt:1587-lancedb-support-efficient-update-strategy-for-chunked-documentsCopy head branch name to clipboardNov 6, 2024

+1,152-295

Commits

Commits on Jul 16, 2024

• Add tests for LanceDB chunking and merging functionality

  Show description for 68e26a0
  Pipboyguy

  committed
  68e26a0

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  7c2d031

Commits on Jul 17, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  4c555e9

Commits on Jul 18, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  6c734d7

• Add TSplitter type alias for LanceDB document splitting function

  Show description for 900c4fa
  Pipboyguy

  committed
  900c4fa

• Refine typing for chunks

  Show description for 16230a7
  Pipboyguy

  committed
  16230a7

Commits on Jul 19, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  d3aeda2

• Add type definitions for chunk splitter function and related types

  Show description for 3f7a82f
  Pipboyguy

  committed
  3f7a82f

• Remove unused ChunkInputT, ChunkOutputT, and TSplitter type definitions

  Show description for 1dda1d5
  Pipboyguy

  committed
  1dda1d5

Commits on Jul 21, 2024

• Implement efficient update strategy for chunked documents in LanceDB

  Show description for 48e14ab
  Pipboyguy

  committed
  48e14ab

• Implement efficient update strategy for chunked documents in LanceDB

  Show description for 32fe174
  Pipboyguy

  committed
  32fe174

• Refactor LanceDB client and tests for improved readability and type safety

  Show description for d974962
  Pipboyguy

  committed
  d974962

• Linting

  Show description for e6cdf5d
  Pipboyguy

  committed
  e6cdf5d

Commits on Jul 23, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  c7c2bc6

Commits on Jul 24, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  bf3c3d8

Commits on Jul 25, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  9c11964

Commits on Jul 27, 2024

• Add document_id parameter to lancedb_adapter and update merge logic

  Show description for a60737a
  Pipboyguy

  committed
  a60737a

Commits on Jul 29, 2024

• Merge remote-tracking branch 'origin/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  cfe1a6d

• Remove resolved comments

  Show description for 518a507
  Pipboyguy

  committed
  518a507

• Implement efficient orphan removal for chunked documents in LanceDB

  Show description for c10bd73
  Pipboyguy

  committed
  c10bd73

Commits on Jul 30, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  24ada84

• Implement efficient update strategy for chunked documents in LanceDB

  Show description for 5b3acb1
  Pipboyguy

  committed
  5b3acb1

• Add test for removing orphaned records in LanceDB

  Show description for cf6d86a
  Pipboyguy

  committed
  cf6d86a

• Update LanceDB orphaned records removal test for chunked documents

  Show description for d338586
  Pipboyguy

  committed
  d338586

• Set test pipeline as dev mode

  Show description for 2376c6a
  Pipboyguy

  committed
  2376c6a

• Fix write disposition check in LanceDBRemoveOrphansJob execute method

  Show description for 7f6f1cd
  Pipboyguy

  committed
  7f6f1cd

Commits on Jul 31, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  b840f8b

• Add FollowupJob trait to LoadLanceDBJob

  Show description for c276211
  Pipboyguy

  committed
  c276211

• Fix file type

  Show description for dbfd5af
  Pipboyguy

  committed
  dbfd5af

• Fix file typing

  Show description for 257fbde
  Pipboyguy

  committed
  257fbde

• Add test for removing orphaned records in LanceDB root table

  Show description for 0502ddf
  Pipboyguy

  committed
  0502ddf

• Enhance LanceDB test to cover nested child removal and update scenarios

  Show description for 2363b51
  Pipboyguy

  committed
  2363b51

Commits on Aug 1, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  a296c77

• Use doc id hint for top level tables

  Show description for 6b363d1
  Pipboyguy

  committed
  6b363d1

• Only join on join columns for orphan removal job

  Show description for aac7647
  Pipboyguy

  committed
  aac7647

• Add ollama to supported embedding providers and test orphaned record removal with embeddings

  Show description for e33b7cf
  Pipboyguy

  committed
  e33b7cf

Commits on Aug 2, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  afa7573

• Add merge_key to document resource for efficient updates in LanceDB

  Show description for f2913e9
  Pipboyguy

  committed
  f2913e9

• Formatting

  Show description for ffe6584
  Pipboyguy

  committed
  ffe6584

• Set default file size to 128MB

  Show description for 0368018
  Pipboyguy

  committed
  0368018

Commits on Aug 3, 2024

• Merge branch 'refs/heads/devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  29fa7fd

• Only use parquet loader file formats

  Show description for 02704d5
  Pipboyguy

  committed
  02704d5

Commits on Aug 4, 2024

• Import pyarrow.parquet

  Show description for eae056a
  Pipboyguy

  committed
  eae056a

• Remove recommended file size from LanceDB destination capabilities

  Show description for dc20a55
  Pipboyguy

  committed
  dc20a55

• Update LanceDB client to use more efficient batch processing methods on loading for Load Jobs

  Show description for 6ed540b
  Pipboyguy

  committed
  6ed540b

Commits on Aug 5, 2024

• Refactor unique identifier handling for LanceDB tables

  Show description for 0a9682f
  Pipboyguy

  committed
  0a9682f

• Optimize UUID column generation for LanceDB tables

  Show description for a99224a
  Pipboyguy

  committed
  a99224a

• Refactor LanceDBClient to use string type hints for Table

  Show description for 895331b
  Pipboyguy

  committed
  895331b

• Minor refactor

  Show description for a881e7a
  Pipboyguy

  committed
  a881e7a

• Implement efficient schema update with Nullability support

  Show description for 7f245e2
  Pipboyguy

  committed
  7f245e2

• Optimize orphaned chunks removal for large datasets

  Show description for 4fc73dd
  Pipboyguy

  committed
  4fc73dd

Commits on Aug 6, 2024

• Projection pushdown

  Show description for 9378f50
  Pipboyguy

  committed
  9378f50

• Format

  Show description for 9b14583
  Pipboyguy

  committed
  9b14583

• Prevent primary key and document ID hint conflict in merge disposition

  Show description for e21f61b
  Pipboyguy

  committed
  e21f61b

Commits on Aug 7, 2024

• Add recommended file size for LanceDB destination

  Show description for 9725d0e
  Pipboyguy

  committed
  9725d0e

• Improve comment clarity for projection push-down in LanceDB

  Show description for 5238c11
  Pipboyguy

  committed
  5238c11

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Show description for 8e74815
  Pipboyguy

  committed
  8e74815

• Update to new load interface

  Show description for c8f7468
  Pipboyguy

  committed
  c8f7468

Commits on Aug 8, 2024

• Remove unnecessary LanceDBLoadJob attributes

  Show description for af56191
  Pipboyguy

  committed
  af56191

• Change instance attributes to run method as variables

  Show description for 7e33011
  Pipboyguy

  committed
  7e33011

Commits on Aug 9, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  e24e961

• Schedule follow up refernce job

  Show description for ee7dd02
  Pipboyguy

  committed
  ee7dd02

Commits on Aug 10, 2024

• Add follow up lancedb remove orphan job skeleron

  Show description for df498ab
  Pipboyguy

  committed
  df498ab

• Write empty follow up file

  Show description for c08f1ba
  Pipboyguy

  committed
  c08f1ba

• Write parquet

  Show description for f9f94e3
  Pipboyguy

  committed
  f9f94e3

• Add support for reference file format in LanceDB destination

  Show description for b374b0b
  Pipboyguy

  committed
  b374b0b

• Handle parent table name resolution if it doesn't exist in Lance db remove orphan job

  Show description for 2ed3301
  Pipboyguy

  committed
  2ed3301

Commits on Aug 12, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  cb0ba1f

Commits on Aug 14, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  99ac100

Commits on Aug 15, 2024

• Refactor specialised orphan follow up job back to reference job

  Show description for 0694859
  Pipboyguy

  committed
  0694859

Commits on Aug 16, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  ad3b750

Commits on Aug 17, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  4701c6e

• Refactor orphan removal for chunked documents

  Show description for 537a2be
  Pipboyguy

  committed
  537a2be

Commits on Aug 18, 2024

• Fix dlt system table check for name instead of object

  Show description for 3d25306
  Pipboyguy

  committed
  3d25306

• Implement staging methods

  Show description for 2ee8da1
  Pipboyguy

  committed
  2ee8da1

Commits on Aug 19, 2024

• Override staging client methods

  Show description for 2947d55
  Pipboyguy

  committed
  2947d55

• Docs

  Show description for ea5914c
  Pipboyguy

  committed
  ea5914c

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  2e7daed

• Merge branch 'devel' into 1705-lancedb-orphan-removal-via-staging-deletion-inserts

  Pipboyguy

  committed
  5018adf

Commits on Aug 20, 2024

• Override staging client methods

  Show description for 1fcce51
  Pipboyguy

  committed
  1fcce51

• Delete with inserts

  Show description for 8849f11
  Pipboyguy

  committed
  8849f11

• Keep with batch reader

  Show description for c7098fd
  Pipboyguy

  committed
  c7098fd

Commits on Aug 21, 2024

• Merge branch 'devel' into 1705-lancedb-orphan-removal-via-staging-deletion-inserts

  Pipboyguy

  committed
  92ba767

Commits on Aug 22, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  abd9b01

• Remove Lancedb client's staging implementation

  Show description for d8ddcae
  Pipboyguy

  committed
  d8ddcae

• Insert in memory arrow table. This will be optimized

  Show description for 17137a6
  Pipboyguy

  committed
  17137a6

Commits on Aug 26, 2024

• Merge branch 'devel' into 1705-lancedb-orphan-removal-via-staging-deletion-inserts

  Pipboyguy

  committed
  1b0b7bb

• Rename classes to the new job implementation classes

  Show description for 53d896a
  Pipboyguy

  committed
  53d896a

• Use namedtuple for table chain to improve readability

  Show description for 26ba0f5
  Pipboyguy

  committed
  26ba0f5

• Remove orphans by loading all ancestor IDs simultaneously

  Show description for 06e04d9
  Pipboyguy

  committed
  06e04d9

• Fix doc_id adapter

  Show description for 470315e
  Pipboyguy

  committed
  470315e

• Fix doc_id adapter

  Show description for 43eb5b4
  Pipboyguy

  committed
  43eb5b4

• Revert to previous

  Show description for 40a5e73
  Pipboyguy

  committed
  40a5e73

Commits on Aug 27, 2024

• Merge branch 'devel' into 1705-lancedb-orphan-removal-via-staging-deletion-inserts

  Pipboyguy

  committed
  04c8489

• Revert "Remove orphans by loading all ancestor IDs simultaneously"

  Show description for 8cd6003
  Pipboyguy

  committed
  8cd6003

• Remove doc_id hint

  Show description for dad103e
  Pipboyguy

  committed
  dad103e

• Infer merge key if not supplied from provided primary key

  Show description for 15a0cf6
  Pipboyguy

  committed
  15a0cf6

• Remove unused utility functions

  Show description for e9462e3
  Pipboyguy

  committed
  e9462e3

• Remove LanceDB doc ID hints and use schema normalizer

  Show description for 8af98d7
  Pipboyguy

  committed
  8af98d7

• LanceDB writes strange code

  Show description for 4195bb4
  Pipboyguy

  committed
  4195bb4

• Minor Formatting

  Show description for 2573d3a
  Pipboyguy

  committed
  2573d3a

Commits on Aug 28, 2024

• Merge branch 'devel' into remove-lancedb-doc-id-hints

  Pipboyguy

  committed
  19e9366

• Support compound primary and merge keys

  Show description for 86c198c
  Pipboyguy

  committed
  86c198c

• Remove old comment

  Show description for aa03930
  Pipboyguy

  committed
  aa03930

Commits on Aug 29, 2024

• Merge branch 'devel' into remove-lancedb-doc-id-hints

  Pipboyguy

  committed
  fb72c03

• - Change default vector column name to "vector" to conform with lancedb standard

  Show description for d1e4173
  Pipboyguy

  committed
  d1e4173

• Format and fix linting

  Show description for 613f5bc
  Pipboyguy

  committed
  613f5bc

• Add custom embedding function registration test

  Show description for 703c4a8
  Pipboyguy

  committed
  703c4a8

• Spawn process in test to make sure registry can be deserialized from arrow files

  Show description for c07c8fc
  Pipboyguy

  committed
  c07c8fc

• Simplify null string handling

  Show description for 8afa7e1
  Pipboyguy

  committed
  8afa7e1

Commits on Aug 30, 2024

• Change NULL string replacement with random string, doc clarification

  Show description for 2395432
  Pipboyguy

  committed
  2395432

Commits on Aug 31, 2024

• Merge branch 'devel' into remove-lancedb-doc-id-hints

  Pipboyguy

  committed
  2507d22

• Update default vector column name in docs

  Show description for 9a347e6
  Pipboyguy

  committed
  9a347e6

Commits on Sep 2, 2024

• Merge branch 'devel' into remove-lancedb-doc-id-hints

  Pipboyguy

  committed
  4eda894

• Set remove_orphans flag to False on tests that don't require it

  Show description for c0bedb7
  Pipboyguy

  committed
  c0bedb7

• Merge branch '1765-lancedb-destination-cant-query-generated-tables' into remove-lancedb-doc-id-hints

  Show description for 99a4f44
  Pipboyguy

  committed
  99a4f44

• Implement starter arrow string placeholder function

  Show description for 5f0d620
  Pipboyguy

  committed
  5f0d620

• Add test for empty arrow string element vectorised replacement utility function

  Show description for b7f3076
  Pipboyguy

  committed
  b7f3076

• Handle NULL values in addition to empty strings in arrow substitution method

  Show description for e3a4ed0
  Pipboyguy

  committed
  e3a4ed0

• More efficient empty value replacement with canonical arrow usage

  Show description for 4ec894f
  Pipboyguy

  committed
  4ec894f

• Format

  Show description for 9866874
  Pipboyguy

  committed
  9866874

• Bump pyarrow version

  Show description for 7099d5f
  Pipboyguy

  committed
  7099d5f

• Use pa.nulls instead of [None]*len

  Show description for 1c770d1
  Pipboyguy

  committed
  1c770d1

• Update tests

  Show description for 0b11ac7
  Pipboyguy

  committed
  0b11ac7

• Invert remove orphans flag

  Show description for e81736e
  Pipboyguy

  committed
  e81736e

• Implement root table orphan deletion, only integer doc_ids

  Show description for 36abec7
  Pipboyguy

  committed
  36abec7

• Cater for string ids as well in doc_id removal process

  Show description for 5ceeda9
  Pipboyguy

  committed
  5ceeda9

• Fix test with wrong primary key

  Show description for a8f9c3b
  Pipboyguy

  committed
  a8f9c3b

• Just send list of ids as is. don't pc.compute on client end

  Show description for b3baf93
  Pipboyguy

  committed
  b3baf93

• Extract schema matching into utils

  Show description for 589071c
  Pipboyguy

  committed
  589071c

• Add utils

  Show description for a86a13a
  Pipboyguy

  committed
  a86a13a

• Pass all tests

  Show description for 0eba25e
  Pipboyguy

  committed
  0eba25e

• Minor format and cleanup

  Show description for 2b7f4c6
  Pipboyguy

  committed
  2b7f4c6

• Merge branch 'remove-lancedb-doc-id-hints' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Show description for 105b388
  Pipboyguy

  committed
  105b388

Commits on Sep 3, 2024

• Docs

  Show description for ea36b00
  Pipboyguy

  committed
  ea36b00

Commits on Sep 5, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Show description for 2010722
  Pipboyguy

  committed
  2010722

• Amend replace test to test with large number of records to catch race conditions with replace disposition

  Show description for 81eaea9
  Pipboyguy

  committed
  81eaea9

• Fix replace race conditions by delegating truncation to dlt

  Show description for f6d243a
  Pipboyguy

  committed
  f6d243a

Commits on Sep 6, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Pipboyguy

  committed
  3521975

Commits on Sep 8, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Show description for e280001
  Pipboyguy

  committed
  e280001

• Update lock file

  Show description for f32d4cd
  Pipboyguy

  committed
  f32d4cd

Commits on Sep 24, 2024

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  Show description for a804e03
  Pipboyguy

  committed
  a804e03

• Refactor type mapping and schema handling in LanceDB client

  Show description for 7bd2e9c
  Pipboyguy

  committed
  7bd2e9c

• Change 'complex' column type to 'json' in LanceDB client

  Show description for d8a6b75
  Pipboyguy

  committed
  d8a6b75

• update lock file

  Show description for a5a1657
  Pipboyguy

  committed
  a5a1657

Commits on Nov 5, 2024

• fixes generating lancedb literals

  rudolfix

  committed
  0fc0473

• verifies merge key early, fixes column override in adapters

  rudolfix

  committed
  5efd0a8

• Merge branch 'devel' into 1587-lancedb-support-efficient-update-strategy-for-chunked-documents

  rudolfix

  committed
  b0f0ea1

Commits on Nov 6, 2024

• fixes linting errors

  rudolfix

  committed
  4dcc282