Skip to content

Commit

Permalink
Clarify embedding facilities in LanceDB docs
Browse files Browse the repository at this point in the history
Signed-off-by: Marcel Coetzee <[email protected]>
  • Loading branch information
Pipboyguy committed Jun 25, 2024
1 parent 433ce28 commit 38f9e11
Show file tree
Hide file tree
Showing 4 changed files with 373 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@
from dlt.sources.helpers.rest_client import RESTClient, AuthConfigBase

# access secrets to get openai key and instantiate embedding function
openai_api_key: str = dlt.secrets.get("destination.lancedb.credentials.embedding_model_provider_api_key")
openai_api_key: str = dlt.secrets.get(
"destination.lancedb.credentials.embedding_model_provider_api_key"
)
func = get_registry().get("openai").create(name="text-embedding-3-small", api_key=openai_api_key)


Expand Down
4 changes: 4 additions & 0 deletions docs/website/docs/dlt-ecosystem/destinations/lancedb.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@ function.

## Using an Adapter to Specify Columns to Vectorise

Out of the box, LanceDB will act as a normal database. To use LanceDB's embedding facilities, you'll need to specify which fields you'd like to embed in your dlt resource.

The `lancedb_adapter` is a helper function that configures the resource for the LanceDB destination:

```py
Expand All @@ -142,6 +144,8 @@ lancedb_adapter(
)
```

Bear in mind that you can't use an adapter on a [dlt source](../../general-usage/source.md), only a [dlt resource](../../general-usage/resource.md).

## Write disposition

All [write dispositions](../../general-usage/incremental-loading.md#choosing-a-write-disposition) are supported by the LanceDB destination.
Expand Down
Loading

0 comments on commit 38f9e11

Please sign in to comment.