Add processors for generating embeddings, adding context to docs, submitting prompts #2008
Conversation
| type embeddingProcConfig struct { | ||
| APIKey string `json:"apiKey" validate:"required"` | ||
| Endpoint string `json:"endpoint" default:"https://api.openai.com/v1"` | ||
| Model string `json:"model" validate:"required,inclusion=gpt-4|gpt-4-turbo|gpt-3.5-turbo|text-davinci-003|text-davinci-002|text-curie-001|text-babbage-001|text-ada-001"` |
There was a problem hiding this comment.
so these are not embeddings models?
There was a problem hiding this comment.
Good catch, I was too quick to copy-paste all the models.:)
| Msg("got embeddings") | ||
|
|
||
| for i, record := range records { | ||
| record.Metadata[EmbeddingMetadataBase64] = embeddings.Data[i].EmbeddingBase64 |
There was a problem hiding this comment.
These can be horribly large, my suggestion here is to:
- add the embedding to
.Payload.After - include the model name used for the embedding.
Particularly 2, since there are others openai compat systems which may allow for different models to be used.
There was a problem hiding this comment.
Additionally, these can be very large (depending on the model) and I find that you can reduce the size of them with compression. All floats too. Base64 will not be very compressible.
There was a problem hiding this comment.
You're right, especially about no. 2. As for 1, that would also mean that raw data records get transformed into structured data records. That might be unexpected for some destinations.
hariso
changed the title
Description
Depends on: https://github.com/conduitio-labs/conduit-connector-weaviate/tree/haris/different-vectors.
Quick checks