Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add processors for generating embeddings, adding context to docs, submitting prompts #2008

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Add processors for generating embeddings, adding context to docs, submitting prompts #2008

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fea96e0
OpenAI embeddings processor
hariso Dec 10, 2024
6133700
prompt processor
hariso Dec 11, 2024
6d74f54
linter
hariso Dec 11, 2024
2bfb465
get context processor
hariso Dec 12, 2024
cd96a21
get context
hariso Dec 12, 2024
33681dc
lint
hariso Dec 12, 2024
28db62d
logs
hariso Dec 12, 2024
956c9f6
wrong list of embedding models
hariso Dec 13, 2024
0c8a703
AI pipeline
hariso Dec 13, 2024
0d50385
hack
hariso Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ go 1.23.2

require (
buf.build/gen/go/grpc-ecosystem/grpc-gateway/protocolbuffers/go v1.35.2-20240617172850-a48fcebcf8f1.1
github.com/Azure/azure-sdk-for-go/sdk/ai/azopenai v0.7.1
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.14.0
github.com/Masterminds/semver/v3 v3.3.1
github.com/Masterminds/sprig/v3 v3.3.0
github.com/NYTimes/gziphandler v1.1.1
github.com/bufbuild/buf v1.47.2
github.com/conduitio-labs/conduit-connector-weaviate v0.1.2-0.20241125183412-ae12206633d7
github.com/conduitio/conduit-commons v0.5.0
github.com/conduitio/conduit-connector-file v0.9.0
github.com/conduitio/conduit-connector-generator v0.9.1
Expand Down Expand Up @@ -47,6 +50,7 @@ require (
github.com/tetratelabs/wazero v1.8.2
github.com/twmb/franz-go/pkg/sr v1.2.0
github.com/twmb/go-cache v1.2.1
github.com/weaviate/weaviate-go-client/v4 v4.16.1
go.uber.org/mock v0.5.0
golang.org/x/exp v0.0.0-20241108190413-2d47ceb2692f
golang.org/x/tools v0.28.0
Expand Down Expand Up @@ -78,6 +82,7 @@ require (
github.com/Antonboom/errname v1.0.0 // indirect
github.com/Antonboom/nilnil v1.0.0 // indirect
github.com/Antonboom/testifylint v1.5.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect
github.com/Crocmagnon/fatcontext v0.5.3 // indirect
Expand All @@ -95,6 +100,7 @@ require (
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/apache/arrow/go/arrow v0.0.0-20200730104253-651201b0f516 // indirect
github.com/apache/thrift v0.14.2 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/ashanbrown/forbidigo v1.6.0 // indirect
github.com/ashanbrown/makezero v1.1.1 // indirect
github.com/aws/aws-sdk-go-v2 v1.32.4 // indirect
Expand Down Expand Up @@ -171,6 +177,15 @@ require (
github.com/go-critic/go-critic v0.11.5 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/analysis v0.21.2 // indirect
github.com/go-openapi/errors v0.22.0 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
github.com/go-openapi/jsonreference v0.20.0 // indirect
github.com/go-openapi/loads v0.21.1 // indirect
github.com/go-openapi/spec v0.20.4 // indirect
github.com/go-openapi/strfmt v0.23.0 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-openapi/validate v0.21.0 // indirect
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
github.com/go-toolsmith/astcast v1.1.0 // indirect
Expand Down Expand Up @@ -224,6 +239,7 @@ require (
github.com/jgautheron/goconst v1.7.1 // indirect
github.com/jingyugao/rowserrcheck v1.1.1 // indirect
github.com/jjti/go-spancheck v0.6.2 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/julz/importas v0.1.0 // indirect
github.com/karamaru-alpha/copyloopvar v1.1.0 // indirect
Expand All @@ -242,6 +258,7 @@ require (
github.com/leonklingele/grouper v1.1.2 // indirect
github.com/macabu/inamedparam v0.1.3 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/maratori/testableexamples v1.0.0 // indirect
github.com/maratori/testpackage v1.1.1 // indirect
github.com/matoous/godox v0.0.0-20240105082147-c5b5e0e7c0c0 // indirect
Expand Down Expand Up @@ -274,6 +291,7 @@ require (
github.com/nunnatsa/ginkgolinter v0.18.3 // indirect
github.com/nxadm/tail v1.4.11 // indirect
github.com/oklog/run v1.1.0 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/onsi/ginkgo/v2 v2.21.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
Expand Down Expand Up @@ -340,6 +358,7 @@ require (
github.com/uudashr/gocognit v1.1.3 // indirect
github.com/uudashr/iface v1.2.1 // indirect
github.com/vbatts/tar-split v0.11.6 // indirect
github.com/weaviate/weaviate v1.27.0 // indirect
github.com/xen0n/gosmopolitan v1.2.2 // indirect
github.com/xitongsys/parquet-go v1.6.2 // indirect
github.com/xitongsys/parquet-go-source v0.0.0-20211228015320-b4f792c43cd0 // indirect
Expand All @@ -353,6 +372,7 @@ require (
go.lsp.dev/pkg v0.0.0-20210717090340-384b27a52fb2 // indirect
go.lsp.dev/protocol v0.12.0 // indirect
go.lsp.dev/uri v0.3.0 // indirect
go.mongodb.org/mongo-driver v1.14.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.57.0 // indirect
go.opentelemetry.io/otel v1.32.0 // indirect
Expand All @@ -367,6 +387,7 @@ require (
golang.org/x/exp/typeparams v0.0.0-20241108190413-2d47ceb2692f // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.32.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sync v0.10.0 // indirect
golang.org/x/sys v0.28.0 // indirect
golang.org/x/term v0.27.0 // indirect
Expand Down
Loading
Loading