diff --git a/qdrant-landing/content/documentation/frameworks/_index.md b/qdrant-landing/content/documentation/frameworks/_index.md index 1f3a027cb9..1ea8086837 100644 --- a/qdrant-landing/content/documentation/frameworks/_index.md +++ b/qdrant-landing/content/documentation/frameworks/_index.md @@ -12,6 +12,7 @@ weight: 33 | [Bubble](./bubble) | Development platform for application development with a no-code interface | | [Canopy](./canopy/) | Framework from Pinecone for building RAG applications using LLMs and knowledge bases. | | [Cheshire Cat](./cheshire-cat/) | Framework to create personalized AI assistants using custom data. | +| [Confluent](./confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. | | [DLT](./dlt/) | Python library to simplify data loading processes between several sources and destinations. | | [DocArray](./docarray/) | Python library for managing data in multi-modal AI applications. | | [DocsGPT](./docsgpt/) | Tool for ingesting documentation sources and enabling conversations and queries. | diff --git a/qdrant-landing/content/documentation/frameworks/confluent.md b/qdrant-landing/content/documentation/frameworks/confluent.md new file mode 100644 index 0000000000..af67c7dbe1 --- /dev/null +++ b/qdrant-landing/content/documentation/frameworks/confluent.md @@ -0,0 +1,283 @@ +--- +title: Confluent +weight: 3700 +--- + +![Confluent Logo](/documentation/frameworks/confluent/confluent-logo.png) + +[Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_campaign=tm.pmm_cd.cwc_partner_Qdrant_generic&utm_source=Qdrant&utm_medium=partnerref) is a fully-managed data streaming platform, available on AWS, GCP, and Azure, with a cloud-native Apache Kafka engine for elastic scaling, enterprise-grade security, stream processing, and governance. + +With our [Qdrant-Kafka Sink Connector](https://github.com/qdrant/qdrant-kafka), Qdrant is part of the [Connect with Confluent](https://www.confluent.io/partners/connect/) technology partner program. It brings fully managed data streams directly to organizations through the Confluent Cloud platform. Making it easier for organizations to stream any data to Qdrant with a fully managed Apache Kafka service. + +## Usage + +### Pre-requisites + +- A Confluent Cloud account. You can begin with a [free trial](https://www.confluent.io/confluent-cloud/tryfree/?utm_campaign=tm.pmm_cd.cwc_partner_qdrant_tryfree&utm_source=qdrant&utm_medium=partnerref) with credits for the first 30 days. +- Qdrant instance to connect to. You can get a free cloud instance at [cloud.qdrant.io](https://cloud.qdrant.io/). + +### Installation + +1) Download the latest connector zip file from [Confluent Hub](https://github.com/qdrant/landing_page). + +2) Configure an environment and cluster on Confluent and create a topic to produce messages for. + +3) Navigate to the `Connectors` section of the Confluent cluster and click `Add Plugin`. Upload the zip file with the following info. + +![Qdrant Connector Install](/documentation/frameworks/confluent/install.png) + +4) Once installed, navigate to the connector and set the following configuration values. + +![Qdrant Connector Config](/documentation/frameworks/confluent/config.png) + +Replace the placeholder values with your credentials. + +5) Add the Qdrant instance host to the allowed networking endpoints. + +![Qdrant Connector Endpoint](/documentation/frameworks/confluent/endpoint.png) + +7) Start the connector. + +## Producing Messages + +You can now produce messages for the configured topic, and they'll be written into the configured Qdrant instance. + +![Qdrant Connector Message](/documentation/frameworks/confluent/message.png) + +## Message Formats + +The connector supports messages in the following formats. + +_Click each to expand._ + +
+ Unnamed/Default vector + +Reference: [Creating a collection with a default vector](https://qdrant.tech/documentation/concepts/collections/#create-a-collection). + +```json +{ + "collection_name": "{collection_name}", + "id": 1, + "vector": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8 + ], + "payload": { + "name": "kafka", + "description": "Kafka is a distributed streaming platform", + "url": "https://kafka.apache.org/" + } +} +``` + +
+ +
+ Named multiple vectors + +Reference: [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors). + +```json +{ + "collection_name": "{collection_name}", + "id": 1, + "vector": { + "some-dense": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8 + ], + "some-other-dense": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8 + ] + }, + "payload": { + "name": "kafka", + "description": "Kafka is a distributed streaming platform", + "url": "https://kafka.apache.org/" + } +} +``` + +
+ +
+ Sparse vectors + +Reference: [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors). + +```json +{ + "collection_name": "{collection_name}", + "id": 1, + "vector": { + "some-sparse": { + "indices": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9 + ], + "values": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8, + 0.9, + 1.0 + ] + } + }, + "payload": { + "name": "kafka", + "description": "Kafka is a distributed streaming platform", + "url": "https://kafka.apache.org/" + } +} +``` + +
+ +
+ Multi-vectors + +Reference: + +- [Multi-vectors](https://qdrant.tech/documentation/concepts/vectors/#multivectors) + +```json +{ + "collection_name": "{collection_name}", + "id": 1, + "vector": { + "some-multi": [ + [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8, + 0.9, + 1.0 + ], + [ + 1.0, + 0.9, + 0.8, + 0.5, + 0.4, + 0.8, + 0.6, + 0.4, + 0.2, + 0.1 + ] + ] + }, + "payload": { + "name": "kafka", + "description": "Kafka is a distributed streaming platform", + "url": "https://kafka.apache.org/" + } +} +``` + +
+ +
+ Combination of named dense and sparse vectors + +Reference: + +- [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors). + +- [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors). + +```json +{ + "collection_name": "{collection_name}", + "id": "a10435b5-2a58-427a-a3a0-a5d845b147b7", + "vector": { + "some-other-dense": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8 + ], + "some-sparse": { + "indices": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9 + ], + "values": [ + 0.1, + 0.2, + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8, + 0.9, + 1.0 + ] + } + }, + "payload": { + "name": "kafka", + "description": "Kafka is a distributed streaming platform", + "url": "https://kafka.apache.org/" + } +} +``` + +
+ +## Further Reading + +- [Kafka Connect Docs](https://docs.confluent.io/platform/current/connect/index.html) +- [Confluent Connectors Docs](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/custom-connector-qs.html) diff --git a/qdrant-landing/static/documentation/frameworks/confluent-social-preview.png b/qdrant-landing/static/documentation/frameworks/confluent-social-preview.png new file mode 100644 index 0000000000..1331fb11d0 Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent-social-preview.png differ diff --git a/qdrant-landing/static/documentation/frameworks/confluent/config.png b/qdrant-landing/static/documentation/frameworks/confluent/config.png new file mode 100644 index 0000000000..3fadf1b12d Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent/config.png differ diff --git a/qdrant-landing/static/documentation/frameworks/confluent/confluent-logo.png b/qdrant-landing/static/documentation/frameworks/confluent/confluent-logo.png new file mode 100644 index 0000000000..ded2c4e2f2 Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent/confluent-logo.png differ diff --git a/qdrant-landing/static/documentation/frameworks/confluent/endpoint.png b/qdrant-landing/static/documentation/frameworks/confluent/endpoint.png new file mode 100644 index 0000000000..056369c213 Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent/endpoint.png differ diff --git a/qdrant-landing/static/documentation/frameworks/confluent/install.png b/qdrant-landing/static/documentation/frameworks/confluent/install.png new file mode 100644 index 0000000000..198c880fdd Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent/install.png differ diff --git a/qdrant-landing/static/documentation/frameworks/confluent/message.png b/qdrant-landing/static/documentation/frameworks/confluent/message.png new file mode 100644 index 0000000000..108c11c88e Binary files /dev/null and b/qdrant-landing/static/documentation/frameworks/confluent/message.png differ