Vector Questions?

[thatcort]

Hi, I'm wondering about the intentions for the vector features in Databend.

• What Vector DB features do you plan to implement?
• Can users upload their own (not OpenAI) embeddings?
• Can users use their own models for generating embeddings?
• How do you index embedding columns?
• Do you plan to add ANN indexing?
• Do you plan to add other distance measures (euclidean)?

Thank you!

[BohuTANG]

Hi, I'm wondering about the intentions for the vector features in Databend.

• What Vector DB features do you plan to implement?

We don't have plans yet, what's the features do you want?

• Can users upload their own (not OpenAI) embeddings?

Sure, Databend vector data type is ARRAY, you can store any dimensions to it.

• Can users use their own models for generating embeddings?
  Sure, Databend is vector storage in this case.

• How do you index embedding columns?

We have no index for ARRAY yet but it's on plan.

• Do you plan to add ANN indexing?

Yes, Can you give some suggestions for it? Like why ANN indexing is better than others for vector.

• Do you plan to add other distance measures (euclidean)?

Sure if users need it, it's a general function in Databend to implement.

Thank you!

[thatcort]

Yes, Can you give some suggestions for it? Like why ANN indexing is better than others for vector.

I like the separated storage/compute aspect of Databend. In that respect it reminds me of Milvus. It would be amazing to have a single object-store based db that can handle both ANN vector data/queries as well as regular SQL queries.

[BohuTANG]

ANN indexing: #11054

[BohuTANG]

Hi @thatcort

Do you plan to add other distance measures (euclidean)?

We currently offer cosine_distance, and OpenAI recommends its use. Could you please clarify the specific requirements or use cases for implementing Euclidean distance?
Thanks.

[thatcort]

Cosine makes sense if the embeddings are normalized (OpenAI's are). Other models don't normalize their embeddings, so in those cases other metrics like Euclidean make more sense.