Enhance (Qdrant) hybrid/sparse search support

[enrico-stauss]

Description

This PR gives more explicit control over sparse_embeddings to the developer of RAG applications. The motivation is provided in this issue and the PR directly adapts the QdrantVectorStore. Other VectorStores that support hybrid retrieval will need to be adapted.

This PR aims at enhancing the useablility of the Qdrant hybrid search functionality by adding sparse embedding fields to both the VectorStoreQuery and QueryBundle and using the provided sparse embedding rather than recomputing it.

Additionally the sparse_embedding field is added to the BaseNode class and if set, used by the QdrantVectorStore.add method rather than recomputing it.

Along with the described changes, a larger refactoring reduces code duplication for the QdrantVectorStore.{query, aquery} and enhances readability. The add method got a minor rework, too.

Use Case

If working with (a subclass of) the QueryFusionRetriever and multipe retrievers (that use the same embedding models), the query embedding would be calculated more than once per embedding string. Caching won't help as the embeddings are generally calculated asynchronously. To avoid the resource overhead of duplicate computation, passing the precompted embeddings was possible for dense embeddings already.

Implementation

The commits within this PR should be self-explanatory and well-structured. The main changes are:

• Introduction of a helper function that either uses the provided sparse embedding or calculates the sparse embedding as before
• using self._{a}client.search_batch also in the default case to enable building the requests that shall be sent step-by-step and thus reduce the number of cases
• Removing the async evaluation of await self.asparse_vector_name() by precomputing it in __init__
• Passing down hybrid_top_k from the VectorIndexRetriever kwargs to the QueryBundle (as done with _sparse_top_k)
• Reducing line count by roughly 160 LOC through removal of duplication

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No
  Bumping the versions of
• llama-index-core
• llama-index-vector-stores-qdrant
• llama-index-node-parser-docling
• llama-index-readers-docling
  remains in TODO for now (see below).

Type of Change

In principle, this is a change to the QdrantVectorStore but as the additional fields in QueryBundle and VectorStoreQuery are required, a parallel version bump of llama-index-core (and specification in the requirements) will be necessary. The type of changes made are:

• Refactoring
• Possibly performance enhancement

Backwards compatibility could be achieved easily by checking if the provided QueryBundle has the necessary field but I am unsure if this is the best approach.

How Has This Been Tested?

I added a test that directly ensures that the getter function _get_sparse_embedding_query reuses the provided sparse embedding.
I furthermore implemented unit tests for all 6 variants of {query, aquery} x {dense, sparse, hybrid} by setting the sparse and dense embeddings in a QdrantVectorStore with enable_hybrid=True and running queries with identically set (sparse_)embeddings

• Added new unit/integration tests
• Added new notebook (that tests end-to-end)
• I stared at the code and made sure it makes sense
• Used my private test-case that I unfortunately cannot share

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[enrico-stauss]

I rebased onto main and noticed that a previous commit (78f9a11) introduced somewhat breaking changes with the qdrant vector_store which I resolved in 7a7f684.

Why I consider the changes breaking:
Users that specified a fastembed_sparse_model would automatically run the vector_store in hybrid mode as the enable_hybrid variable would be overwritten before passed to the super init call. If existing code specified fastembed_sparse_model but not ´enable_hybrid=True`, their code will stop working. Easy to resolve on the user-side, but nonetheless...

Is there anything I can do to help integrate this PR within a foreseable timespan? Or are there specific reasons that keep you from doing so? Please let me know if that is the case. I'm happy to continue working on it if there is something missing!

[enrico-stauss]

Hi @logan-markewich, I think you reviewed my last PR on llama-index, could you maybe take a look at this and let me know what you think? I believe that, aside from the enhancement of code quality, the possibility to avoid duplicate computation really adds some value! If you have other concerns, please do share them!

Thanks and kind regards,
Enrico

[enrico-stauss]

I'm not sure why the tests here fail after rebasing. It seems that some dependencies are not available but the failing tests are actually not relevant to the changes in this PR.

[enrico-stauss]

Hi @logan-markewich, this PR keeps growing :D

I finally found some time to write unit tests. In the process I also had to fix unit tests for the docling reader and node_parser which failed due to the newly introduced sparse_embedding field in the BaseNode.

If you, for any reason, don't plan to merge this PR then please let me know, so that I can stop rebasing it over and over again :)