wip improved object retrieval

[logan-markewich]

A WIP approach to improve recurisve retrieval.

Basically, if index_node.obj is serializable, then just throw it into the vector db/storage layer, no need for mappings

Example Usage

from llama_index import VectorStoreIndex, StorageContext
from llama_index.schema import IndexNode, TextNode
from llama_index.vector_stores import QdrantVectorStore

index = VectorStoreIndex(nodes=[TextNode(text="bad_node")])

# this isn't serializable, and is maintained as a mapping
bad_node = IndexNode(index_id="bad1", obj=index.as_retriever(), text="bad summary")

# this is serializable, no mapping needed
good_node = IndexNode(index_id="good1", obj=TextNode(text="good_node"), text="good_summary")

index2 = VectorStoreIndex(
        nodes=[good_node],
        objects=[bad_node],
)
nodes = index2.as_retriever(verbose=True).retrieve("test")

print(nodes[0].text)  # -> "good node"
print(nodes[1].text)  # -> "bad node"

# save
index.storage_context.persist()

# loading -- need to provide unserializable objects at load time
load_index_from_storage(storage_context, objects=[bad_node])

TODO

• unit tests
• confirm it works with vector dbs

[jerryjliu]

help me understand, is object_map now only used for retrievers/query engines?

If it's a Node it should now be serialized/deserialized directly on the IndexNode right?

at a high-level once we make retrievers/query engine serializable i was thinking object_map would go away, and we'd replace with a proper docstore

[logan-markewich]

Yes if query engines/retrievers were serializable, this would go away.

Right now, unserializable index nodes have to be passed in under the objects kwarg -- from there, we can build a map of index id to object

Then we can serialize and retrieve the index node without the object.

If an index node is retrieved, the object map is checked if we have its object

[hatianzhang]

nice will upd some notebooks!