[Bug]: QueryNode memory keeps increasing on search and search_Iterator with DiskANN.

[akmalmasud96]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:v2.5.1
- Deployment mode(standalone or cluster): Aws EKS Cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.5.2

Current Behavior

We have been using search_iterator and the query_nodes' memory keeps increasing, with searching and ultimately the nodes gets crash due to memory overflow. Similar issue of non-release of memory was observed on search, however, this behavior is observed steadily on search, but on the search_iterator it is observed to be increasing exponential.There are 6 query_nodes with 60 GBs of memory, each. The vectors we are searching are 16-bits, 512 dimensional. And the total data is around 653 Million.
At the start, prior to searching, the memory consumed was at 50%, however, it keep on increasing as we performed searching. Attached are the Grafana graphs for query_nodes memory consumption.

Following is the code-block that we are using for the search_iterator.

def worker_function(vector_embedding):
    iterator = collection.search_iterator(
                                        data=[vector_embedding],
                                        anns_field="embeddings",
                                        param=search_params,
                                        batch_size=16384,
                                        expr=None,
                                       output_fields=["id", "vector_id"]
                                    )
    res_dict = []
    stop_iteration = False
    while True:
        res = iterator.next()
        if not res.ids():
            iterator.close()
            break
        else:
            for hit in tqdm.tqdm( res ):
                if hit.distance >= threshold:
                    res_dict.append( {
                       'distance': hit.distance, 
                       'vector_id': hit.entity.get('vector_id'), 
                        'id':hit.entity.get('id'), 
                    }
                    )
                else:
                    stop_iteration = True
                    iterator.close()
                    break
            if stop_iteration == True:
                break
    return  res_dict

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

attach is the query nodes behaviour while using search_iterator.

[PwzXxm]

Thanks for opening the issue. I will investigate into this. May I ask how many iterators are there concurrently?

Are you trying to obtain results with a specific distance threshold? Maybe you can try out range search https://milvus.io/docs/range-search.md#Range-Search

[yanliang567]

/assign @PwzXxm
/unassign

[akmalmasud96]

@PwzXxm This behavior was observed while running one concurrent iterator.

[akmalmasud96]

@PwzXxm it seems that after using the range_search and radius parameters, the query node's memory_consumption remains constant for search_interator. any specific reason for that?
I will also share the behavior about the simple search.

[yanliang567]

/assign @qixuan0212
please keep an eye on this issue

[PwzXxm]

@akmalmasud96 Could you help confirming the behavior, please?

1. Use search_iterator without radius parameter, batch_size=16384, iterate until a threshold was met -> Memory increases from 32GB to 64GB and causing OOM.
2. Use search_iterator with radius parameter (set it to the threshold as above, I guess?), other parameters remain unchanged -> memory remains constant.
3. Use search, not search_iterator, memory remains constant.

[akmalmasud96]

@PwzXxm
Yes, for search iterator, the memory increases as described, with no search params ( range_filter and radius) and stays constant with parameters.
With search-only, if we dont use radius parameters, it also slightly keep increasing with every search.

Also, I want to ask for confirmation for the expected time to perform search with search_iterator , to extract all records. Attached is the image, where we are extracting the given number of records in the mentioned time.