[FEATURE]Add Metrics for Neural Search Usage

[heemin32]

Is your feature request related to a problem?

I would like to track metrics related to the neural search feature, such as the number of neural query calls and the number of text_embedding processor calls.

What solution would you like?

Introducing stats APIs to display these metrics stored in memory.

What alternatives have you considered?

Operating without access to these metrics.

Do you have any additional context?

Cluster operators could use these metrics to analyze usage patterns and gain insights into how frequently specific features are being utilized.

[q-andy]

Hi, you can assign this to me instead

[minalsha]

Hi @heemin32 , can we track features at the neural plugin level instead? This would help both hybrid and neural sparse queries as well.

[q-andy]

@minalsha I'm looking at adding these at a plugin level through a /stats API similar to how k-NN and ml-commons stats APIs work, so neural, neural-sparse, and hybrid can all hook into the same interface. Our goal is also to make these stats declarative and easy to implement on the developer side all through a single unified API.

For neural search use cases, this should include stats for upcoming features on our roadmap like #803, #1138, and #1135.

@yuye-aws @zhichao-aws I'm wondering if there are any existing neural-sparse stats you would be interested in exposing?

Tagging @vibrantvarun for hybrid as well

[vibrantvarun]

#1146 Created the issue for hybrid.

cc: @minalsha @martin-gaievski

[zhichao-aws]

@q-andy I think it would be good to track the sparse encoding processors usage and neural sparse query usage

[minalsha]

@q-andy for hybrid, we want to track hybrid query usage as well as normalization processor and rrf usage(going to GA in 2.19).

[martin-gaievski]

+1 to @minalsha comment, for processors it would be great if we can do drill down and check what techniques and parameters they use. I put similar comment here #1146 (comment)

[heemin32]

Yes, this feature originated from a simple need to track neural feature usage. However, as @q-andy mentioned, we plan to develop a framework that will enable feature developers to easily add their own metrics. This framework will address all metrics requirements across features in neural plugins.