[Feature Request] Remote Search Processor

[austintlee]

Is your feature request related to a problem? Please describe

We are doing a lot of experimentations with search results and today, we are having to do most of this work client-side. When we need to make code changes in a search response process to try something new, it is quite tedious and requires multiple steps. We want more flexibility when it comes to changing the behavior of a processor.

Describe the solution you'd like

We basically want a RPC version of the processor. Something that can be executed remotely, e.g. AWS Lambda or a REST endpoint. It can be a Python extension? I'm open to suggestions.

Related component

Search

Describe alternatives you've considered

No response

Additional context

No response

[austintlee]

@dbwiddis What would it take for me to add remote processor functionality as one or more Python extensions. I would like to expose this to OpenSearch as TransportActions.

1/ A SearchRequestProcessor TransportAction, e.g. SearchRequest processRequest(SearchRequest request)
2/ A SearchResponseProcessor TransportAction, e.g. SearchResponse processResponse(SearchRequest request, SearchResponse response)

Ideally, I would like to leverage the new Protobuf transport layer feature for this as well.

It looks like opensearch-sdk-py does not yet support registering transport actions?

Alternatively, we are looking at standing up a microservice outside OpenSearch (Flask or FastAPI). We would really like Python to remote processors which is why we're considering the Python extension (opensearch-sdk-java seems to support what I need in terms of the transport action registration).

[minalsha]

@austintlee could you please share a usecase that would require this support?