Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 · 2024-12-13T00:00:40Z

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

Adds Verbose Mode for Search Pipelines:
- Introduced the verbose_pipeline parameter to search requests, default to false.
- Tracks each processor’s input, output, execution time, and status (success/failure).
- Provides detailed logs of the data flow through request and response processors.
Improves Pipeline Debugging:
- Captures step-by-step data transformations applied by each processor.
- Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.
Supports All Pipeline Configurations:
- Works seamlessly with:
  - Default pipelines configured at the index level.
  - Pipelines explicitly specified in the search request.
  - Ad-hoc pipelines defined inline.
Test Framework Enhancements:
- Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…etails Signed-off-by: Junwei Dai <[email protected]>

Signed-off-by: Junwei Dai <[email protected]>

junweid62 · 2024-12-13T00:49:43Z

@joshpalis Hey Josh, can you take a look when you have time? :)

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-13T01:41:41Z

❌ Gradle check result for 488377f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-13T04:10:48Z

❌ Gradle check result for 719fa1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Junwei Dai <[email protected]>

joshpalis

Great work @junweid62 on implementing this feature. A few comments from my initial pass

server/src/main/java/org/opensearch/search/SearchHits.java

server/src/main/java/org/opensearch/search/internal/InternalSearchResponse.java

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

github-actions · 2024-12-13T18:38:27Z

❌ Gradle check result for e4e30f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-17T22:29:34Z

❕ Gradle check result for 615a4b6: UNSTABLE

TEST FAILURES:

      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-12-17T22:30:22Z

Codecov Report

Attention: Patch coverage is 42.45810% with 103 lines in your changes missing coverage. Please review.

Project coverage is 72.04%. Comparing base (b67cdf4) to head (615a4b6).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/search/pipeline/ProcessorExecutionDetail.java	18.44%	84 Missing ⚠️
.../java/org/opensearch/search/pipeline/Pipeline.java	68.18%	6 Missing and 1 partial ⚠️
...a/org/opensearch/action/search/SearchResponse.java	16.66%	4 Missing and 1 partial ⚠️
...opensearch/search/builder/SearchSourceBuilder.java	78.57%	0 Missing and 3 partials ⚠️
...ensearch/action/search/SearchResponseSections.java	77.77%	1 Missing and 1 partial ⚠️
...pensearch/rest/action/search/RestSearchAction.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16843      +/-   ##
============================================
+ Coverage     72.03%   72.04%   +0.01%     
- Complexity    65230    65238       +8     
============================================
  Files          5318     5319       +1     
  Lines        304051   304224     +173     
  Branches      43990    44022      +32     
============================================
+ Hits         219021   219187     +166     
- Misses        67121    67134      +13     
+ Partials      17909    17903       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-18T22:34:50Z

❌ Gradle check result for 0ae4d06: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

junweid62 · 2024-12-19T23:26:51Z

Hi, @dbwiddis Could you please take a moment to review this PR when you have the time? I’d appreciate any feedback you might have :)

CHANGELOG.md

martin-gaievski · 2024-12-20T04:09:58Z

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

@@ -302,6 +305,9 @@ public SearchSourceBuilder(StreamInput in) throws IOException {
        if (in.getVersion().onOrAfter(Version.V_2_18_0)) {
            searchPipeline = in.readOptionalString();
        }
+        if (in.getVersion().onOrAfter(Version.CURRENT)) {


I think you need to have exact version here, CURRENT will change with every next release

I initially used the exact version here, but it caused failures in the BWC integration tests. @dbwiddis suggested switching to CURRENT for now, with the plan to update the version in a separate PR when preparing for the release.

This is correct: once this gets backported to 2.x, then a PR should be submitted to 2.x branch changing it to 2_19_0 and then forward-ported. The problem is that current 2.19.0 code (2.x branch) doesn't have this code but would pass the onOrAfter check if we set it now.

Here's some examples of that workflow in progress: #16086 and #16174

That said, it's useful to include a TODO comment here as a reminder so it doesn't get forgotten, leading to bugs like #16590 :)

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

server/src/main/java/org/opensearch/search/internal/InternalSearchResponse.java

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

server/src/main/java/org/opensearch/search/pipeline/PipelineProcessingContext.java

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-26T20:20:52Z

❌ Gradle check result for 500ac7c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dbwiddis

LGTM generally. Some comments and suggestions

dbwiddis · 2024-12-27T23:43:13Z

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

@@ -302,6 +305,9 @@ public SearchSourceBuilder(StreamInput in) throws IOException {
        if (in.getVersion().onOrAfter(Version.V_2_18_0)) {
            searchPipeline = in.readOptionalString();
        }
+        if (in.getVersion().onOrAfter(Version.CURRENT)) {


This is correct: once this gets backported to 2.x, then a PR should be submitted to 2.x branch changing it to 2_19_0 and then forward-ported. The problem is that current 2.19.0 code (2.x branch) doesn't have this code but would pass the onOrAfter check if we set it now.

Here's some examples of that workflow in progress: #16086 and #16174

That said, it's useful to include a TODO comment here as a reminder so it doesn't get forgotten, leading to bugs like #16590 :)

dbwiddis · 2024-12-28T00:04:33Z

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

                beforeResponseProcessor(processor);
                final long start = relativeTimeSupplier.getAsLong();
                processor.processResponseAsync(request, r, requestContext, ActionListener.wrap(rr -> {
                    long took = TimeUnit.NANOSECONDS.toMillis(relativeTimeSupplier.getAsLong() - start);
                    afterResponseProcessor(processor, took);
+                    if (request.source().verbosePipeline()) {
+                        detail.addOutput(Arrays.asList(rr.getHits().deepCopy().getHits()));


Can you help me understand why this deepCopy is needed?

dbwiddis · 2024-12-28T00:09:12Z

server/src/main/java/org/opensearch/search/pipeline/PipelineProcessingContext.java

@@ -17,6 +20,9 @@
 public class PipelineProcessingContext {
    private final Map<String, Object> attributes = new HashMap<>();

+    // Key for processor execution details
+    private static final String PROCESSOR_EXECUTION_DETAILS_KEY = "processorExecutionDetails";


Something seems off to me about having this private constant in this class. I see it's used in a getter so nobody needs to know what it is, but it's also a key in an attributes map that could have a name collision.

Are other attributes defined (should this constant be somewhere else, and public)? If not, is re-using this map for an undocumented purpose a good idea?

dbwiddis · 2024-12-28T00:14:50Z

server/src/main/java/org/opensearch/search/pipeline/ProcessorExecutionDetail.java

+ *
+ * @opensearch.internal
+ */
+@PublicApi(since = "1.0.0")


Is this new? Should this version be different?

dbwiddis · 2024-12-28T00:23:01Z

server/src/main/java/org/opensearch/search/pipeline/ProcessorExecutionDetail.java

+    private static Object parseFieldFromXContent(XContentParser parser) throws IOException {
+        XContentParser.Token token = parser.currentToken();
+        if (token == XContentParser.Token.VALUE_NULL) {
+            return null;
+        } else if (token == XContentParser.Token.START_ARRAY) {
+            return parseArrayFromXContent(parser);
+        } else if (token == XContentParser.Token.START_OBJECT) {
+            return parser.map();
+        } else {
+            return parser.textOrNull();
+        }
+    }


I would be surprised if this utility method (and the related array one) doesn't already exist somewhere (like here?). If it doesn't, perhaps this code should be in such a util class?

github-actions bot added enhancement Enhancement or improvement to existing feature or request Other Priority-High Search Search query, autocomplete ...etc v2.19.0 Issues and PRs related to version 2.19.0 labels Dec 13, 2024

junweid62 added the backport 2.x Backport to 2.x branch label Dec 13, 2024

Add verbose pipeline parameter to output each processor's execution d…

1c3b946

…etails Signed-off-by: Junwei Dai <[email protected]>

add change log

d931750

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from d5a2c4c to d931750 Compare December 13, 2024 00:26

Add merge missing change log

1f879c1

Signed-off-by: Junwei Dai <[email protected]>

Add missing line

488377f

Signed-off-by: Junwei Dai <[email protected]>

opensearch-ci-bot mentioned this pull request Dec 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

Refactor ProcessorExecutionDetail to improve field handling

e4e30f5

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from 719fa1c to e4e30f5 Compare December 13, 2024 17:41

joshpalis reviewed Dec 13, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Dec 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

Fix ITtest Fail

615a4b6

Signed-off-by: Junwei Dai <[email protected]>

opensearch-ci-bot mentioned this pull request Dec 18, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

Add more unit test

0ae4d06

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from 820849b to 0ae4d06 Compare December 18, 2024 21:59

opensearch-ci-bot mentioned this pull request Dec 19, 2024

[AUTOCUT] Gradle Check Flaky Test Report for VerifyVersionConstantsIT #14585

Open

martin-gaievski reviewed Dec 20, 2024

View reviewed changes

resolve comments

500ac7c

Signed-off-by: Junwei Dai <[email protected]>

dbwiddis reviewed Dec 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add verbose pipeline parameter to output each processor's execution details #16843

Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 commented Dec 13, 2024 •

edited

Loading

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 17, 2024

codecov bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024

junweid62 commented Dec 19, 2024

martin-gaievski Dec 20, 2024

junweid62 Dec 26, 2024

dbwiddis Dec 27, 2024

github-actions bot commented Dec 26, 2024

dbwiddis left a comment

dbwiddis Dec 27, 2024

dbwiddis Dec 28, 2024

dbwiddis Dec 28, 2024

dbwiddis Dec 28, 2024

dbwiddis Dec 28, 2024

Add verbose pipeline parameter to output each processor's execution details #16843

Are you sure you want to change the base?

Add verbose pipeline parameter to output each processor's execution details #16843

Conversation

junweid62 commented Dec 13, 2024 • edited Loading

Description

Related Issues

Check List

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 17, 2024

codecov bot commented Dec 17, 2024 • edited Loading

Codecov Report

github-actions bot commented Dec 18, 2024

junweid62 commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 26, 2024

dbwiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junweid62 commented Dec 13, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading