Add broker setting to override default implicit query response limit #14452

bziobrowski · 2024-11-14T10:57:14Z

By default single stage query engine (aka v1) uses implicit limit of 10 rows when no limit is set.
This PR adds pinot.broker.default.query.limit setting that allows for overriding it.
It only applies when limit is not set explicitly in the query and value is greater of equal 0.
By default, the new setting is set to 10 to be consistent with previous default.

…of 10.

vrajat

A reasonable chunk of the diff is because of rename of:

_queryResponseLimit -> _queryResponseLimitOverride
DEFAULT_BROKER_QUERY_RESPONSE_LIMIT -> DEFAULT_BROKER_QUERY_RESPONSE_LIMIT_OVERRIDE

The logic is the same though ?

Also the semantics of DEFAULT_BROKER_QUERY_RESPONSE_LIMIT has been changed.

Am I interpreting the diff correctly ?

bziobrowski · 2024-11-14T14:25:32Z

I renamed the variables (but not setting keys) because, in my opinion, existing query response limit is badly named. It is not a default but rather an override because it applies even when sql command contains limit clause (and only applies conditionally ).
The new setting only applies when limit is not set.

codecov-commenter · 2024-11-14T14:32:52Z

Codecov Report

Attention: Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Project coverage is 63.99%. Comparing base (59551e4) to head (a49a276).
Report is 1475 commits behind head on master.

Files with missing lines	Patch %	Lines
...sthandler/BaseSingleStageBrokerRequestHandler.java	72.72%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14452      +/-   ##
============================================
+ Coverage     61.75%   63.99%   +2.24%     
- Complexity      207     1605    +1398     
============================================
  Files          2436     2703     +267     
  Lines        133233   148925   +15692     
  Branches      20636    22819    +2183     
============================================
+ Hits          82274    95307   +13033     
- Misses        44911    46624    +1713     
- Partials       6048     6994     +946

Flag	Coverage Δ
custom-integration1	`100.00% <ø> (+99.99%)`	⬆️
integration	`100.00% <ø> (+99.99%)`	⬆️
integration1	`100.00% <ø> (+99.99%)`	⬆️
integration2	`0.00% <ø> (ø)`
java-11	`63.95% <72.72%> (+2.24%)`	⬆️
java-21	`63.88% <72.72%> (+2.26%)`	⬆️
skip-bytebuffers-false	`63.97% <72.72%> (+2.22%)`	⬆️
skip-bytebuffers-true	`63.86% <72.72%> (+36.13%)`	⬆️
temurin	`63.99% <72.72%> (+2.24%)`	⬆️
unittests	`63.99% <72.72%> (+2.24%)`	⬆️
unittests1	`56.28% <ø> (+9.39%)`	⬆️
unittests2	`34.44% <72.72%> (+6.71%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vrajat

I think I understand the change now. Repeating to verify:

Current Behavior:

If pinot.broker.enable.query.limit.override is set AND PinotQuery.limit > pinot.broker.enable.query.limit, then override the limit.

PR behavior

If pinot.broker.default.query.response.limit is set AND no limit in query, then set limit.
Override logic continues to apply.

A couple of things that threw me off course are:

Rename to OVERRIDE.
New config param name has default in it and rest is the same. I assumed it had something to do with default override limit.

bziobrowski · 2024-11-15T08:52:40Z

Current formula is :
if pinot.broker.enable.query.limit.override and PinotQuery.limit > pinot.broker.query.response.limit then use pinot.broker.query.response.limit`
so it's basically an upper bound on allowed limit value.
PR behavior is as you wrote above.

pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java

vrajat · 2024-11-15T10:15:20Z

PR behavior is as you wrote above.

Got it. And the test cases test the new functionality only.

yashmayya

Thanks for the patch @bziobrowski!

...rc/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

...integration-tests/src/test/java/org/apache/pinot/integration/tests/BrokerQueryLimitTest.java

…imit_setting

yashmayya

LGTM

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

...rc/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java

gortiz · 2024-12-16T09:58:54Z

Can we merge this?

…imit_setting

Jackie-Jiang · 2024-12-16T20:26:04Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

+    public static final String CONFIG_OF_BROKER_DEFAULT_QUERY_LIMIT =
+        "pinot.broker.default.query.limit";
+
+    public static final int DEFAULT_BROKER_QUERY_LIMIT_OVERRIDE = Integer.MAX_VALUE;


Let's keep the name consistent with config key, and make it next to CONFIG_OF_BROKER_QUERY_RESPONSE_LIMIT (i.e. keep line 258 unchanged)

Jackie-Jiang · 2024-12-16T20:28:15Z

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java

+
+    public static final int DEFAULT_BROKER_QUERY_LIMIT_OVERRIDE = Integer.MAX_VALUE;
+
+    // -1 means no limit; value of 10 aligns limit with PinotQuery's defaults.


I don't think v1 allows no limit. What is the behavior when limit is not set?

Jackie-Jiang · 2024-12-16T20:28:59Z

...rc/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java

@@ -140,7 +140,9 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
  protected final int _defaultHllLog2m;
  protected final boolean _enableQueryLimitOverride;
  protected final boolean _enableDistinctCountBitmapOverride;
-  protected final int _queryResponseLimit;
+  protected final int _queryResponseLimitOverride;


Let's keep the variable name unchanged. This is not only for override, but actually the cap of the allowed limit

Jackie-Jiang · 2024-12-16T20:30:56Z

...rc/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java

@@ -308,6 +313,10 @@ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndO
        }
      }

+      if (isDefaultQueryResponseLimitEnabled() && !pinotQuery.isSetLimit()) {


Will we ever hit this? In PinotQuery thrift definition, limit has a default value 10. Will limit always be set?

bziobrowski added 3 commits November 14, 2024 11:51

Add broker setting to override default implicit query response limit …

e06f33f

…of 10.

Fixed style issues.

0f061a6

Fixed style issues.

6a1403a

vrajat reviewed Nov 14, 2024

View reviewed changes

Fixed style issues.

f5d7951

vrajat reviewed Nov 15, 2024

View reviewed changes

bziobrowski added 2 commits November 15, 2024 10:41

Fixed integration test initialization issue.

35f06fb

Fixed integration test initialization issue.

4a9170d

vrajat reviewed Nov 15, 2024

View reviewed changes

pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java Outdated Show resolved Hide resolved

bziobrowski added 2 commits November 15, 2024 13:14

Removed changed made to generated class.

9ab16bb

Fixed style issue.

460be0a

yashmayya reviewed Nov 22, 2024

View reviewed changes

bziobrowski added 2 commits November 29, 2024 12:51

Merge remote-tracking branch 'origin/master' into add_default_query_l…

bab10a4

…imit_setting

Rename default query limit settings.

2517bd8

vrajat approved these changes Nov 29, 2024

View reviewed changes

yashmayya approved these changes Dec 2, 2024

View reviewed changes

pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java Outdated Show resolved Hide resolved

...rc/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java Outdated Show resolved Hide resolved

bziobrowski added 2 commits December 16, 2024 18:50

Merge remote-tracking branch 'origin/master' into add_default_query_l…

cc75244

…imit_setting

Small refactor.

a49a276

Jackie-Jiang reviewed Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add broker setting to override default implicit query response limit #14452

Add broker setting to override default implicit query response limit #14452

bziobrowski commented Nov 14, 2024 •

edited

Loading

vrajat left a comment

bziobrowski commented Nov 14, 2024

codecov-commenter commented Nov 14, 2024 •

edited

Loading

vrajat left a comment

bziobrowski commented Nov 15, 2024

vrajat commented Nov 15, 2024

yashmayya left a comment

yashmayya left a comment

gortiz commented Dec 16, 2024

Jackie-Jiang Dec 16, 2024

Jackie-Jiang Dec 16, 2024

Jackie-Jiang Dec 16, 2024

Jackie-Jiang Dec 16, 2024


		public static final int DEFAULT_BROKER_QUERY_LIMIT_OVERRIDE = Integer.MAX_VALUE;

		// -1 means no limit; value of 10 aligns limit with PinotQuery's defaults.

Add broker setting to override default implicit query response limit #14452

Are you sure you want to change the base?

Add broker setting to override default implicit query response limit #14452

Conversation

bziobrowski commented Nov 14, 2024 • edited Loading

vrajat left a comment

Choose a reason for hiding this comment

bziobrowski commented Nov 14, 2024

codecov-commenter commented Nov 14, 2024 • edited Loading

Codecov Report

vrajat left a comment

Choose a reason for hiding this comment

Current Behavior:

PR behavior

bziobrowski commented Nov 15, 2024

vrajat commented Nov 15, 2024

yashmayya left a comment

Choose a reason for hiding this comment

yashmayya left a comment

Choose a reason for hiding this comment

gortiz commented Dec 16, 2024

Jackie-Jiang Dec 16, 2024

Choose a reason for hiding this comment

Jackie-Jiang Dec 16, 2024

Choose a reason for hiding this comment

Jackie-Jiang Dec 16, 2024

Choose a reason for hiding this comment

Jackie-Jiang Dec 16, 2024

Choose a reason for hiding this comment

bziobrowski commented Nov 14, 2024 •

edited

Loading

codecov-commenter commented Nov 14, 2024 •

edited

Loading