resource_control: support config background tasks resource limit

dynamic-config.md

@@ -169,6 +169,7 @@ The following TiKV configuration items can be modified dynamically:
 | `readpool.unified.max-thread-count` | The maximum number of threads in the thread pool that uniformly processes read requests, which is the size of the UnifyReadPool thread pool |
 | `readpool.unified.max-tasks-per-worker` | The maximum number of tasks allowed for a single thread in the unified read pool. `Server Is Busy` error is returned when the value is exceeded. |
 | `readpool.unified.auto-adjust-pool-size` | Determines whether to automatically adjust the UnifyReadPool thread pool size |
+| `resource-control.priority-ctl-strategy` | Config the control strategy of low-priority tasks. |
 | `coprocessor.split-region-on-table` | Enables to split Region by table |
 | `coprocessor.batch-split-limit` | The threshold of Region split in batches |
 | `coprocessor.region-max-size` | The maximum size of a Region |

sql-statements/sql-statement-alter-resource-group.md

@@ -30,7 +30,7 @@ ResourceGroupOptionList ::=
 |   ResourceGroupOptionList ',' DirectResourceGroupOption
 
 DirectResourceGroupOption ::=
-    "RU_PER_SEC" EqOpt stringLit
+    "RU_PER_SEC" EqOpt LengthNum
 |   "PRIORITY" EqOpt ResourceGroupPriorityOption
 |   "BURSTABLE"
 |   "BURSTABLE" EqOpt Boolean
@@ -72,6 +72,7 @@ BackgroundOptionList ::=
 
 DirectBackgroundOption ::=
     "TASK_TYPES" EqOpt stringLit
+|   "UTILIZATION_LIMIT" EqOpt LengthNum
 ```
 
 TiDB supports the following `DirectResourceGroupOption`, where [Request Unit (RU)](/tidb-resource-control.md#what-is-request-unit-ru) is a unified abstraction unit in TiDB for CPU, IO, and other system resources.
@@ -82,7 +83,7 @@ TiDB supports the following `DirectResourceGroupOption`, where [Request Unit (RU
 | `PRIORITY`    | The absolute priority of tasks to be processed on TiKV  | `PRIORITY = HIGH` indicates that the priority is high. If not specified, the default value is `MEDIUM`. |
 | `BURSTABLE`   | If the `BURSTABLE` attribute is set, TiDB allows the corresponding resource group to use the available system resources when the quota is exceeded. |
 | `QUERY_LIMIT` | When the query execution meets this condition, the query is identified as a runaway query and the corresponding action is executed. | `QUERY_LIMIT=(EXEC_ELAPSED='60s', ACTION=KILL, WATCH=EXACT DURATION='10m')` indicates that the query is identified as a runaway query when the execution time exceeds 60 seconds. The query is terminated. All SQL statements with the same SQL text will be terminated immediately in the coming 10 minutes. `QUERY_LIMIT=()` or `QUERY_LIMIT=NULL` means that runaway control is not enabled. See [Runaway Queries](/tidb-resource-control.md#manage-queries-that-consume-more-resources-than-expected-runaway-queries). |
-| `BACKGROUND`  | Configure the background tasks. For more details, see [Manage background tasks](/tidb-resource-control.md#manage-background-tasks). | `BACKGROUND=(TASK_TYPES="br,stats")` indicates that the backup and restore and statistics collection related tasks are scheduled as background tasks. |
+| `BACKGROUND`  | Configure the background tasks. For more details, see [Manage background tasks](/tidb-resource-control.md#manage-background-tasks). | `BACKGROUND=(TASK_TYPES="br,stats", UTILIZATION_LIMIT=30)` indicates that the backup and restore and statistics collection related tasks are scheduled as background tasks and background tasks can consume 30% of the resources of TiKV at most. |
 
 > **Note:**
 >
@@ -152,7 +153,7 @@ SELECT * FROM information_schema.resource_groups WHERE NAME ='rg1';
 Modify the `BACKGROUND` option for the `default` resource group.
 
 ```sql
-ALTER RESOURCE GROUP default BACKGROUND = (TASK_TYPES = "br,ddl");
+ALTER RESOURCE GROUP default BACKGROUND = (TASK_TYPES = "br,ddl", UTILIZATION_LIMIT=30);
 ```
 
 ```sql
@@ -164,11 +165,11 @@ SELECT * FROM information_schema.resource_groups WHERE NAME ='default';
 ```
 
 ```sql
-+---------+------------+----------+-----------+-------------+---------------------+
-| NAME    | RU_PER_SEC | PRIORITY | BURSTABLE | QUERY_LIMIT | BACKGROUND          |
-+---------+------------+----------+-----------+-------------+---------------------+
-| default | UNLIMITED  | MEDIUM   | YES       | NULL        | TASK_TYPES='br,ddl' |
-+---------+------------+----------+-----------+-------------+---------------------+
++---------+------------+----------+-----------+-------------+-------------------------------------------+
+| NAME    | RU_PER_SEC | PRIORITY | BURSTABLE | QUERY_LIMIT | BACKGROUND                                |
++---------+------------+----------+-----------+-------------+-------------------------------------------+
+| default | UNLIMITED  | MEDIUM   | YES       | NULL        | TASK_TYPES='br,ddl', UTILIZATION_LIMIT=30 |
++---------+------------+----------+-----------+-------------+-------------------------------------------+
 1 rows in set (1.30 sec)
 ```
 

tidb-resource-control.md

@@ -401,8 +401,6 @@ You can get more information about runaway queries from the following system tab
 ### Manage background tasks
 
 > **Warning:**
->
-> This feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://docs.pingcap.com/tidb/stable/support) on GitHub.
 > 
 > The background task management in resource control is based on TiKV's dynamic adjustment of resource quotas for CPU/IO utilization. Therefore, it relies on the available resource quota of each instance. If multiple components or instances are deployed on a single server, it is mandatory to set the appropriate resource quota for each instance through `cgroup`. It is difficult to achieve the expected effect in deployment with shared resources like TiUP Playground.
 
@@ -412,7 +410,8 @@ Starting from v7.4.0, the TiDB resource control feature supports managing backgr
 
 #### `BACKGROUND` parameters
 
-`TASK_TYPES`: specifies the task types that need to be managed as background tasks. Use commas (`,`) to separate multiple task types.
+- `TASK_TYPES`: specifies the task types that need to be managed as background tasks. Use commas (`,`) to separate multiple task types.
+- `UTILIZATION_LIMIT`: specifies the maximum resource percentage(0-100) of each TiKV instance that background tasks can consume. By default, the resource granted for background tasks are calculated based on the total resource quota and current foreground workload dynamically, if `UTILIZATION_LIMIT` is configured, the resource granted for background tasks will not exceed this limit.
 
 TiDB supports the following types of background tasks:
 
@@ -444,10 +443,10 @@ By default, the task types that are marked as background tasks are `""`, and the
 
 #### Examples
 
-1. Modify the `default` resource group and mark `br` and `ddl` as background tasks.
+1. Modify the `default` resource group and mark `br` and `ddl` as background tasks and set background tasks resource limit to 30%.
 
     ```sql
-    ALTER RESOURCE GROUP `default` BACKGROUND=(TASK_TYPES='br,ddl');
+    ALTER RESOURCE GROUP `default` BACKGROUND=(TASK_TYPES='br,ddl', UTILIZATION_LIMIT=30);
     ```
 
 2. Change the `default` resource group to revert the background task type to its default value.
@@ -471,11 +470,11 @@ By default, the task types that are marked as background tasks are `""`, and the
     The output is as follows:
 
     ```
-    +---------+------------+----------+-----------+-------------+---------------------+
-    | NAME    | RU_PER_SEC | PRIORITY | BURSTABLE | QUERY_LIMIT | BACKGROUND          |
-    +---------+------------+----------+-----------+-------------+---------------------+
-    | default | UNLIMITED  | MEDIUM   | YES       | NULL        | TASK_TYPES='br,ddl' |
-    +---------+------------+----------+-----------+-------------+---------------------+
+    +---------+------------+----------+-----------+-------------+-------------------------------------------+
+    | NAME    | RU_PER_SEC | PRIORITY | BURSTABLE | QUERY_LIMIT | BACKGROUND                                |
+    +---------+------------+----------+-----------+-------------+-------------------------------------------+
+    | default | UNLIMITED  | MEDIUM   | YES       | NULL        | TASK_TYPES='br,ddl', UTILIZATION_LIMIT=30 |
+    +---------+------------+----------+-----------+-------------+-------------------------------------------+
     ```
 
 5. To explicitly mark tasks in the current session as the background type, you can use `tidb_request_source_type` to explicitly specify the task type. The following is an example:

tikv-configuration-file.md

@@ -2426,6 +2426,16 @@
 + Enabling this configuration item only works when [`tidb_enable_resource_control](/system-variables.md#tidb_enable_resource_control-new-in-v660) is enabled on TiDB. When this configuration item is enabled, TiKV will use the priority queue to schedule the queued read/write requests from foreground users. The scheduling priority of a request is inversely related to the amount of resources already consumed by the resource group that receives this request, and positively related to the quota of the corresponding resource group.
 + Default value: `true`, which means scheduling based on the RU of the resource group is enabled.
 
+### `priority-ctl-strategy` <span class="version-mark">从 v8.4.0 版本开始引入</span>
+
+Configure the control strategy for low priority tasks. TiKV ensures the priority execution of higher priority tasks by applying dynamic quota limit to low-priority tasks. This control strategy is used to calculates the quota for low priority tasks.
+
++ Value Options:
+    + `aggressive`: using this strategy, the flow control policy will prioritize the performance of high-priority tasks, ensuring that the throughput and latency of high-priority tasks are basically unaffected, but low-priority tasks will run slower.
+    + `moderate`: using this strategy, TiKV will impose a balanced flow control limit on low-priority tasks, ensuring that low-priority tasks can use more system available resources while causing little impact on high-priority tasks.
+    + `conservative`: using this strategy, the flow control policy will prioritize ensuring that system resources are fully utilized, and low-priority tasks will try to use available system resources as much as possible, resulting in a greater impact on the performance of high-priority tasks.
++ Default value: `moderate`.
+
 ## split
 
 Configuration items related to [Load Base Split](/configure-load-base-split.md).