perftest: init v8.5.0 LTS perftest general-mode vs tuning-mode result

tidb-cloud/v8.5-performance-benchmarking-with-sysbench.md

@@ -0,0 +1,347 @@
+---
+title: TiDB Cloud Sysbench Performance Test Report for TiDB v8.5.0
+summary: Introduce the Sysbench performance test results for a TiDB Cloud Dedicated cluster with the TiDB version of v8.5.0.
+---
+
+# TiDB Cloud Sysbench Performance Test Report for TiDB v8.5.0
+
+This document provides the Sysbench performance test steps and results for a TiDB Cloud Dedicated cluster with the TiDB version of v8.5.0. This report can also be used as a reference for the performance of TiDB Self-Managed v8.5.0 clusters.
+
+## Test overview
+
+This test aims at showing the Sysbench performance of TiDB v8.5.0 in the Online Transactional Processing (OLTP) scenario.
+
+## Test environment
+
+### TiDB cluster
+
+The test is conducted on a TiDB cluster with the following settings:
+
+- **Cluster type**: [TiDB Cloud Dedicated](/tidb-cloud/select-cluster-tier.md#tidb-cloud-dedicated)
+- **Cluster version**: v8.5.0
+- **Cloud provider**: AWS (us-west-2)
+- **Cluster configuration**:
+
+    | Node type | Node size | Node quantity | Node storage |
+    |:----------|:----------|:----------|:----------|
+    | TiDB      | 16 vCPU, 32 GiB | 2 | N/A |
+    | TiKV      | 16 vCPU, 64 GiB | 3 | 1000 GiB |
+
+### Parameter configuration
+
+> **Note:**
+>
+> For TiDB Cloud, to modify the TiDB component parameters of your cluster, you can contact [PingCAP Support](/tidb-cloud/tidb-cloud-support.md) for help.
+
+**General Mode Parameters and Variables:**
+The system variable [`tidb_session_plan_cache_size`](https://docs.pingcap.com/tidb/stable/system-variables#tidb_session_plan_cache_size-new-in-v710) controls the maximum number of plans that can be cached. The default value is `100`. For each workload, this document conducts tests with `tidb_session_plan_cache_size` set to `1000`:
+
+```sql
+SET GLOBAL tidb_session_plan_cache_size = 1000;
+```
+
+**Tuning Mode Parameters and Variables:**
+
+- For the tuning mode general parameters and variables, use the follwing value:
+
+- TiDB parameters:
+
+    ```yaml
+    [performance]
+    concurrently-init-stats = true
+    force-init-stats = true
+    lite-init-stats = false
+
+    [pessimistic-txn]
+    max-retry-count = 15
+
+    [tikv-client]
+    region-cache-ttl = 1200
+    ```
+
+- TiKV paramters:
+
+    ```yaml
+    [server]
+    concurrent-send-snap-limit = 64
+    concurrent-recv-snap-limit = 64
+    snap-io-max-bytes-per-sec = "400MB"
+
+    [raftstore]
+    store-io-pool-size = 1
+
+    [raft-engine]
+    prefill-for-recycle = true
+
+    [rocksdb.titan]
+    enabled = true
+    [rocksdb.defaultcf.titan]
+    min-blob-size = "1KB"
+    blob-file-compression = "zstd"
+
+    [storage.flow-control]
+    l0-files-threshold = 60
+    ```
+- TiDB variables:
+
+    ```sql
+    set global tidb_prepared_plan_cache_size=200; 
+    set global tidb_enable_non_prepared_plan_cache=on;
+    set global tidb_ignore_prepared_cache_close_stmt=on;
+    set global tidb_enable_inl_join_inner_multi_pattern=on;
+    set global tidb_opt_derive_topn=on;
+    set global tidb_opt_ordering_index_selectivity_threshold=0.0001;
+    set global tidb_runtime_filter_mode=LOCAL;
+    set global tidb_opt_enable_mpp_shared_cte_execution=on;
+    set global tidb_rc_read_check_ts=on;
+    set global tidb_guarantee_linearizability=off;
+    set global tidb_enable_historical_stats=off;
+    set global tidb_analyze_skip_column_types="json,blob,mediumblob,longblob,mediumtext,longtext";
+    set global tidb_opt_prefer_range_scan=on;
+    set global tidb_max_chunk_size=128;
+    set @@global.tidb_enable_stmt_summary=0;
+    ```
+- For the `oltp_point_select` and `oltp_read_only` workload, use the follwing additional value of the parameters and variables:
+    PD Parameters:
+    ```yaml
+    [schedule]
+    merge-schedule-limit = 0
+    ```
+
+    TiKV Parameters:
+    ```yaml
+    [server]
+    grpc-concurrency = 6
+    ```
+
+    TiDB variables:
+    ```sql
+    set global tidb_enable_stmt_summary=off;
+    set global tidb_enable_collect_execution_info=off;
+    set global tidb_use_plan_baselines=off;
+    set global tidb_executor_concurrency=1;
+    ```
+
+- For the `oltp_write_only` and `oltp_read_write` workload, use the follwing additional value of the parameters and variables:
+    TiKV Parameters:
+    ```yaml
+    [server]
+    grpc-concurrency = 6
+    [raftstore]
+    apply-pool-size = 3
+    store-pool-size = 3
+    ```
+
+    TiDB variables:
+    ```sql
+    set global tidb_enable_stmt_summary=off;
+    set global tidb_enable_collect_execution_info=off;
+    set global tidb_txn_assertion_level=off;
+    set global tidb_guarantee_linearizability=off;
+    set global tidb_executor_concurrency=1;
+    ```
+
+### Benchmark executor
+
+The benchmark executor sends SQL queries to the TiDB cluster. In this test, its hardware configuration is as follows:
+
+- Machine type: Amazon EC2 (us-west-2)
+- Instance type: c6a.2xlarge
+- Sysbench version: sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
+
+## Test steps
+
+This section introduces how to perform the Sysbench performance test step by step.
+
+1. In the [TiDB Cloud console](https://tidbcloud.com/), create a TiDB Cloud Dedicated cluster that meets the [test environment](#tidb-cluster) requirements.
+
+    For more information, see [Create a TiDB Cloud Dedicated cluster](/tidb-cloud/create-tidb-cluster.md).
+
+2. On the benchmark executor, connect to the newly created cluster and create a database named `sbtest`.
+
+    To connect to the cluster, see [Connect to TiDB Cloud Dedicated via Private Endpoint](/tidb-cloud/set-up-private-endpoint-connections.md).
+
+    To create the `sbtest` database, execute the following SQL statement:
+
+    ```sql
+    CREATE DATABASE sbtest;
+    ```
+
+3. Load Sysbench data to the `sbtest` database.
+
+    - The test in this document is implemented based on [sysbench](https://github.com/akopytov/sysbench). To install sysbench, see [Building and installing from source](https://github.com/akopytov/sysbench#building-and-installing-from-source).
+
+    - Run the following `sysbench prepare` command to import 12 tables and 10,000,000 rows to the `sbtest` database. Replace `${HOST}`, `${PORT}`, `${THREAD}`, and `${PASSWORD}` with your actual values.
+
+      ```shell
+        sysbench \
+            --threads=${THREAD} \
+            --rand-type=uniform \
+            --db-driver=mysql \
+            --mysql-db=sbtest \
+            --mysql-host=${HOST} \
+            --mysql-port=${PORT} \
+            --mysql-user=root \
+            --tables=12 \
+            --table-size=10000000 \
+            --mysql-password=${PASSWORD} \
+            oltp_common prepare
+
+        for i in {1..12}; do
+        mysql -u root -h tidb-0 -P 4000 test -e "ANALYZE TABLE sbtest$i"
+        done
+      ```
+
+4. Run the following `sysbench run` command to warmup Sysbench performance tests on different workloads. Replace `${HOST}`, `${PORT}`, `${THREAD}`, and `${PASSWORD}` with your actual values. 
+
+    ```shell
+    sysbench
+        --rand-type=uniform \
+        --db-driver=mysql \
+        --mysql-db=sbtest \
+        --mysql-host=${HOST} \
+        --mysql-port=${PORT} \
+        --mysql-user=root \
+        --tables=12 \
+        --table-size=10000000 \
+        --time=0 \
+        --events=1000000 \
+        --threads=${THREAD} \
+        --mysql-password=${PASSWORD} \
+        oltp_write_only run   
+    ```
+
+5. Run the following `sysbench run` command to conduct Sysbench performance tests on different workloads. This document conducts tests on four workloads: `oltp_point_select`, `oltp_read_only`, `oltp_write_only`, and `oltp_read_write`. Replace `${HOST}`, `${PORT}`, `${THREAD}`, and `${PASSWORD}` with your actual values. 
+
+    **oltp_point_select:**
+    ```shell
+    for thread in 100 200 400; do
+    sleep 60
+    sysbench
+        --rand-type=uniform \
+        --db-driver=mysql \
+        --mysql-db=sbtest \
+        --mysql-host=${HOST} \
+        --mysql-port=${PORT} \
+        --mysql-user=root \
+        --table-size=100000000 \
+        --time=600 \
+        --threads=$thread \
+        oltp_point_select run 
+    done
+    ```
+    **oltp_read_only:**
+    ```shell
+    for thread in 100 200 400; do
+    sleep 60
+    sysbench
+        --rand-type=uniform \
+        --db-driver=mysql \
+        --mysql-db=sbtest \
+        --mysql-host=${HOST} \
+        --mysql-port=${PORT} \
+        --mysql-user=root \
+        --table-size=100000000 \
+        --time=600 \
+        --threads=$thread \
+        oltp_read_only run
+    done
+    ```
+    **oltp_write_only:**
+    ```shell
+    loads=(
+        "--threads=100 --events=6000000"
+        "--threads=200 --events=8000000"
+        "--threads=400 --events=10000000"
+    )
+    for load in "${loads[@]}"; do
+    sleep 60
+    sysbench
+        --rand-type=uniform \
+        --db-driver=mysql \
+        --mysql-db=sbtest \
+        --mysql-host=${HOST}\
+        --mysql-port=${PORT} \
+        --tables=12 \
+        --table-size=10000000 \
+        --time=0\
+        $load
+        oltp_write_only run
+    done
+    ```
+    **oltp_read_write:**
+    ```shell
+    loads=(
+        "--threads=100 --events=6000000"
+        "--threads=200 --events=8000000"
+        "--threads=400 --events=10000000"
+    )
+    for load in "${loads[@]}"; do
+    sleep 60
+    sysbench
+        --rand-type=uniform \
+        --db-driver=mysql \
+        --mysql-db=sbtest \
+        --mysql-host=${HOST}\
+        --mysql-port=${PORT} \
+        --tables=12 \
+        --table-size=10000000 \
+        --time=0\
+        $load
+        oltp_read_write run
+    done
+    ```
+
+6. Configure the tuning parameters above and rerun the command from step 5 to obtain performance data in tuning mode. Compare the performance results obtained from testing with the default configuration.
+
+## Test results
+
+This section introduces the Sysbench performance of v8.5.0 in the [test environment](#test-environment).
+
+### Point select performance
+
+The performance on the `oltp_point_select` workload is as follows:
+
+| Threads |  TPS | 95% latency (ms)| Tuning Mode TPS ｜ 95% latency（ms）｜
+|:--------|:----------|:----------|
+| 50 | 32,081 | 2.00 | 33,038｜1.89 ｜
+| 100 | 60,086 | 2.11 | 63,383| 2.00 |
+| 200 | 110,107 | 2.61 | 111,046 | 2.48 |
+
+![Sysbench point select performance](/media/tidb-cloud/v8.5.0_oltp_point_select.png)
+
+### Read only performance
+
+The performance on the `oltp_read_only` workload is as follows:
+
+| Threads |  TPS | 95% latency (ms)| Tuning Mode TPS ｜ 95% latency（ms）｜
+|:--------|:----------|:----------|
+| 50 | 2,099 | 28.67 | 2,064 ｜29.19 ｜
+| 100 | 3,924 | 30.81 | 3,843 | 30.81 |
+| 200 | 5,658 | 44.17 | 5,893 | 43,39 |
+
+![Sysbench read write performance](/media/tidb-cloud/v8.5.0_oltp_read_only.png)
+
+### Write only performance
+
+The performance on the `oltp_write_only` workload is as follows:
+
+| Threads |  TPS | 95% latency (ms)| Tuning Mode TPS ｜ 95% latency（ms）｜
+|:--------|:----------|:----------|
+| 50 | 3,056 | 19.65 | 3,405 ｜16.71 ｜
+| 100 | 5,644 | 25.28 | 5,894 | 23.95 |
+| 200 | 9,105 | 37.56 | 9,358 | 36.89 |
+
+![Sysbench read write performance](/media/tidb-cloud/v8.5.0_oltp_write_only.png)
+
+### Read write performance
+
+The performance on the `oltp_read_write` workload is as follows:
+
+| Threads |  TPS | 95% latency (ms)| Tuning Mode TPS ｜ 95% latency（ms）｜
+|:--------|:----------|:----------|
+| 50 | 1,062 | 52.89 | 1,054 ｜52.89 ｜
+| 100 | 1,819 | 63.32 | 1,811 | 63.32 |
+| 200 | 2,293 | 110.66 | 2,219 | 116.80 |
+
+![Sysbench update index performance](/media/tidb-cloud/v8.5.0_oltp_read_write.png)