diff --git a/TOC.md b/TOC.md index 809d639762f97..c4116f291de93 100644 --- a/TOC.md +++ b/TOC.md @@ -672,6 +672,8 @@ - Release Notes - [All Releases](/releases/release-notes.md) - [Release Timeline](/releases/release-timeline.md) + - v5.4 + - [5.4.0](/releases/release-5.4.0.md) - v5.3 - [5.3.0](/releases/release-5.3.0.md) - v5.2 diff --git a/_index.md b/_index.md index 0ca603dd913c8..4a9fba50ac97e 100644 --- a/_index.md +++ b/_index.md @@ -15,7 +15,7 @@ Designed for the cloud, TiDB provides flexible scalability, reliability and secu - [TiDB Introduction](/overview.md) - [Basic Features](/basic-features.md) -- [TiDB 5.3 Release Notes](/releases/release-5.3.0.md) +- [TiDB 5.4 Release Notes](/releases/release-5.4.0.md) - [TiDB Release Timeline](/releases/release-timeline.md) - [Compatibility with MySQL](/mysql-compatibility.md) - [Usage Limitations](/tidb-limitations.md) diff --git a/basic-features.md b/basic-features.md index 11c87251820a5..d243f5088e486 100644 --- a/basic-features.md +++ b/basic-features.md @@ -9,168 +9,169 @@ This document lists the features supported in each TiDB version. Note that suppo ## Data types, functions, and operators -| Data types, functions, and operators | 5.3 | 5.2 | 5.1 | 5.0 | 4.0 | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Numeric types](/data-type-numeric.md) | Y | Y | Y | Y | Y | -| [Date and time types](/data-type-date-and-time.md) | Y | Y | Y | Y | Y | -| [String types](/data-type-string.md) | Y | Y | Y | Y | Y | -| [JSON type](/data-type-json.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Control flow functions](/functions-and-operators/control-flow-functions.md) | Y | Y | Y | Y | Y | -| [String functions](/functions-and-operators/string-functions.md) | Y | Y | Y | Y | Y | -| [Numeric functions and operators](/functions-and-operators/numeric-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Date and time functions](/functions-and-operators/date-and-time-functions.md) | Y | Y | Y | Y | Y | -| [Bit functions and operators](/functions-and-operators/bit-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Cast functions and operators](/functions-and-operators/cast-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Encryption and compression functions](/functions-and-operators/encryption-and-compression-functions.md) | Y | Y | Y | Y | Y | -| [Information functions](/functions-and-operators/information-functions.md) | Y | Y | Y | Y | Y | -| [JSON functions](/functions-and-operators/json-functions.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Aggregation functions](/functions-and-operators/aggregate-group-by-functions.md) | Y | Y | Y | Y | Y | -| [Window functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | -| [Miscellaneous functions](/functions-and-operators/miscellaneous-functions.md) | Y | Y | Y | Y | Y | -| [Operators](/functions-and-operators/operators.md) | Y | Y | Y | Y | Y | -| [Character sets and collations](/character-set-and-collation.md) [^1] | Y | Y | Y | Y | Y | +| Data types, functions, and operators | **5.4** | 5.3 | 5.2 | 5.1 | 5.0 | 4.0 | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Numeric types](/data-type-numeric.md) | Y | Y | Y | Y | Y | Y | +| [Date and time types](/data-type-date-and-time.md) | Y | Y | Y | Y | Y | Y | +| [String types](/data-type-string.md) | Y | Y | Y | Y | Y | Y | +| [JSON type](/data-type-json.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Control flow functions](/functions-and-operators/control-flow-functions.md) | Y | Y | Y | Y | Y | Y | +| [String functions](/functions-and-operators/string-functions.md) | Y | Y | Y | Y | Y | Y | +| [Numeric functions and operators](/functions-and-operators/numeric-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Date and time functions](/functions-and-operators/date-and-time-functions.md) | Y | Y | Y | Y | Y | Y | +| [Bit functions and operators](/functions-and-operators/bit-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Cast functions and operators](/functions-and-operators/cast-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Encryption and compression functions](/functions-and-operators/encryption-and-compression-functions.md) | Y | Y | Y | Y | Y | Y | +| [Information functions](/functions-and-operators/information-functions.md) | Y | Y | Y | Y | Y | Y | +| [JSON functions](/functions-and-operators/json-functions.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Aggregation functions](/functions-and-operators/aggregate-group-by-functions.md) | Y | Y | Y | Y | Y | Y | +| [Window functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | Y | +| [Miscellaneous functions](/functions-and-operators/miscellaneous-functions.md) | Y | Y | Y | Y | Y | Y | +| [Operators](/functions-and-operators/operators.md) | Y | Y | Y | Y | Y | Y | +| [Character sets and collations](/character-set-and-collation.md) [^1] | Y | Y | Y | Y | Y | Y | ## Indexing and constraints -| Indexing and constraints | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Expression indexes](/sql-statements/sql-statement-create-index.md#expression-index) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Columnar storage (TiFlash)](/tiflash/tiflash-overview.md) | Y | Y | Y | Y | Y | -| [RocksDB engine](/storage-engine/rocksdb-overview.md) | Y | Y | Y | Y | Y | -| [Titan plugin](/storage-engine/titan-overview.md) | Y | Y | Y | Y | Y | -| [Invisible indexes](/sql-statements/sql-statement-add-index.md) | Y | Y | Y | Y | N | -| [Composite `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | -| [Unique indexes](/constraints.md) | Y | Y | Y | Y | Y | -| [Clustered index on integer `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | -| [Clustered index on composite or non-integer key](/constraints.md) | Y | Y | Y | Y | N | +| Indexing and constraints | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Expression indexes](/sql-statements/sql-statement-create-index.md#expression-index) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Columnar storage (TiFlash)](/tiflash/tiflash-overview.md) | Y | Y | Y | Y | Y | Y | +| [RocksDB engine](/storage-engine/rocksdb-overview.md) | Y | Y | Y | Y | Y | Y | +| [Titan plugin](/storage-engine/titan-overview.md) | Y | Y | Y | Y | Y | Y | +| [Invisible indexes](/sql-statements/sql-statement-add-index.md) | Y | Y | Y | Y | Y | N | +| [Composite `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Unique indexes](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Clustered index on integer `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Clustered index on composite or non-integer key](/constraints.md) | Y | Y | Y | Y | Y | N | ## SQL statements -| **SQL statements** [^2] | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| Basic `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE` | Y | Y | Y | Y | Y | -| `INSERT ON DUPLICATE KEY UPDATE` | Y | Y | Y | Y | Y | -| `LOAD DATA INFILE` | Y | Y | Y | Y | Y | -| `SELECT INTO OUTFILE` | Y | Y | Y | Y | Y | -| `INNER JOIN`, `LEFT\|RIGHT [OUTER] JOIN` | Y | Y | Y | Y | Y | -| `UNION`, `UNION ALL` | Y | Y | Y | Y | Y | -| [`EXCEPT` and `INTERSECT` operators](/functions-and-operators/set-operators.md) | Y | Y | Y | Y | N | -| `GROUP BY`, `ORDER BY` | Y | Y | Y | Y | Y | -| [Window Functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | -| [Common Table Expressions (CTE)](/sql-statements/sql-statement-with.md) | Y | Y | Y | N | N | -| `START TRANSACTION`, `COMMIT`, `ROLLBACK` | Y | Y | Y | Y | Y | -| [`EXPLAIN`](/sql-statements/sql-statement-explain.md) | Y | Y | Y | Y | Y | -| [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) | Y | Y | Y | Y | Y | -| [User-defined variables](/user-defined-variables.md) | Experimental | Experimental | Experimental | Experimental | Experimental | +| **SQL statements** [^2] | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| Basic `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE` | Y | Y | Y | Y | Y | Y | +| `INSERT ON DUPLICATE KEY UPDATE` | Y | Y | Y | Y | Y | Y | +| `LOAD DATA INFILE` | Y | Y | Y | Y | Y | Y | +| `SELECT INTO OUTFILE` | Y | Y | Y | Y | Y | Y | +| `INNER JOIN`, `LEFT\|RIGHT [OUTER] JOIN` | Y | Y | Y | Y | Y | Y | +| `UNION`, `UNION ALL` | Y | Y | Y | Y | Y | Y | +| [`EXCEPT` and `INTERSECT` operators](/functions-and-operators/set-operators.md) | Y | Y | Y | Y | Y | N | +| `GROUP BY`, `ORDER BY` | Y | Y | Y | Y | Y | Y | +| [Window Functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | Y | +| [Common Table Expressions (CTE)](/sql-statements/sql-statement-with.md) | Y | Y | Y | Y | N | N | +| `START TRANSACTION`, `COMMIT`, `ROLLBACK` | Y | Y | Y | Y | Y | Y | +| [`EXPLAIN`](/sql-statements/sql-statement-explain.md) | Y | Y | Y | Y | Y | Y | +| [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) | Y | Y | Y | Y | Y | Y | +| [User-defined variables](/user-defined-variables.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | ## Advanced SQL features -| **Advanced SQL features** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Prepared statement cache](/sql-prepare-plan-cache.md) | Y | Experimental | Experimental | Experimental | Experimental | -| [SQL plan management (SPM)](/sql-plan-management.md) | Y | Y | Y | Y | Y | -| [Coprocessor cache](/coprocessor-cache.md) | Y | Y | Y | Y | Experimental | -| [Stale Read](/stale-read.md) | Y | Y | Y | N | N | -| [Follower reads](/follower-read.md) | Y | Y | Y | Y | Y | -| [Read historical data (tidb_snapshot)](/read-historical-data.md) | Y | Y | Y | Y | Y | -| [Optimizer hints](/optimizer-hints.md) | Y | Y | Y | Y | Y | -| [MPP Exection Engine](/explain-mpp.md) | Y | Y | Y | Y | N | -| [Index Merge Join](/explain-index-merge.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Placement Rules in SQL](/placement-rules-in-sql.md) | Experimental | N | N | N | N | +| **Advanced SQL features** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Prepared statement cache](/sql-prepare-plan-cache.md) | Y | Y | Experimental | Experimental | Experimental | Experimental | +| [SQL plan management (SPM)](/sql-plan-management.md) | Y | Y | Y | Y | Y | Y | +| [Coprocessor cache](/coprocessor-cache.md) | Y | Y | Y | Y | Y | Experimental | +| [Stale Read](/stale-read.md) | Y | Y | Y | Y | N | N | +| [Follower reads](/follower-read.md) | Y | Y | Y | Y | Y | Y | +| [Read historical data (tidb_snapshot)](/read-historical-data.md) | Y | Y | Y | Y | Y | Y | +| [Optimizer hints](/optimizer-hints.md) | Y | Y | Y | Y | Y | Y | +| [MPP Exection Engine](/explain-mpp.md) | Y | Y | Y | Y | Y | N | +| [Index Merge Join](/explain-index-merge.md) | Y | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Placement Rules in SQL](/placement-rules-in-sql.md) | Experimental | Experimental | N | N | N | N | ## Data definition language (DDL) -| **Data definition language (DDL)** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| Basic `CREATE`, `DROP`, `ALTER`, `RENAME`, `TRUNCATE` | Y | Y | Y | Y | Y | -| [Generated columns](/generated-columns.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Views](/views.md) | Y | Y | Y | Y | Y | -| [Sequences](/sql-statements/sql-statement-create-sequence.md) | Y | Y | Y | Y | Y | -| [Auto increment](/auto-increment.md) | Y | Y | Y | Y | Y | -| [Auto random](/auto-random.md) | Y | Y | Y | Y | Y | -| [DDL algorithm assertions](/sql-statements/sql-statement-alter-table.md) | Y | Y | Y | Y | Y | -| Multi schema change: add column(s) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Change column type](/sql-statements/sql-statement-modify-column.md) | Y | Y | Y | N | N | -| [Temporary tables](/temporary-tables.md) | Y | N | N | N | N | +| **Data definition language (DDL)** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| Basic `CREATE`, `DROP`, `ALTER`, `RENAME`, `TRUNCATE` | Y | Y | Y | Y | Y | Y | +| [Generated columns](/generated-columns.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Views](/views.md) | Y | Y | Y | Y | Y | Y | +| [Sequences](/sql-statements/sql-statement-create-sequence.md) | Y | Y | Y | Y | Y | Y | +| [Auto increment](/auto-increment.md) | Y | Y | Y | Y | Y | Y | +| [Auto random](/auto-random.md) | Y | Y | Y | Y | Y | Y | +| [DDL algorithm assertions](/sql-statements/sql-statement-alter-table.md) | Y | Y | Y | Y | Y | Y | +| Multi schema change: add column(s) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Change column type](/sql-statements/sql-statement-modify-column.md) | Y | Y | Y | Y | N | N | +| [Temporary tables](/temporary-tables.md) | Y | Y | N | N | N | N | ## Transactions -| **Transactions** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | N | -| [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | N | -| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | -| [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | -| [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | -| [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | -| [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | +| **Transactions** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ---- | :-----: | :-----: | :-----: | :-----: | :-----: | +| [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | Y | N | +| [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | Y | N | +| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | +| [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | Y | +| [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | Y | +| [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | +| [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | ## Partitioning -| **Partitioning** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Range partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | -| [Hash partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | -| [List partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [List COLUMNS partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [`EXCHANGE PARTITION`](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [Dynamic Pruning](/partitioned-table.md#dynamic-pruning-mode) | Experimental | Experimental | Experimental | N | N | +| **Partitioning** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :-----: | +| [Range partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | Y | +| [Hash partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | Y | +| [List partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [List COLUMNS partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [`EXCHANGE PARTITION`](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [Dynamic Pruning](/partitioned-table.md#dynamic-pruning-mode) | Experimental | Experimental | Experimental | Experimental | N | N | ## Statistics -| **Statistics** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [CMSketch](/statistics.md) | Deprecated | Deprecated | Deprecated | Deprecated | Y | -| [Histograms](/statistics.md) | Y | Y | Y | Y | Y | -| [Extended statistics (multiple columns)](/statistics.md) | Experimental | Experimental | Experimental | Experimental | N | -| [Statistics Feedback](/statistics.md#automatic-update) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Fast Analyze](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental | Experimental | +| **Statistics** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [CMSketch](/statistics.md) | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | Y | +| [Histograms](/statistics.md) | Y | Y | Y | Y | Y | Y | +| [Extended statistics (multiple columns)](/statistics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [Statistics Feedback](/statistics.md#automatic-update) | Deprecated | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Fast Analyze](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | ## Security -| **Security** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Transparent layer security (TLS)](/enable-tls-between-clients-and-servers.md) | Y | Y | Y | Y | Y | -| [Encryption at rest (TDE)](/encryption-at-rest.md) | Y | Y | Y | Y | Y | -| [Role-based authentication (RBAC)](/role-based-access-control.md) | Y | Y | Y | Y | Y | -| [Certificate-based authentication](/certificate-authentication.md) | Y | Y | Y | Y | Y | -| `caching_sha2_password` authentication | Y | Y | N | N | N | -| [MySQL compatible `GRANT` system](/privilege-management.md) | Y | Y | Y | Y | Y | -| [Dynamic Privileges](/privilege-management.md#dynamic-privileges) | Y | Y | Y | N | N | -| [Security Enhanced Mode](/system-variables.md#tidb_enable_enhanced_security) | Y | Y | Y | N | N | -| [Redacted Log Files](/log-redaction.md) | Y | Y | Y | Y | N | +| **Security** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ---- | :-----: | :-----: | :-----: | :-----: | :-----: | +| [Transparent layer security (TLS)](/enable-tls-between-clients-and-servers.md) | Y | Y | Y | Y | Y | Y | +| [Encryption at rest (TDE)](/encryption-at-rest.md) | Y | Y | Y | Y | Y | Y | +| [Role-based authentication (RBAC)](/role-based-access-control.md) | Y | Y | Y | Y | Y | Y | +| [Certificate-based authentication](/certificate-authentication.md) | Y | Y | Y | Y | Y | Y | +| `caching_sha2_password` authentication | Y | Y | Y | N | N | N | +| [MySQL compatible `GRANT` system](/privilege-management.md) | Y | Y | Y | Y | Y | Y | +| [Dynamic Privileges](/privilege-management.md#dynamic-privileges) | Y | Y | Y | Y | N | N | +| [Security Enhanced Mode](/system-variables.md#tidb_enable_enhanced_security) | Y | Y | Y | Y | N | N | +| [Redacted Log Files](/log-redaction.md) | Y | Y | Y | Y | Y | N | ## Data import and export -| **Data import and export** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Fast Importer (TiDB Lightning)](/tidb-lightning/tidb-lightning-overview.md) | Y | Y | Y | Y | Y | -| mydumper logical dumper | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | -| [Dumpling logical dumper](/dumpling-overview.md) | Y | Y | Y | Y | Y | -| [Transactional `LOAD DATA`](/sql-statements/sql-statement-load-data.md) | Y | Y | Y | Y | N [^3] | -| [Database migration toolkit (DM)](/migration-overview.md) | Y | Y | Y | Y | Y | -| [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) | Y | Y | Y | Y | Y | -| [Change data capture (CDC)](/ticdc/ticdc-overview.md) | Y | Y | Y | Y | Y | +| **Data import and export** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:| +| [Fast Importer (TiDB Lightning)](/tidb-lightning/tidb-lightning-overview.md) | Y | Y | Y | Y | Y | Y | +| mydumper logical dumper | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | +| [Dumpling logical dumper](/dumpling-overview.md) | Y | Y | Y | Y | Y | Y | +| [Transactional `LOAD DATA`](/sql-statements/sql-statement-load-data.md) | Y | Y | Y | Y | Y | N [^3] | +| [Database migration toolkit (DM)](/migration-overview.md) | Y | Y | Y | Y | Y | Y | +| [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) | Y | Y | Y | Y | Y | Y | +| [Change data capture (CDC)](/ticdc/ticdc-overview.md) | Y | Y | Y | Y | Y | Y | ## Management, observability, and tools -| **Management, observability, and tools** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [TiDB Dashboard](/dashboard/dashboard-intro.md) | Y | Y | Y | Y | Y | -| [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Information schema](/information-schema/information-schema.md) | Y | Y | Y | Y | Y | -| [Metrics schema](/metrics-schema.md) | Y | Y | Y | Y | Y | -| [Statements summary tables](/statement-summary-tables.md) | Y | Y | Y | Y | Y | -| [Slow query log](/identify-slow-queries.md) | Y | Y | Y | Y | Y | -| [TiUP deployment](/tiup/tiup-overview.md) | Y | Y | Y | Y | Y | -| Ansible deployment | N | N | N | N | Deprecated | -| [Kubernetes operator](https://docs.pingcap.com/tidb-in-kubernetes/) | Y | Y | Y | Y | Y | -| [Built-in physical backup](/br/backup-and-restore-use-cases.md) | Y | Y | Y | Y | Y | -| [Global Kill](/sql-statements/sql-statement-kill.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Lock View](/information-schema/information-schema-data-lock-waits.md) | Y | Y | Experimental | Experimental | Experimental | -| [`SHOW CONFIG`](/sql-statements/sql-statement-show-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [`SET CONFIG`](/dynamic-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Continuous Profiling](/dashboard/continuous-profiling.md) | Experimental | N | N | N | N | +| **Management, observability, and tools** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [TiDB Dashboard](/dashboard/dashboard-intro.md) | Y | Y | Y | Y | Y | Y | +| [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Information schema](/information-schema/information-schema.md) | Y | Y | Y | Y | Y | Y | +| [Metrics schema](/metrics-schema.md) | Y | Y | Y | Y | Y | Y | +| [Statements summary tables](/statement-summary-tables.md) | Y | Y | Y | Y | Y | Y | +| [Slow query log](/identify-slow-queries.md) | Y | Y | Y | Y | Y | Y | +| [TiUP deployment](/tiup/tiup-overview.md) | Y | Y | Y | Y | Y | Y | +| Ansible deployment | N | N | N | N | N | Deprecated | +| [Kubernetes operator](https://docs.pingcap.com/tidb-in-kubernetes/) | Y | Y | Y | Y | Y | Y | +| [Built-in physical backup](/br/backup-and-restore-use-cases.md) | Y | Y | Y | Y | Y | Y | +| [Global Kill](/sql-statements/sql-statement-kill.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Lock View](/information-schema/information-schema-data-lock-waits.md) | Y | Y | Y | Experimental | Experimental | Experimental | +| [`SHOW CONFIG`](/sql-statements/sql-statement-show-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [`SET CONFIG`](/dynamic-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Continuous Profiling](/dashboard/continuous-profiling.md) | Experimental | Experimental | N | N | N | N | +| [Top SQL](/dashboard/top-sql.md) | Experimental | N | N | N | N | N | [^1]: TiDB incorrectly treats latin1 as a subset of utf8. See [TiDB #18955](https://github.com/pingcap/tidb/issues/18955) for more details. diff --git a/experimental-features.md b/experimental-features.md index cb457c7a7fefe..87088887807a2 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -7,6 +7,13 @@ summary: Learn the experimental features of TiDB. This document introduces the experimental features of TiDB in different versions. It is **NOT** recommended to use these features in the production environment. +## Performance + ++ [Automatically scale TiFlash thread pool](/tiflash/tiflash-configuration.md). (Introduced in v5.4) ++ [Raft Engine](/tikv-configuration-file.md#raft-engine). (Introduced in v5.4) ++ [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-on-some-columns) (Introduced in v5.4) ++ [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) + ## Stability + TiFlash limits the use of I/O resources by compressing or sorting data, mitigating the contention for I/O resources between background tasks and front-end data reading and writing (Introduced in v5.0) @@ -46,10 +53,6 @@ This document introduces the experimental features of TiDB in different versions + [Titan Level Merge](/storage-engine/titan-configuration.md#level-merge-experimental) (Introduced in v4.0) + TiFlash supports distributing the new data of the storage engine on multiple hard drives to share the I/O pressure. (Introduced in v4.0) -## Data migration - -+ [DM OpenAPI](/dm/dm-open-api.md) (Introduced in v5.3) - ## Backup and restoration + [Back up Raw KV](/br/use-br-command-line-tool.md#back-up-raw-kv-experimental-feature) (Introduced in v3.1) @@ -63,4 +66,5 @@ This document introduces the experimental features of TiDB in different versions + [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) (Introduced in v4.0) + [Cluster diagnostics](/dashboard/dashboard-diagnostics-access.md) (Introduced in v4.0) + [Continuous profiling](/dashboard/continuous-profiling.md) (Introduced in v5.3) -+ [Online Unsafe Recovery](/online-unsafe-recovery.md) (Introduced in v5.3) \ No newline at end of file ++ [Online Unsafe Recovery](/online-unsafe-recovery.md) (Introduced in v5.3) ++ [Top SQL](/dashboard/top-sql.md) (Introduced in v5.4) diff --git a/pd-control.md b/pd-control.md index 909d3e8e3497e..d9c909ce465c8 100644 --- a/pd-control.md +++ b/pd-control.md @@ -27,7 +27,7 @@ If you want to download the latest version of `pd-ctl`, directly download the Ti > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.3.0`, the package download link is `https://download.pingcap.org/tidb-v5.3.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ### Compile from source code diff --git a/pd-recover.md b/pd-recover.md index 920778bbb0173..355430ac733d1 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -26,7 +26,7 @@ To download the latest version of PD Recover, directly download the TiDB package > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.3.0`, the package download link is `https://download.pingcap.org/tidb-v5.3.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ## Quick Start diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md new file mode 100644 index 0000000000000..0f61c31655175 --- /dev/null +++ b/releases/release-5.4.0.md @@ -0,0 +1,436 @@ +--- +title: TiDB 5.4 Release Notes +--- + +# TiDB 5.4 Release Notes + +Release date：February 15, 2022 + +TiDB version: 5.4.0 + +In v5.4, the key new features or improvements are as follows: + +- Support the GBK character set +- Support using Index Merge to access data, which merges the filtering results of indexes on multiple columns +- Support reading stale data using a session variable +- Support persisting the configuration for collecting statistics +- Support using Raft Engine as the log storage engine of TiKV (experimental) +- Optimize the impact of backup on the cluster +- Support using Azure Blob storage as the backup storage +- Continuously improve the stability and performance of TiFlash and the MPP engine +- Add a switch in TiDB Lightning to determine whether to allow importing to an existing table with data +- Optimize the Continuous Profiling feature (experimental) +- TiSpark supports user identification and authentication + +## Compatibility changes + +> **Note:** +> +> When upgrading from an earlier TiDB version to v5.4.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. + +### System variables + +| Variable name | Change type | Description | +| :---------- | :----------- | :----------- | +| [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v540) | Newly added | Controls whether to allow TiDB to collect `PREDICATE COLUMNS`. The default value is `OFF`. | +| [`tidb_enable_paging`](/system-variables.md#tidb_enable_paging-new-in-v540) | Newly added | Controls whether to use the method of paging to send coprocessor requests in `IndexLookUp` operator. The default value is `OFF`.
For read queries that use `IndexLookup` and `Limit` and that `Limit` cannot be pushed down to `IndexScan`, there might be high latency for the read queries and high CPU usage for TiKV's `unified read pool`. In such cases, because the `Limit` operator only requires a small set of data, if you set `tidb_enable_paging` to `ON`, TiDB processes less data, which reduces query latency and resource consumption. | +| [`tidb_enable_top_sql`](/system-variables.md#tidb_enable_top_sql-new-in-v540) | Newly added | Controls whether to enable the Top SQL feature. The default value is `OFF`. | +| [`tidb_persist_analyze_options`](/system-variables.md#tidb_persist_analyze_options-new-in-v540) | Newly added | Controls whether to enable the [ANALYZE configuration persistence](/statistics.md#persist-analyze-configurations) feature. The default value is `OFF`. | +| [`tidb_read_staleness`](/system-variables.md#tidb_read_staleness-new-in-v540) | Newly added | Controls the range of historical data that can be read in the current session. The default value is `0`.| +| [`tidb_regard_null_as_point`](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can use a query condition including null equivalence as a prefix condition for index access. | +| [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | Controls whether to enable the synchronously loading statistics feature. The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded. When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | +| [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | Controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics (`ON`). The default value is `ON`. | +| [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | +| [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) | Modified | The default value is changed from `OFF` to `ON`.
| +| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-in-v304-and-v40) | Modified | Before v5.4.0, this variable can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | + +### Configuration file parameters + +| Configuration file | Configuration | Change type | Description | +| :---------- | :----------- | :----------- | :----------- | +| TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5`. | +| TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | +| TiKV | [`snap-generator-pool-size`](/tikv-configuration-file.md#snap-generator-pool-size-new-in-v540) | Newly added | The size of the `snap-generator` thread pool. The default value is `2`. | +| TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | +| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| +| TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | +| TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are consistent with TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, and `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | +| TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. After this timespan passes, a log file is rotated, which means a timestamp is appended to the file name of the current log file, and a new log file is created. | +| TiKV | `allow-remove-leader` | Deleted | Determines whether to allow deleting the main switch. | +| TiKV | `raft-msg-flush-interval` | Deleted | Determines the interval at which Raft messages are sent in batches. The Raft messages are sent in batches at every interval specified by this configuration item. | +| PD | [`log.level`](/pd-configuration-file.md#level) | Modified | The default value is changed from "INFO" to "info", guaranteed to be case-insensitive. | +| TiFlash | [`profile.default.enable_elastic_threadpool`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Determines whether to enable or disable the elastic thread pool function. Enabling this configuration item can significantly improve TiFlash CPU utilization in high concurrency scenarios. The default value is `false`. | +| TiFlash | [`storage.format_version`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Specifies the version of DTFile. The default value is `2`, under which hashes are embedded in the data file. You can also set the value to `3`. When it is `3`, the data file contains metadata and token data checksum, and supports multiple hash algorithms. | +| TiFlash | [`logger.count`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `10`. | +| TiFlash | [`status.metrics_port`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `8234`. | +| TiFlash | [`raftstore.apply-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads in the pool that flushes Raft data to storage. The default value is `4`. | +| TiFlash | [`raftstore.store-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads that process Raft, which is the size of the Raftstore thread pool. The default value is `4`. | +| TiDB Data Migration (DM) | [`collation_compatible`](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) | Newly added | The mode to sync the default collation in `CREATE` SQL statements. The value options are "loose" (by default) and "strict". | +| TiCDC | `max-message-bytes` | Modified| Change the default value of `max-message-bytes` in Kafka sink to `104857601` (10MB) | +| TiCDC | `partition-num` | Modified | Change the default value of `partition-num` in Kafka Sink from `4` to `3`. It makes TiCDC send messages to Kafaka partitions more evenly. | +| TiDB Lightning | `meta-schema-name` | Modified | Specifies the schema name for the metadata in the target TiDB. From v5.4.0, this schema is created only if you have enabled [parallel import](/tidb-lightning/tidb-lightning-distributed-import.md) (the corresponding parameter is `tikv-importer.incremental-import = true`). | +| TiDB Lightning | `task-info-schema-name` | Newly added | Specifies the name of the database where duplicated data is stored when TiDB Lightning detects conflicts. By default, the value is "lightning_task_info". Specify this parameter only if you have enabled the "duplicate-resolution" feature. | +| TiDB Lightning | `incremental-import` | Newly added | Determines whether to allow importing data to tables where data already exists. The default value is `false`. | + +### Others + +- An interface is added between TiDB and PD. When you use the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use PD in the corresponding version. +- TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). +- Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. +- In v5.3 and earlier versions, [TiDB Data Migration (DM)](https://docs.pingcap.com/tidb-data-migration/v5.3/) documentation is independent of TiDB documentation. Since v5.4, DM documentation is integrated into TiDB documentation with the same version. You can directly read [DM documentation](/dm/dm-overview.md) without accessing the DM documentation site. +- Remove the experimental feature of Point-in-time recovery (PITR) along with cdclog. Since v5.4.0, cdclog-based PITR and cdclog are no longer supported. + +## New features + +### SQL + +- **TiDB supports the GBK character set since v5.4.0** + + Before v5.4.0, TiDB supports `ascii`, `binary`, `latin1`, `utf8`, and `utf8mb4` character sets. + + To better support Chinese users, TiDB supports the GBK character set since v5.4.0. After enabling the [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) option in the TiDB configuration file when initializing a TiDB cluster for the first time, the TiDB GBK character set supports both `gbk_bin` and `gbk_chinese_ci` collations. + + When using the GBK character set, you need to pay attention to the compatibility restrictions. For details, see [Character Set and Collation - GBK](/character-set-gbk.md). + +### Security + +- **TiSpark supports user authentication and authorization** + + Since TiSpark 2.5.0, TiSpark supports both database user authentication and read/write authorization at a database or table level. After enabling this feature, you can prevent the business from running unauthorized batch tasks such as draws to obtain data, which improves the stability and data security of online clusters. + + This feature is disabled by default. When it is enabled, if a user operating through TiSpark does not have the needed permissions, the user gets an exception from TiSpark. + + [User document](/tispark-overview.md#security) + +- **TiUP supports generating an initial password for the root user** + + An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, TiUP generates an initial strong password for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. + + [User document](/production-deployment-using-tiup.md#step-7-start-a-tidb-cluster) + +### Performance + +- **Continue improving the stability and performance of the columnar storage engine TiFlash and the computing engine MPP** + + - Support pusing down more functions to the MPP engine: + + - String functions: `LPAD()`, `RPAD()`, `STRCMP()` + - Date functions: `ADDDATE()`, `DATE_ADD()`, `DATE_SUB()`, `SUBDATE()`, `QUARTER()` + + - Introduce the elastic thread pool feature to improve resource utilization (experimental) + - Improve the efficiency of converting data from row-based storage format to column-based storage format when replicating data from TiKV, which brings 50% improvement in the overall performance of data replication + - Improve TiFlash performance and stability by tuning the default values of some configuration items. In an HTAP hybrid load, the performance of simple queries on a single table improves up to 20%. + + User documents: [Supported push-down calculations](/tiflash/use-tiflash.md#supported-push-down-calculations), [Configure the tiflash.toml file](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) + +- **Read historical data within a specified time range through a session variable** + + TiDB is a multi-replica distributed database based on the Raft consensus algorithm. In face of high-concurrency and high-throughput application scenarios, TiDB can scale out its read performance through follower replicas, separating read and write requests. + + For different application scenarios, TiDB provides two modes of follower read: strongly consistent read and weakly consistent history read. The strongly consistent read mode is suitable for application scenarios that require real-time data. However, in this mode, because of the data replication latency between leaders and followers and the reduced throughput, the read request might have high latency, especially for geo-distributed deployments. + + For the application scenarios that have less strict requirements on real-time data, the history read mode is recommended. This mode can reduce latency and improve throughput. TiDB currently supports reading historical data by the following methods: Use SQL statements to read data from a time point in the past, or start a read-only transaction based on a time point in the past. Both methods support reading the historical data of a specific point in time or within a specified time range. For details, refer to [Read Historical Data Using the `AS OF TIMESTAMP` Clause](/as-of-timestamp.md). + + Since v5.4.0, TiDB improves the usability of the history read mode by supporting reading historical data within a specified time range through a session variable. This mode serves low-latency, high-throughput read requests in quasi-real-time scenarios. You can set the variable as follows: + + ```sql + set @@tidb_replica_read=leader_and_follower + set @@tidb_read_staleness="-5" + ``` + + By this setting, TiDB can select the nearest leader or follower node and read the latest historical data within 5 seconds. + + [User document](/tidb-read-staleness.md) + +- **GA for Index Merge** + + Index Merge is introduced in TiDB v4.0 as an experimental feature for SQL optimization. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, if the filtering conditions connected by `OR` have their respective indexes on columns _key1_ and _key2_, the Index Merge feature filters the respective indexes at the same time, merges the query results, and returns the merged result. + + ```sql + SELECT * FROM table WHERE key1 <= 100 OR key2 = 200; + ``` + + Before TiDB v4.0, a query on a table supports using only one index for filtering at one time. If you want to query multiple columns of data, you can enable Index Merge to get the exact query result in a short time by using the indexes in individual columns. Index Merge avoids unnecessary full table scans and does not require establishing a large number of composite indexes. + + In v5.4.0, Inex Merge becomes GA. However, you still need to pay attention to the following restrictions: + + - Index Merge supports only disjunctive normal form (X1 ⋁ X2 ⋁ …Xn). That is, this feature only works when filtering conditions in a `WHERE` clause are connected by `OR`. + + - For newly deployed TiDB clusters of v5.4.0 or later, this feature is enabled by default. For v5.4.0 or later TiDB clusters upgraded from earlier versions, this feature inherits the setting before the upgrade and you can change the setting as required (for TiDB clusters earlier than v4.0, this feature does not exist and is disabled by default). + + [User document](/explain-index-merge.md) + +- **Support Raft Engine (experimental)** + + Support using [Raft Engine](https://github.com/tikv/raft-engine) as the log storage engine in TiKV. Compared with RocksDB, Raft Engine can reduce TiKV I/O write traffic by up to 40% and CPU usage by 10%, while improving foreground throughput by about 5% and reducing tail latency by 20% under certain loads. In addition, Raft Engine improves the efficiency of log recycling and fixes the issue of log accumulation in extreme conditions. + + Raft Engine is still an experimental feature and is disabled by default. Note that the data format of Raft Engine in v5.4.0 is not compatible with previous versions. Before upgrading or downgrading the cluster, you need to make sure that Raft Engine on all TiKV nodes is disabled. It is recommended to use Raft Engine only in v5.4.0 or a later version. + + [User document](/tikv-configuration-file.md#raft-engine) + +- **Support collecting statistics on `PREDICATE COLUMNS` (experimental)** + + In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. + + Since v5.4.0, you can set the value of the [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v540) system variable to `ON` to enable TiDB to collect `PREDICATE COLUMNS`. + + After the setting, TiDB writes the `PREDICATE COLUMNS` information to the `mysql.column_stats_usage` system table every 100 * [`stats-lease`](/tidb-configuration-file.md#stats-lease). When the query pattern of your business is stable, you can use the `ANALYZE TABLE TableName PREDICATE COLUMNS` syntax to collect statistics on the `PREDICATE COLUMNS` columns only, which can greatly reduce the overhead of collecting statistics. + + [User document](/statistics.md#collect-statistics-on-some-columns) + +- **Support synchronously loading statistics (experimental)** + + Since v5.4.0, TiDB introduces the synchronously loading statistics feature. This feature is disabled by default. After enabling the feature, TiDB can synchronously load large-sized statistics (such as histograms, TopN, and Count-Min Sketch statistics) into memory when you execute SQL statements, which improves the completeness of statistics for SQL optimization. + + [User document](/statistics.md#load-statistics) + +### Stability + +- **Support persisting ANALYZE configurations** + + Statistics are one type of the basic information that the optimizer refers to when generating execution plans. The accuracy of the statistics directly affects whether the generated execution plans are reasonable. To ensure the accuracy of the statistics, sometimes it is necessary to set different collection configurations for different tables, partitions, and indexes. + + Since v5.4.0, TiDB supports persisting some `ANALYZE` configurations. With this feature, the existing configurations can be easily reused for future statistics collection. + + The `ANALYZE` configuration persistence feature is enabled by default (the system variable `tidb_analyze_version` is `2` and `tidb_persist_analyze_options` is `ON` by default). You can use this feature to record the persistence configurations specified in the `ANALYZE` statement when executing the statement manually. Once recorded, the next time TiDB automatically updates statistics or you manually collect statistics without specifying these configurations, TiDB will collect statistics according to the recorded configurations. + + [User document](/statistics.md#persist-analyze-configurations) + +## High availability and disaster recovery + +- **Reduce the impact of backup tasks on the cluster** + + Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature can reduce the impact of backup tasks on the cluster by monitoring the cluster resource usage and adjusting the number of threads used by the backup tasks. In some cases, if you increase the cluster hardware resource for backup and enable the auto-tune feature, it can limit the impact of backup tasks on the cluster to 10% or less. + + [User document](/br/br-auto-tune.md) + +- **Support Azure Blob Storage as a target storage for backup** + + Backup & Restore (BR) supports Azure Blob Storage as a remote backup storage. If you deploy TiDB in Azure Cloud, now you can back up the cluster data to the Azure Blob Storage service. + + [User document](/br/backup-and-restore-azblob.md) + +### Data migration + +- **TiDB Lightning introduces a new feature to determine whether to allow importing data to tables with data** + + TiDB Lightning introduces a configuration item`incremental-import`. It determines whether to allow importing data to tables with data. The default value is `false`. When using the parallel import mode, you must set the configuration to `true`. + + [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) + +- **TiDB Lightning introduces the schema name that stores the meta information for parallel import** + + TiDB Lightning introduces the `meta-schema-name` configuration item. In parallel import mode, this parameter specifies the schema name that stores the meta information for each TiDB Lightning instance in the target cluster. By default, the value is "lightning_metadata". The value set for this parameter must be the same for each TiDB Lightning instance that participates in the same parallel import; otherwise, the correctness of the imported data can not be ensured. + + [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) + +- **TiDB Lightning introduces duplicate resolution** + + In Local-backend mode, TiDB Lightning outputs duplicated data before the data import is completed, and then removes that duplicated data from the database. You can resolve the duplicated data after the import is completed and select suitable data to insert according to application rules. It is recommended to clean upstream data sources based on duplicated data to avoid data inconsistency caused by duplicated data encountered in the subsequent incremental data migration phase. + + [User document](/tidb-lightning/tidb-lightning-error-resolution.md) + +- **Optimize the usage of relay log in TiDB Data Migration (DM)** + + - Recover the `enable-relay` switch in the `source` configuration. + + - Support dynamically enabling and disabling relay log using the `start-relay` and `stop-relay` commands. + + - Bind the status of relay log to `source`. `source` keeps its original status of being enabled or disabled after it is migrated to any DM-worker. + + - Move the storage path of relay log to the DM-worker configuration file. + + [User document](/dm/relay-log.md) + +- **Optimize the processing of [collation](/character-set-and-collation.md) in DM** + + Add the `collation_compatible` configuration item. The value options are `loose` (default) and `strict`: + + - If your application does not have strict requirements on collation, and the collation of query results can be different between the upstream and downstream, you can use the default `loose` mode to avoid reporting errors. + - If your application has strict requirements on collation, and the collation must be consistent between the upstream and downstream, you can use the `strict` mode. However, if the downstream does not support the upstream's default collation, the data replication might report errors. + + [User document](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) + +- **Optimize `transfer source` in DM to support running replication tasks smoothly** + + When the DM-worker nodes have an unbalanced load, the `transfer source` command can be used to manually transfer the configuration of a `source` to another load. After the optimization, the `transfer source` command simplifies the manual operation. You can smoothly transfer the source instead of pausing all related tasks, because DM completes other operations internally. + +- **DM OpenAPI becomes generally available (GA)** + + DM supports daily management via API, including adding data sources and managing tasks. In v5.4.0, DM OpenAPI becomes GA. + + [User document](/dm/dm-open-api.md) + +### Diagnostic efficiency + +- **Top SQL (experimental feature)** + + A new experimental feature, Top SQL (disabled by default), is introduced to help you easily find source-consuming queries. + + [User document](/dashboard/top-sql.md) + +### TiDB data share subscription + +- **Optimize the impact of TiCDC on clusters** + + Significantly reduces the impact on the performance of TiDB clusters when you use TiCDC. In the test environment, the performance impact of TiCDC on TiDB can be reduced to less than 5%. + +### Deployment and maintenance + +- **Enhance Continuous Profiling (experimental)** + + - More components supported: Besides TiDB, PD, and TiKV, TiDB v5.4.0 also supports CPU profiling of TiFlash. + - More forms of profiling display: Supports showing CPU profiling and Goroutine results on flame charts. + + - More deployment environments supported: Continuous Profiling can also be used for clusters deployed using TiDB Operator. + + Continuous Profiling is disabled by default and can be enabled on TiDB Dashboard. + + Continuous Profiling is applicable to clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. + + [User document](/dashboard/continuous-profiling.md) + +## Improvements + ++ TiDB + + - Add a new system variable `tidb_enable_paging` to determine whether to use paging to send Coprocessor requests. Enabling this feature can reduce the amount of data to process and to reduce latency and resource consumption [#30578](https://github.com/pingcap/tidb/issues/30578) + - Support the `ADMIN {SESSION | INSTANCE | GLOBAL} PLAN_CACHE` syntax to clear the cached query plan [#30370](https://github.com/pingcap/tidb/pull/30370) + ++ TiKV + + - Coprocessor supports paging API to process requests in a stream-like way [#11448](https://github.com/tikv/tikv/issues/11448) + - Support `read-through-lock` so that read operations do not need to wait for secondary locks to be resolved [#11402](https://github.com/tikv/tikv/issues/11402) + - Add a disk protection mechanism to avoid panic caused by disk space drainage [#10537](https://github.com/tikv/tikv/issues/10537) + - Support archiving and rotating logs [#11651](https://github.com/tikv/tikv/issues/11651) + - Reduce the system call by the Raft client and increase CPU efficiency [#11309](https://github.com/tikv/tikv/issues/11309) + - Coprocessor supports pushing down substring to TiKV [#11495](https://github.com/tikv/tikv/issues/11495) + - Improve the scan performance by skip reading locks in the Read Committed isolation level [#11485](https://github.com/tikv/tikv/issues/11485) + - Reduce the default thread pool size used by backup operations and limit the use of thread pool when the stress is high [#11000](https://github.com/tikv/tikv/issues/11000) + - Support dynamically adjusting the sizes of the Apply thread pool and the Store thread pool [#11159](https://github.com/tikv/tikv/issues/11159) + - Support configuring the size of the `snap-generator` thread pool [#11247](https://github.com/tikv/tikv/issues/11247) + - Optimize the issue of global lock race that occurs when there are many files with frequent reads and writes [#250](https://github.com/tikv/rocksdb/pull/250) + ++ PD + + - Record the historic hotspot information by default [#25281](https://github.com/pingcap/tidb/issues/25281) + - Add signature for the HTTP component to identify the request source [#4490](https://github.com/tikv/pd/issues/4490) + - Update TiDB Dashboard to v2021.12.31 [#4257](https://github.com/tikv/pd/issues/4257) + ++ TiFlash + + - Optimize the communication of local operators + - Increase the non-temporary thread count of gRPC to avoid the frequent creation or destruction of threads + ++ Tools + + + Backup & Restore (BR) + + - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + + + TiCDC + + - Reduce the count of "EventFeed retry rate limited" logs [#4006](https://github.com/pingcap/tiflow/issues/4006) + - Reduce the replication latency when replicating many tables [#3900](https://github.com/pingcap/tiflow/issues/3900) + - Reduce the time for the KV client to recover when a TiKV store is down [#3191](https://github.com/pingcap/tiflow/issues/3191) + + + TiDB Data Migration (DM) + + - Lower the usage rate of CPU when the relay is enabled [#2214](https://github.com/pingcap/dm/issues/2214) + + + TiDB Lightning + + - Use optimistic transactions by default to write data to improve performance in TiDB-backend mode [#30953](https://github.com/pingcap/tidb/pull/30953) + + + Dumpling + + - Improve compatibility when Dumpling checks the database version [#29500](https://github.com/pingcap/tidb/pull/29500) + - Add the default collation when dumping `CREATE DATABASE` and `CREATE TABLE` [#3420](https://github.com/pingcap/tiflow/issues/3420) + +## Bug fixes + ++ TiDB + + - Fix the issue of the `tidb_analyze_version` value change that occurs when upgrading the cluster from v4.x to v5.x [#25422](https://github.com/pingcap/tidb/issues/25422) + - Fix the issue of the wrong result that occurs when using different collations in a subquery [#30748](https://github.com/pingcap/tidb/issues/30748) + - Fix the issue that the result of `concat(ifnull(time(3))` in TiDB is different from that in MySQL [#29498](https://github.com/pingcap/tidb/issues/29498) + - Fix the issue of potential data index inconsistency in optimistic transaction mode [#30410](https://github.com/pingcap/tidb/issues/30410) + - Fix the issue that the query execution plan of IndexMerge is wrong when an expression cannot be pushed down to TiKV [#30200](https://github.com/pingcap/tidb/issues/30200) + - Fix the issue that concurrent column type change causes inconsistency between the schema and the data [#31048](https://github.com/pingcap/tidb/issues/31048) + - Fix the issue that the IndexMerge query result is wrong when there is a subquery [#30913](https://github.com/pingcap/tidb/issues/30913) + - Fix the panic issue that occurs when the FetchSize is set too large in the client [#30896](https://github.com/pingcap/tidb/issues/30896) + - Fix the issue that LEFT JOIN might be mistakenly converted to INNER JOIN [#20510](https://github.com/pingcap/tidb/issues/20510) + - Fix the issue that panic might occur when the `CASE-WHEN` expression and collation are used together [#30245](https://github.com/pingcap/tidb/issues/30245) + - Fix the issue of wrong query result that occurs when the `IN` value contains a binary constant [#31261](https://github.com/pingcap/tidb/issues/31261) + - Fix the issue of wrong query result that occurs when CTE has a subquery [#31255](https://github.com/pingcap/tidb/issues/31255) + - Fix the issue that executing the `INSERT ... SELECT ... ON DUPLICATE KEY UPDATE` statement gets panic [#28078](https://github.com/pingcap/tidb/issues/28078) + - Fix the issue that INDEX HASH JOIN returns the `send on closed channel` error [#31129](https://github.com/pingcap/tidb/issues/31129) + ++ TiKV + + - Fix the issue that the MVCC deletion records are not cleared by GC [#11217](https://github.com/tikv/tikv/issues/11217) + - Fix the issue that retrying prewrite requests in the pessimistic transaction mode might cause the risk of data inconsistency in rare cases [#11187](https://github.com/tikv/tikv/issues/11187) + - Fix the issue that GC scan causes memory overflow [#11410](https://github.com/tikv/tikv/issues/11410) + - Fix the issue that RocksDB flush or compaction causes panic when the disk capacity is full [#11224](https://github.com/tikv/tikv/issues/11224) + ++ PD + + - Fix the issue that Region statistics are not affected by `flow-round-by-digit` [#4295](https://github.com/tikv/pd/issues/4295) + - Fix the issue that the scheduling operator cannot fail fast because the target store is down [#3353](https://github.com/tikv/pd/issues/3353) + - Fix the issue that Regions on offline stores cannot be merged [#4119](https://github.com/tikv/pd/issues/4119) + - Fix the issue that the cold hotspot data cannot be deleted from the hotspot statistics [#4390](https://github.com/tikv/pd/issues/4390) + ++ TiFlash + + - Fix the issue that TiFlash might panic when an MPP query is stopped + - Fix the issue that queries with the `where ` clause return wrong results + - Fix the potential issue of data inconsistency that might occur when setting the column type of an integer primary key to a larger range + - Fix the issue that when an input time is earlier than 1970-01-01 00:00:01 UTC, the behavior of `unix_timestamp` is inconsistent with that of TiDB or MySQL + - Fix the issue that TiFlash might return the `EstablishMPPConnection` error after it is restarted + - Fix the issue that the `CastStringAsDecimal` behavior is inconsistent in TiFlash and in TiDB/TiKV + - Fix the issue that the `DB::Exception: Encode type of coprocessor response is not CHBlock` error is returned in the query result + - Fix the issue that the `castStringAsReal` behavior is inconsistent in TiFlash and in TiDB/TiKV + - Fix the issue that the returned result of the `date_add_string_xxx` function in TiFlash is inconsistent with that in MySQL + ++ Tools + + + Backup & Restore (BR) + + - Fix the potential issue that Region distribution might be uneven after a restore operation is finished [#30425](https://github.com/pingcap/tidb/issues/30425) + - Fix the issue that `'/'` cannot be specified in endpoint when `minio` is used as the backup storage [#30104](https://github.com/pingcap/tidb/issues/30104) + - Fix the issue that system tables cannot be restored because concurrently backing up system tables makes the table name fail to update [#29710](https://github.com/pingcap/tidb/issues/29710) + + + TiCDC + + - Fix the issue that replication cannot be performed when `min.insync.replicas` is smaller than `replication-factor` [#3994](https://github.com/pingcap/tiflow/issues/3994) + - Fix the issue that the `cached region` monitoring metric is negative [#4300](https://github.com/pingcap/tiflow/issues/4300) + - Fix the issue that `mq sink write row` does not have monitoring data [#3431](https://github.com/pingcap/tiflow/issues/3431) + - Fix the compatibility issue of `sql mode` [#3810](https://github.com/pingcap/tiflow/issues/3810) + - Fix the potential panic issue that occurs when a replication task is removed [#3128](https://github.com/pingcap/tiflow/issues/3128) + - Fix the issue of panic and data inconsistency that occurs when outputting the default column value [#3929](https://github.com/pingcap/tiflow/issues/3929) + - Fix the issue that default values cannot be replicated [#3793](https://github.com/pingcap/tiflow/issues/3793) + - Fix the potential issue that the deadlock causes a replication task to get stuck [#4055](https://github.com/pingcap/tiflow/issues/4055) + - Fix the issue that no log is output when the disk is fully written [#3362](https://github.com/pingcap/tiflow/issues/3362) + - Fix the issue that special comments in DDL statements cause the replication task to stop [#3755](https://github.com/pingcap/tiflow/issues/3755) + - Fix the issue that the service cannot be started because of a timezone issue in the RHEL release [#3584](https://github.com/pingcap/tiflow/issues/3584) + - Fix the issue of potential data loss caused by inaccurate checkpoint [#3545](https://github.com/pingcap/tiflow/issues/3545) + - Fix the OOM issue in the container environment [#1798](https://github.com/pingcap/tiflow/issues/1798) + - Fix the issue of replication stop caused by the incorrect configuration of `config.Metadata.Timeout` [#3352](https://github.com/pingcap/tiflow/issues/3352) + + + TiDB Data Migration (DM) + + - Fix the issue that the `CREATE VIEW` statement interrupts data replication [#4173](https://github.com/pingcap/tiflow/issues/4173) + - Fix the issue the schema needs to be reset after a DDL statement is skipped [#4177](https://github.com/pingcap/tiflow/issues/4177) + - Fix the issue that the table checkpoint is not updated in time after a DDL statement is skipped [#4184](https://github.com/pingcap/tiflow/issues/4184) + - Fix a compatibility issue of the TiDB version with the Parser version [#4298](https://github.com/pingcap/tiflow/issues/4298) + - Fix the issue that syncer metrics are updated only when querying the status [#4281](https://github.com/pingcap/tiflow/issues/4281) + + + TiDB Lightning + + - Fix the issue of wrong import result that occurs when TiDB Lightning does not have the privilege to access the `mysql.tidb` table [#31088](https://github.com/pingcap/tidb/issues/31088) + - Fix the issue that some checks are skipped when TiDB Lightning is restarted [#30772](https://github.com/pingcap/tidb/issues/30772) + - Fix the issue that TiDB Lighting fails to report the error when the S3 path does not exist [#30674](https://github.com/pingcap/tidb/pull/30674) + + + TiDB Binlog + + - Fix the issue that Drainer fails because it is incompatible with the `CREATE PLACEMENT POLICY` statement [#1118](https://github.com/pingcap/tidb-binlog/issues/1118) diff --git a/releases/release-timeline.md b/releases/release-timeline.md index e1ef19e83a511..6bb121fdb8d4e 100644 --- a/releases/release-timeline.md +++ b/releases/release-timeline.md @@ -9,6 +9,7 @@ This document shows all the released TiDB versions in reverse chronological orde | Version | Release Date | | :--- | :--- | +| [5.4.0](/releases/release-5.4.0.md) | 2022-02-15 | | [5.0.6](/releases/release-5.0.6.md) | 2021-12-31 | | [4.0.16](/releases/release-4.0.16.md) | 2021-12-17 | | [5.1.3](/releases/release-5.1.3.md) | 2021-12-03 | diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 3f7f09841c6cb..69e393658d051 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -608,6 +608,13 @@ Configuration items related to Raftstore. + Default value: `"4h"` + Minimum value: `0` +## `snap-generator-pool-size` New in v5.4.0 + ++ Configures the size of the `snap-generator` thread pool. ++ To make Regions generate snapshot faster in TiKV in recovery scenarios, you need to increase the count of the `snap-generator` threads of the corresponding worker. You can use this configuration item to increase the size of the `snap-generator` thread pool. ++ Default value: `2` ++ Minimum value: `0` + ### `lock-cf-compact-interval` + The time interval at which TiKV triggers a manual compaction for the Lock Column Family