RELEASE-NOTES.md

Release 5.5.1-SNAPSHOT

API Change

1. Authority: Use UserConfiguration instead of ShardingSphereUser in AuthorityConfiguration for decouple API and implementation - #33073

New Feature

1. Kernel: Add new doris, hive and presto sql parser module and database type - #31256

Enhancement

1. Encrypt: Add unsupported check for combine statement with encrypt columns - #30916
2. SQL Parser: Support PostgreSQL COLLATION and CONSTRAINT keywords - #31027
3. SQL Parser: Support Mysql change replication with no server ids - #31110
4. SQL Parser: Support Mysql row and column aliases with ON DUPLICATE KEY UPDATE - #31132
5. SQL Parser: Support Mysql CALL sql with mysql prefix - #31133
6. SQL Parser: Support Mysql create loadable function returns int/dec - #31135
7. SQL Parser: Support parsing MySQL CREATE TABLESPACE/TABLE in Mysql-8 - #31137
8. SQL Parser: Support parsing MySQL CREATE INDEX with ENGINE_ATTRIBUTE - #31140
9. SQL Parser: Support parsing MySQL LOAD DATA with @ variable - #31142
10. SQL Parser: Support parsing MySQL START REPLICA statement - #31152
11. SQL Parser: Support parsing MySQL json table function - #31158
12. Sharding: Revise all local index for sharding table and add object uniqueness level spi to control index token generator - #31176
13. SQL Parser: Enhance mysql json function visit statement result - #31190
14. SQL Federation: Support sql federation bit_count function for mysql - #31701
15. SQL Federation: Improve atan and anan2 sql function for mysql - #31924
16. Kernel: Support sql hint extract when sql contains dbeaver hint comment - #32331
17. Encrypt: Support select distinct(column) encrypt rewrite and refactor SubstitutableColumnNameToken build logic - #32344
18. SQL Federation: Support federated query mysql bin function - #32373
19. SQL Parser: Enhance MySQL select statement - #32395
20. Kernel: Add extract combine left select in extractFromSelectStatementWithoutProjection method - #32429
21. Proxy Native: Add more graalvm reachability metadata for caffeine cache - #32633
22. Sharding: Add NullsOrderType.LOW and NullsOrderType.HIGH to handle NULL order by in sharding feature - #32813
23. SQL Parser: Enhance Oracle update statement as alias parse and add test case - #32823
24. SQL Federation: Support for federated query NOT operator - #32961
25. Encrypt: Support like concat nested concat statement rewrite with encrypt feature - #32970
26. SQL Parser: Support PostgreSQL, openGauss function table and update from segment parse - #32994
27. SQL Parser: Support MariaDB DML Returning Clause - #33094
28. DistSQL: Support running DistSQL under Proxy Native in the form of GraalVM Native Image - #33095
29. DistSQL: Support connecting to Postgres via DistSQL in Proxy Native - #33124
30. DistSQL: Check duplicate actual data nodes when creating or altering sharding table rule - #33138
31. DistSQL: Check user privilege when registering or altering storage unit - #32172
32. Infra: Enable Safe Composition of Metadata for ShardingSphere Proxy Native - #33179
33. Infra: Support compiling and using ShardingSphere under OpenJDK 23 - #33025
34. Hive: Support Hive integration module to connect to HiveServer2 4.0.1 - #33212
35. Infra: Support building Example module with OpenJDK 23 - #33224
36. Proxy Native: Change the Base Docker Image of ShardingSphere Proxy Native - #33263

Bug Fix

1. SQL Federation: Fix federated query LocalDateTime conversion - #30935
2. SQL Federation: Fix push down sql execute error when sql contains chinese character with sql federation - #30997
3. Encrypt: Fix show create table wrong result with encrypt when data type contains float - #31024
4. SQL Parser: Support PostgreSQL do nothing conflict action - #31048
5. Kernel: Replace ThreadLocal with TransmittableThreadLocal in HintManager to support cross Thread usage - #31286
6. SQL Federation: Fix federation query binary type data query - #31346
7. Kernel: Use ConnectionPropertiesParser spi to parse jdbcUrl to solve oracle url parse error - #31746
8. Sharding: Fix Sharding column not tracked through aliases - #31687
9. SQL Binder: Add TableAvailable interface for CloseStatementContext, MoveStatementContext, FetchStatementContext and fix sql rewrite test case - #31756
10. Kernel: Fix column count assert exception caused by postgresql system table judge - #32163
11. SQL Federation: Fix null result in federated query for a single projected column - #32444
12. SQL Parser: Fix oracle nvl function and interval hour parse error - #32549
13. Sharding: Fix alter view exception when config sharding rule and binding table rule - #32700
14. SQL Federation: Fix sql federation unknown type exception caused by calcite wrong result type with bigint - #32730
15. Shadow: Use hintValueContext to replace extract sql hint from sql statement for solving shadow sql hint bug - #33063
16. Shadow: Make Shadow feature available again in GraalVM Native Image - #33080
17. DistSQL & Kernel: Fix table exist exception when execute preview create table statement with exist table - #33171

Change Log

1. MILESTONE

Release 5.5.0

API Change

1. Proxy: Remove the outdated schemaName configuration of the Proxy
2. Kernel: Remove the sqlCommentParseEnabled configuration in SQL Parser Rule to reduce code complexity
3. SQL Parser: Consider removing sqlCommentParseEnabled config in SQLParser rule
4. Federation: Add allQueryUseSQLFederation config for sql federation

New Feature

1. DistSQL: New syntax for query plugin (SPI) implementation
2. DistSQL: New syntax for managing SQL_TRANSLATOR rule

Enhancement

1. Pipeline: Improve CDC stability and performance
2. Pipeline: MemoryPipelineChannel supports configurable zero queue size for less memory consumption
3. Pipeline: Show dedicated error when mode type is not Cluster
4. Authority: Add isSuper option for user
5. Pipeline: Ignore error message in status DistSQL result on job cancelling
6. DistSQL: Use JSON format to output props in RQL
7. DistSQL: Optimize REFRESH DATABASE METADATA logic
8. DistSQL: SHOW COMPUTE NODES supports JDBC nodes
9. DistSQL: Optimize syntax of REFRESH DATABASE METADATA
10. DistSQL: Optimize the output of SHOW TABLE METADATA
11. Proxy: Add MySQL precompiled parameter verification to avoid turning on rewriteBatchedStatements=true when BenchmarkSQL connects to Proxy, causing an ArrayIndexOutOfBoundsException exception when the Proxy parameterCount exceeds 65535.
12. SQL Parser: Add EOF to throw exception when parse distsql rollback migration statement
13. SQL Parser: Support more plsql statement parse and add plsql parse assert logic
14. SQL Parser: Support parse index hint
15. SQL Parser: Support mysql intersect combine operation sql parse
16. SQL Parser: Supoort parse chinese white space for oracle
17. SQL Parser: Fix mysql TimeStampDiff function parse
18. SQL Parser: Fix sqlServer unqualified shorthand parsing
19. SQL Parser: Support sqlServer SEARCH unreserved word parsing
20. SQL Parser: Add sql server MEMBER unreserved word
21. Binder: Add ParameterMarkerSegmentBinder logic for Oracle MergeStatementBinder
22. Federation: Support mysql cross join statement for sql federation
23. Transaction: Add advice message in XATransactionCheckPrivilegeFailedException
24. Sharding: Validate duplicate sharding actual data nodes
25. Sharding: Support null condition value routing
26. Proxy: Fix mysql client multi statements option in protocol

Bug Fix

1. Pipeline: Fix commit/rollback migration job doesn't drop related consistency check job when check job is not completed
2. Pipeline: Fix show consistency check status stop_time display
3. Proxy: Restore original databaseName in connectionSession after unicast
4. Proxy: Fix show tables can be executed without use database
5. Proxy: Fix the incorrect current database after unicast routing
6. Proxy: Fix the problem that show tables can be executed before use database
7. Readwrite-splitting: Fix check exception when using shadow data source
8. Governance: Fix thread blocking problem when create logic database for Etcd register center
9. Governance：Fix register storage units and create feature rules failure when use Standalone mode
10. MetaData: Fix database system schema is not built when create database
11. Metadata: Fix NPE of system schema builder rule
12. SQL Parser: Fix PostgreSQL NPE when parse columnRef
13. SQL Parser: Fix npe cause by parse MySQL select window statement
14. Sharding: Fix routing error when joining tables in uppercase
15. Sharding: Fix drop sharding table exception when table name is uppercase
16. Sharding: Fix generated key with upper case column name
17. Federation: Fix Object 'DUAL' not found exception when execute select 1 from dual with sql federation
18. Transaction: Fix xa auto commit in executeQuery
19. Proxy: Fix multi statements with specified database name

Change Log

1. MILESTONE

Release 5.4.1

New Feature

1. Proxy: Supports dbcp and c3p0 connection pools
2. Metadata: Standalone mode adapts to metadata new structure
3. Governance: Governance supports register instance level data source

Enhancement

1. Mode: Improve Standalone mode JDBC type impl reset data on initialization
2. Scaling: Isolate ShardingSphereDataSource Standalone repository in pipeline
3. Scaling: Disable system-schema-metadata-enabled in pipeline
4. JDBC: Move jdbc core META-INF/services/java.sql.Driver from test to main
5. Scaling: Add algorithm columns in SHOW MIGRATION CHECK ALGORITHMS DistSQL result
6. Scaling: Add type_aliaes column in SHOW MIGRATION CHECK ALGORITHMS DistSQL result
7. Encrypt: Add duplicate name check for derived columns and logical columns
8. Encrypt: Deny DDL for cipher columns in Proxy
9. Encrypt: Add the default type for derived columns to varchar(4000)

Bug Fix

1. Scaling: Fix get inventory position not correctly on breakpoint resuming when table names are similar
2. Scaling: Fix CDC importer not start on breakpoint resuming when first inventory task is finished
3. Single Table: Fix not switched active version when CREATE/DROP table modifies the Single rule configuration
4. JDBC: Fix JDBC memory leak with MySQL in the 5.4.0 version

Change Log

1. MILESTONE

Release 5.4.0

API Changes

1. Metadata: Change sharding broadcast tables to global broadcast tables
2. JDBC: Remove exclamation mark (!) for global rules
3. DistSQL: Simplify keywords ASSISTED_QUERY, LIKE_QUERY in encrypt DistSQL
4. DistSQL: Optimize SQL_PARSER rule syntax
5. Encrypt: Adjust encryption yaml API to distinguish between encrypt, like, and assisted query configurations
6. Encrypt: Remove plain column and queryWithCipherColumn configuration in encrypt feature
7. Readwrite-splitting: Refactor read/write splitting api
8. Proxy: Remove property proxy-instance-type configuration
9. Proxy: Remove property proxy-backend-executor-suitable
10. Proxy: Remove property proxy-mysql-default-version
11. Scaling: Refactor commit rollback streaming to drop streaming
12. Sharding: Merge ShardingCacheRule into ShardingRule (Experimental)

New Features

1. DistSQL: New syntax to manage SQL_FEDERATION rule
2. Proxy: Support Unix Domain Socket

Enhancements

1. Scaling: CDC supports pure incremental mode
2. Scaling: CDC supports exporting data by transaction
3. Scaling: CDC supports MySQL and PostgreSQL
4. Scaling: CDC supports single table
5. Scaling: CDC supports all openGauss data types
6. Scaling: CDC supports replication reconnection
7. Scaling: Remove DataConsistencyCalculateAlgorithmChooser
8. Scaling: Improve performance of integer unique key table inventory data splitting
9. Scaling: Adjust process configuration default value to reduce resource consumption
10. Scaling: Auto refresh table metadata for migration
11. Scaling: Compatible with openGauss existing replication slot reuse when database not existing
12. Scaling: Show data consistency check status result should be empty when it's not completed
13. Scaling: Enable concurrent CRC32 match on source and target
14. Scaling: Pipeline job compatible with sharding rule audit strategy
15. Metadata: Refactor metadata persistence structure
16. Metadata: Optimize the process of loading single table metadata
17. Metadata: Support MySQL/PostgreSQL/openGauss system tables empty query
18. DistSQL: Add support for transactionalReadQueryStrategy for read/write splitting rule
19. DistSQL: Enhanced algorithm properties check
20. Transaction: Add privilege check
21. Transaction: Remove the TransactionTypeHolder and only create the current transaction manager
22. Parser: Support MySQL LOAD DATA and LOAD XML statement with single table or broadcast table
23. Parser: Improve the parsing support of high-priority SQL statements in the test results of the MySQL test program
24. Parser: Oracle dialect parser now supports Chinese comma
25. Encrypt: Support query of encrypt column in projection subquery when use encrypt feature
26. Kernel: Adds table existence metadata check for INSERT, DELETE, UPDATE and SELECT statements
27. JDBC: Implement batch execution for ShardingSphereStatement
28. Proxy: Frontend supports SSL/TLS
29. Proxy: Support Flush message for PostgreSQL/openGauss Proxy
30. Proxy: Support data type bit, bool for PostgreSQL Proxy

Bug Fixes

1. Scaling: Fix pipeline job failure status persistence and usage
2. Scaling: Fix CDC DELETE event Record.beforeList is null
3. Scaling: Fix openGauss mppdb decoding plugin single quote issue
4. Scaling: Fix execute engine not closed after job stopping
5. Scaling: Fix stop job before task starting
6. Metadata: Fix case sensitive issue when loading schema meta data with H2 database
7. Metadata: Fix "object not found" exception when config PostgreSQL/openGauss schema name as database name
8. DistSQL: Fix wrong result of check_table_metadata_enabled when execute SHOW DIST VARIABLE
9. Encrypt: Fix SQL rewrite exception when use PostgreSQL/openGauss encrypt like feature
10. Sharding: Support null sharding condition pass to sharding algorithm to allow user control null value route
11. Parser: Support BETWEEN AND expression parsing in MySQL Projection
12. Mask: Fix wrong mask result when config same value of from-x and to-y with KEEP_FROM_X_TO_Y
13. Infra: Fix ClassNotFoundException may occur when missing pgjdbc
14. Proxy: Fix MySQL packet out of order when client sending pipelining requests

Change Log

1. MILESTONE

5.3.2

API Changes

1. Proxy: Add property system-log-level, support dynamic change of log level by DistSQL.
2. DistSQL: Remove Hint-related DistSQL, users can use SQL Hint instead

New Features

1. Scaling: Support any type of column unique key table

Enhancements

1. Scaling: Use stream query for inventory dump and data consistency check
2. Scaling: Compatible with VARBINARY column type in MySQL binlog parsing
3. Scaling: Refactor AbstractSimplePipelineJob.execute to blocking
4. Scaling: Improve table records count calculation
5. Scaling: Support proxy sharding rule absent for migration job
6. Scaling: Add useServerPrepStmts=false for pipeline job on MySQL
7. Scaling: Improve datetime/time parsing for MySQL binlog
8. Scaling: Add global status for prepare stage
9. Scaling: Add netTimeoutForStreamingResults for pipeline job on MySQL
10. Authority: Support specifying password authentication method
11. Authority: Add md5 authentication support for openGauss protocol
12. Agent: Add more metrics for JDBC
13. Kernel: ShardingSphere Driver configuration supports Apollo
14. Kernel: Adjust SKIP_ENCRYPT_REWRITE SQL Hint to SKIP_SQL_REWRITE to support more scenarios
15. Kernel: Support openGauss EXPLAIN PERFORMANCE
16. Encrypt: Like supports concat function

Bug Fixes

1. Scaling: Fix unicode char and special char decoding for PostgreSQL incremental task
2. Scaling: Fix Migration not support PostgreSQL json type
3. DistSQL: CREATE SHARDING TABLE RULE supports NONE strategy
4. Kernel: Fix use system database error when data sources are empty
5. Kernel: Fix set worker-id does not take effect with Standalone mode
6. Kernel: Clear storage node information when delete readwrite-splitting and database discovery rules
7. Kernel: Fix the abnormal problem of Column index out of range in single table complex query
8. Kernel: Fix PostgreSQL like lower case failed.
9. Kernel: Fixed the exception of built-in metabase data collection when the front and back database types were inconsistent
10. Kernel: Fix the problem of routing error reporting under certain table names
11. Kernel: Fix MySQL create procedure parse error
12. Kernel: Fix union extract table name NPE
13. Kernel: Fix upper case table constraint not rewrite error
14. Kernel: Fix failed to parse PostgreSQL / openGauss SQL contains money type
15. Kernel: Fix PostgreSQL / openGauss positional parameter rule
16. Kernel: Fix PostgreSQL / openGauss failed to parse const with type cast
17. Kernel: Fix Chinese characters encode exception when execute select with sql federation engine
18. Kernel: Fix IndexOutOfBoundsException when execute set variable statement int jdbc adapter
19. Kernel: Fix index does not exist exception when execute drop index statement
20. Proxy: Properly handle number sign in MySQL Proxy binary protocol
21. Proxy: Fix PostgreSQL Proxy failed to handle bytea data type
22. Proxy: Fix PostgreSQL Proxy failed to parse time value with microseconds
23. Proxy: Fix PostgreSQL protocol codec for date type in binary format
24. Proxy: Fix possible CCE PostgreSQLInt2BinaryProtocolValue
25. Proxy: Fix possible error when client pass quoted charset to PostgreSQL/openGauss Proxy

Change Log

1. MILESTONE

5.3.1

New Features

1. Kernel: Add new data masking, dynamic data masking features, and built-in data masking algorithms
2. Scaling: Basic support of CDC feature
3. DistSQL: Add masking rule related DistSQL

Enhancements

1. Kernel: Cluster mode avoids secondary refresh of metadata
2. Kernel: SHOW COMPUTE NODES supports displaying the version number of each instance
3. Kernel: System database add cluster information table
4. Kernel: Standalone mode persistent metadata supports MySQL
5. Kernel: SQL HINT performance improvement
6. Kernel: Restore routing to the specified database with Hint
7. Encrypt: Supports underscore wildcards for Encrypt CharDigestLikeEncryptAlgorithm
8. Kernel: Support SQL federation SELECT NULLS LAST/FIRST statement
9. Kernel: Refactor encrypt integration test logic and add more test cases
10. Kernel: Add salt props for MD5MaskAlgorithm, MD5EncryptAlgorithm
11. Kernel: Refactor ShardingConditionEngine to support SPI configuration
12. DistSQL: Add algorithm type check for CREATE SHARDING TABLE RULE

Bug Fixes

1. Fix the problem of ZooKeeper cluster error reporting when ShardingSphere connects to Kubernetes
2. Kernel: Fix use Consul in cluster mode start up failure
3. DB Discovery: Close heartbeat job when drop discovery rule
4. Kernel: Fix wrong decide result when execute same sharding condition subquery with SQL federation
5. Kernel: Fix priority problem of UNION, INTERSECT, EXCEPT set operation in SQL Federation for PostgreSQL and openGauss dialect
6. Kernel: Fix create view index out of range exception when view contains set operator
7. Kernel: Add XA resource exceeds length check
8. Kernel: Fix transaction support for spring requires_new
9. Encrypt: Fix AESEncryptAlgorithm decrypt exception when config char type with PostgreSQL and openGauss
10. Encrypt: Fix abnormal expansion result for shorthand when encrypt subquery contains an alias
11. Kernel: Fix unsigned flag of column metadata was not serialized
12. Kernel: Fix PostgreSQL / openGauss select fetch parsing issue to support federation execution engine
13. Proxy: Fix packet sequence ID may be incorrect if error occurred in MySQL Proxy
14. Proxy: Fix error occur in Proxy when using PostgreSQL composite type
15. Proxy: Set proper column definition flag for MySQL COM_STMT_PREPARE
16. Proxy: When querying PG metadata through Proxy and the result set is empty, the labels are lost

Change Log

1. MILESTONE

5.3.0

API Changes

1. DistSQL: Refactor syntax API, please refer to the user manual
2. Proxy: Change the configuration style of global rule, remove the exclamation mark
3. Proxy: Allow zero-configuration startup, enable the default account root/root when there is no Authority configuration
4. Proxy: Remove the default logback.xml and use API initialization
5. JDBC: Remove the Spring configuration and use Driver + YAML configuration instead

Enhancements

1. DistSQL: New syntax REFRESH DATABASE METADATA, refresh logic database metadata
2. Kernel: Support DistSQL REFRESH DATABASE METADATA to load configuration from the governance center and rebuild MetaDataContext
3. Support postgresql/openGauss setting transaction isolation level
4. Scaling: Increase inventory task progress update frequency
5. Scaling: DATA_MATCH consistency check support breakpoint resume
6. Scaling: Support drop consistency check job via DistSQL
7. Scaling: Rename column from sharding_total_count to job_item_count in job list DistSQL response
8. Scaling: Add sharding column in incremental task SQL to avoid broadcast routing
9. Scaling: Sharding column could be updated when generating SQL
10. Scaling: Improve column value reader for DATA_MATCH consistency check
11. DistSQL: Encrypt DistSQL syntax optimization, support like query algorithm
12. DistSQL: Add properties value check when REGISTER STORAGE UNIT
13. DistSQL: Remove useless algorithms at the same time when DROP RULE
14. DistSQL: EXPORT DATABASE CONFIGURATION supports broadcast tables
15. DistSQL: REGISTER STORAGE UNIT supports heterogeneous data sources
16. Encrypt: Support Encrypt LIKE feature
17. Automatically start distributed transactions when executing DML statements across multiple shards
18. Kernel: Support client \d for PostgreSQL and openGauss
19. Kernel: Support select group by, order by statement when column contains null values
20. Kernel: Support parse RETURNING clause of PostgreSQL/openGauss Insert
21. Kernel: SQL HINT performance improvement
22. Kernel: Support mysql case when then statement parse
23. Kernel: Supporting data source level heterogeneous database gateway
24. (Experimental) Sharding: Add sharding cache plugin
25. Proxy: Support more PostgreSQL datetime formats
26. Proxy: Support MySQL COM_RESET_CONNECTION
27. Scaling: Improve MySQLBinlogEventType.valueOf to support unknown event type
28. Kernel: Support case when for federation

Bug Fix

1. Scaling: Fix barrier node created at job deletion
2. Scaling: Fix part of columns value might be ignored in DATA_MATCH consistency check
3. Scaling: Fix jdbc url parameters are not updated in consistency check
4. Scaling: Fix tables sharding algorithm type INLINE is case-sensitive
5. Scaling: Fix incremental task on MySQL require mysql system database permission
6. Proxy: Fix the NPE when executing select SQL without storage node
7. Proxy: Support DATABASE_PERMITTED permission verification in unicast scenarios
8. Kernel: Fix the wrong value of worker-id in show compute nodes
9. Kernel: Fix route error when the number of readable data sources and weight configurations of the Weight algorithm are not equal
10. Kernel: Fix multiple groups of readwrite-splitting refer to the same load balancer name, and the load balancer fails problem
11. Kernel: Fix can not disable and enable compute node problem
12. JDBC: Fix data source is closed in ShardingSphereDriver cluster mode when startup problem
13. Kernel: Fix wrong rewrite result when part of logical table name of the binding table is consistent with the actual table name, and some are inconsistent
14. Kernel: Fix startup exception when use SpringBoot without configuring rules
15. Encrypt: Fix null pointer exception when Encrypt value is null
16. Kernel: Fix oracle parsing does not support varchar2 specified type
17. Kernel: Fix serial flag judgment error within the transaction
18. Kernel: Fix cursor fetch error caused by wasNull change
19. Kernel: Fix alter transaction rule error when refresh metadata
20. Encrypt: Fix EncryptRule cast to TransparentRule exception that occurs when the call procedure statement is executed in the Encrypt scenario
21. Encrypt: Fix exception which caused by ExpressionProjection in shorthand projection
22. Proxy: Fix PostgreSQL Proxy int2 negative value decoding incorrect
23. Proxy: PostgreSQL/openGauss support describe insert returning clause
24. Proxy: Fix gsql 3.0 may be stuck when connecting Proxy
25. Proxy: Fix parameters are missed when checking SQL in Proxy backend
26. Proxy: Enable MySQL Proxy to encode large packets
27. Kernel: Fix oracle parse comment without whitespace error
28. DistSQL: Fix show create table for encrypt table

Refactor

1. Scaling: Reverse table name and column name when generating SQL if it's SQL keyword
2. Scaling: Improve increamental task failure handling
3. Kernel: Governance center node adjustment, unified hump to underscore

Change Log

1. MILESTONE

5.2.1

New Feature

1. Add ShardingSphere default system database to support global metadata management
2. Support asynchronous data consistency check
3. Added support for Consul governance center
4. Added support for Nacos governance center
5. Added support for the view function in the governance center

Enhancement

1. SQL Federation engine adds ADVANCED executor and adapts to openGauss database
2. Support ShardingSphere Proxy startup after read-write splitting read database is disabled
3. SQL HINT supports force sharding route
4. Show processlist supports showing Proxy connections (sleep, active)
5. Optimized ShardingSphere-JDBC data source configuration error message
6. ShardingSphere-JDBC supports SpringBoot 3.x version
7. Support load MySQL, PostgreSQL, openGauss and SQLServer view metadata
8. Update Snakeyaml to 1.33 and open YAML 3MB limit
9. Reuse cached connections as possible when unicast sharding
10. Support Parsing ALTER ROLE in Oracle
11. Add support of ALTER RESOURCE COST for Oracle
12. Support parsing Drop Materialized View in Oracle
13. Support parsing DROP LIBRARY in Oracle
14. Support parsing DROP JAVA in Oracle
15. Support parsing DROP PLUGGABLE DATABASE in Oracle
16. Support parsing DROP INDEX TYPE in Oracle
17. Support Parsing ALTER PACKAGE in openGauss
18. Support openGauss select offset, count statement parse and remove useless syntax in PostgreSQL grammar
19. Add max_size to openGauss syntax
20. Optimize alter view/drop view parse logic and fix alter view refresher bug
21. Add sql parser error detail to ParseCancellationException
22. Add support for parse OptOnConflict for postgres
23. Enhance support for ALTER TABLE and ALTER VIEW in PostgreSQL
24. Add a missing keyword in PostgreSQL Declare Statement
25. Add json function support to mysql parser
26. ShardingSphere-Proxy automatically adapts to cgroup memory limits in Docker environment
27. Show migration status DistSQL respond new added error_message column
28. Show migration status respond new added processed_records_count column
29. Support MySQL 8 caching_sha2_password authentication in incremental dump
30. Improve drop pipeline process configuration
31. Support unique key table migration
32. Support migrate table to new name table
33. Improve thread pool usage in pipeline task and runner
34. Support cancelable data consistency check
35. DistSQL: When creating or altering readwrite-splitting rule, check duplicate write or read resources
36. DistSQL: Add validity check for ALTER SHARDING BINDING TABLE RULES
37. Standalone mode H2 support persistent metadata
38. Fix openGauss cursor execution in xa transaction
39. Added transaction related exceptions

Bug Fix

1. Generate proper placeholder for PostgreSQL when rewriting
2. Fix opengauss update set parse exception
3. Fix parse exception when execute insert statement with negative value
4. Fix wrong connectDescriptorUrlPattern in OracleDataSourceMetaData
5. Fix insert SQL garbled after sharding rewrote in special rules
6. Fix exception when execute select * from information_schema.tables
7. Fix exception when execute alter view rename
8. Fix PostgreSQL check data source permission when using rolsuper
9. DistSQL: fix NPE for REFRESH TABLE METADATA when there is no resource
10. Fix Unmodified table metadata when modify rules
11. Fix database discovery
12. The MySQL.NORMAL_REPLICATION algorithm cannot find primary node"
13. Fixed using etcd to build a cluster event not aware
14. Fix NPE occurred when transaction management is not created
15. Fix routing exception that occurs when the sharding value of the InlineShardingAlgorithm algorithm exceeds Integer

API Changes

1. SQL HINT syntax format adjust to SQL-style format
2. DistSQL: Remove syntax COUNT DATABASE RULES
3. ShardingSphere mode remove overwrite configuration
4. Agent: Optimize configuration of agent.yaml

Change Log

1. MILESTONE

5.2.0

New Feature

1. Support SQL audit for sharding feature
2. Support MySQL show processlist and kill process id feature
3. Scaling: Add dedicated DistSQL for data migration
4. Scaling: Basic support migrate data to heterogeneous database
5. DistSQL: New syntax CREATE MIGRATION PROCESS CONFIGURATION
6. DistSQL: New syntax ALTER MIGRATION PROCESS CONFIGURATION
7. DistSQL: New syntax SHOW MIGRATION PROCESS CONFIGURATION
8. DistSQL: New syntax ADD MIGRATION SOURCE RESOURCE
9. DistSQL: New syntax SHOW SQL_TRANSLATOR RULE
10. DistSQL: New syntax CREATE SHARDING AUDITOR
11. DistSQL: New syntax ALTER SHARDING AUDITOR
12. DistSQL: New syntax SHOW SHARDING AUDIT ALGORITHMS

Enhancement

1. Support column visible feature for MySQL, Oracle, SQLServer and H2
2. Support cartesian product configuration for read write splitting
3. Support spring namespace and spring boot usage for sql translator
4. Support JSR-310 Year and Month in IntervalShardingAlgorithm
5. Support broadcast table update/delete limit statement
6. Support create index on table(column) statement rewrite when config encrypt
7. Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
8. Support encrypt column rewrite when execute column is null in predicate
9. Support encrypt show create table return logic columns
10. Support create table with index statement rewrite when config encrypt
11. Support PostgreSQL create operator statement parse
12. Support PostgreSQL create materialized view statement parse
13. Support PostgreSQL nested comments parse
14. Support PostgreSQL alter subscription statement parse
15. Support PostgreSQL create group statement parse
16. Support PostgreSQL alter statictics statement parse
17. Support PostgreSQL create foreign table statement parse
18. Support PostgreSQL alter server statement parse
19. Support PostgreSQL create foreign data wrapper statement parse
20. Support PostgreSQL create event trigger statement parse
21. Support PostgreSQL security label statement parse
22. Support PostgreSQL reindex statement parse
23. Support PostgreSQL reassign owned statement and refresh materialized view statement parse
24. Support PostgreSQL prepare transaction statement parse
25. Support PostgreSQL create collation statement parse
26. Support PostgreSQL lock statement parse
27. Support PostgreSQL alter rule statement parse
28. Support PostgreSQL notify statement parse
29. Support PostgreSQL unlisten statement parse
30. Support Oracle alter function and alter hierarchy statement parse
31. Support Oracle alter pluggable database statement parse
32. Support Oracle alter materialized view log statement parse
33. Support Oracle alter diskgroup statement parse
34. Support Oracle alter operator statement parse
35. Support oracle alter cluster statement parse
36. Support oracle alter audit policy statement parse
37. Support Oracle alter index type statement parse
38. Support Oracle lock table statement parse
39. Support Oracle alter java statement parse
40. Support Oracle inline constraint statement parse
41. Support openGauss geometric operator statement parse
42. Optimize MySQL visible/invisible parse of create/alter table statements
43. Support scope of variable prefixed with @@ in MySQL SET statement parse
44. Support MySQL create procedure with create view parse
45. Support column segments parse in create index on table statement
46. Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
47. Support encrypt column rewrite when execute column is null in predicate
48. Support encrypt show create table return logic columns
49. Support create table with index statement rewrite when config encrypt
50. Support parsing ALTER LOCKDOWN PROFILE in Oracle
51. Support parsing ALTER MATERIALIZED VIEW in Oracle
52. Support parsing ALTER MATERIALIZED ZONEMAP in Oracle
53. Support parsing ALTER LIBRARY in Oracle
54. Support parsing ALTER INMEMORY JOIN GROUP in Oracle
55. Support parsing DROP OPERATOR in Oracle
56. Support parsing DROP RESTORE POINT in Oracle
57. Support parsing CREATE RESTORE POINT in Oracle
58. Support parsing DROP INMEMORY JOIN GROUP in Oracle
59. Support parsing create_bit_xor_table in MySQL
60. Support parsing MySQL DO statement
61. Support parsing DropServer in openGauss
62. Support parsing CREATE AGGREGATE In openGauss
63. Support parsing ALTER ROUTINE in PostgreSQL
64. Add PostgreSQL Create Cast Statement
65. Add PostgreSQL Create Aggregate Statement
66. Support fetch/move/close cursor statement in PostgreSQL
67. Support Parsing ALTER PUBLICATION in PostgreSQL
68. Add PostgreSQL Create Access Method Statement
69. Support Parsing ALTER POLICY in PostgreSQL
70. Support parsing ALTER OPERATOR in PostgreSQL
71. Add PostgreSQL Copy Statement
72. Add PostgreSQL Comment Statement
73. Support listen statement in PostgreSQL
74. Support DECLARE cursor statement
75. Add default serverConfig in helm charts
76. Assemble openGauss JDBC Driver into Proxy distribution
77. ShardingSphere-Proxy listen on specified IP addresses
78. Support COM_STMT_SEND_LONG_DATA in MySQL Proxy
79. SELECT VERSION() support alias in MySQL Proxy
80. Fix openGauss Proxy could not be connected if no resource defined
81. Support using JRE defined in JAVA_HOME in ShardingSphere-Proxy's startup script
82. Avoid client blocked when OOM occurred in ShardingSphere-Proxy
83. Support using c3p0 in ShardingSphere-JDBC
84. Support SET NAMES with value quoted by double-quote
85. Connection.prepareStatement with columns arguments is available in ShardingSphere-JDBC
86. Scaling: Improve MySQL connect and reconnect
87. Scaling: Fix MySQL json column may cause leak at incremental task
88. Scaling: Add permission check for PostgreSQL data sources
89. Scaling: Incremental migration support for MySQL MGR mode
90. Scaling: Improve job progress persistence
91. Scaling: Start job DistSQL execute and return synchronously
92. Scaling: Inventory migration support table has primary key and unique key
93. Scaling: Close unerlying ElasticJob when stopping job
94. Scaling: Improve logical replication slot name generation for PostgreSQL and openGauss
95. Scaling: Make query DistSQL could be executed when no database selected
96. DistSQL: Add worker_id to the result set of SHOW INSTANCE LIST & SHOW INSTANCE INFO
97. DistSQL: Improve the result of EXPORT DATABASE CONFIG
98. DistSQL: Support more databases for FORMAT SQL
99. DistSQL: Optimize the execution logic of CREATE TRAFFIC RULE
100. DistSQL: Support assistEncryptor for Encrypt RDL
101. DistSQL: Add sharding algorithm type check when CREATE SHARDING TABLE RULE
102. Support database discovery to configure multiple groups of high availability under the same logic database
103. Support ShardingSphere-Proxy to start up under empty logic library
104. Support for isolating EventBus events by instance
105. Support the database to detect changes in the master node and restart the detection heartbeat task
106. Support ShardingSphere-Proxy to generate new worker-id when re-registering in cluster mode
107. Thrown exception when inserting expression value in shadow column on executing insert
108. Support distributed transactions across multiple logical databases
109. Support executing truncate in XA & PostgreSQL
110. Support alter local transaction rule with DistSQL
111. Support global transaction manager
112. Delete support for branch transaction on proxy

Bug Fix

1. Fix single table metadata refresh error caused by filtering DataSourceContainedRule
2. Fix parsing exception caused by the null value of MySQL blob type
3. Fix PostgreSQL/openGauss reset statement parse error
4. Fix wrong parameter rewrite when use sharding and encrypt
5. Fix the failed conversion of Month related classes on IntervalShardingAlgorithm
6. Fix NullPointerException when execute select union statement contains subquery
7. Fix wrong encrypt rewrite result due to incorrect order of metadata
8. Fix MySQL trim function parse error
9. Fix MySQL insert values with _binary parse error
10. Fix MySQL syntax error cannot be thrown to client
11. Avoid EventLoop blocked because of closing JDBC resources
12. Correct server status flags returned by MySQL Proxy
13. Fix a possible connection leak issue if Proxy client disconnected in transaction
14. Fixed a possible consistency issue with the statement being executed when the Proxy client is disconnected
15. Avoid pooled connection polluted by executing SET statements
16. Make SHOW TABLES FROM work in ShardingSphere-Proxy
17. Fix PostgreSQL DDL could not be executed by Extended Query
18. Fix SHOW VARIABLES could not be executed in PostgreSQL Proxy without resource
19. Fix FileNotFoundException when use ShardingSphere Driver with SpringBoot fatjar
20. Scaling: Fix the problem that the table contains both primary key and unique index at inventory migration
21. Scaling: Improve incremental migration, support the latest position in the middle of batch insert event
22. Scaling: Fix the error caused by null field value in openGauss incremental migration
23. DistSQL: Fix incorrect strategy name in result of SHOW SHARDING TABLE RULES
24. DistSQL: Fix current rule config is modified in advance when ALTER SHARDING TABLE RULE
25. DistSQL: Fix connection leak when ALTER RESOURCE
26. DistSQL: Fix CREATE TRAFFIC RULE failed when load balance algorithm is null
27. Fix that the monitoring heartbeat task was not stopped when the database was discovered and the logical library was deleted
28. Fix cluster mode ShardingSphere-JDBC load all logic database
29. Fix worker-id generated by SnowflakeKeyGenerateAlgorithm in cluster mode may exceed the maximum value
30. Fix DistSQL adding shadow algorithm exception without shadow data source
31. Fix cross-database data source confusion caused by same data source name in multiple logical databases
32. Fix RUL DistSQL execution failure in transaction
33. Fix begin for PostgreSQL & openGauss
34. Agent: Fixed the error of null value in contextManager when collecting metric data

API Changes

1. Remove SQL passthrough to data source feature
2. Add new assistedQueryEncryptorName and remove QueryAssistedEncryptAlgorithm interface
3. Refactor readwrite-splitting api to improve user experience
4. Remove check-duplicate-table-enabled configuration
5. Remove useless config item show-process-list-enabled configuration
6. Scaling: Change keyword for part of data migration DistSQL
7. Scaling: Redesign part of data migration DistSQL
8. DistSQL: Unify parameter type specification
9. DistSQL: Split SHOW INSTANCE MODE to SHOW MODE INFO and SHOW INSTANCE INFO
10. DistSQL: Change DROP SCALING jobId to CLEAN MIGRATION jobId
11. DistSQL: Remove COUNT INSTANCE RULES
12. Add database found that high availability supports all the slave libraries to go offline, and the main library undertakes the read traffic configuration
13. SnowflakeKeyGenerateAlgorithm supports configuring worker-id in standalone mode
14. Replace sourceDataSourceName with productionDataSourceName in Shadow API Configuration
15. Authority: Remove deprecated native authority provider

Refactor

1. ShardingSphere metadata refactoring for splitting actual metadata and logical metadata
2. Use ConnectionContext, QueryContext to remove ThreadLocal in FetchOrderByValueQueuesHolder, TrafficContextHolder, SQLStatementDatabaseHolder and TransactionHolder
3. Modify the default value of the ShardingSphere-Proxy version in the helm chart
4. Docker container will exit if ShardingSphere-Proxy failed to startup
5. Helm Charts in ShardingSphere repository are transferred to sub-project shardingsphere-on-cloud
6. Scaling: Plenty of refactor for better code reuse
7. DistSQL: Add a new category named RUL
8. Refactor the schedule module and split it into cluster schedule and standalone schedule
9. Remove memory mode, keep standalone mode and cluster mode
10. Refactoring metadata table loading logic and persistence logic
11. Refactoring distributed locks to retain the most concise interface design
12. Refactor : Unify The Creation for Proxy Containers in IT from ENV Modules
13. Refactor : Unify The Configuration for container created by testcontainer

Change Log

1. MILESTONE

5.1.2

New Feature

1. Kernel: Alpha version to support SQL dialect translate for MySQL and PostgreSQL
2. Kernel: Support custom schema for PostgreSQL and openGauss
3. Kernel: Support create/alter/drop view statement for PostgreSQL and openGauss
4. Kernel: Support openGauss cursor statement
5. Kernel: Support use customize system database
6. Kernel: Support get openGauss and MySQL create SQL
7. Kernel: Support get postgres create SQL
8. Proxy: Official support for quickly deploying a ShardingSphere-Proxy cluster with a ZooKeeper cluster in Kubernetes using Helm
9. JDBC: Support ShardingSphere JDBC Driver
10. Scaling: Support PostgreSQL auto create table
11. Scaling: Support scaling for table with customized schema in PostgreSQL and openGauss
12. Scaling: Support scaling for table with text primary key and without integer primary key
13. Mode: Registry center supports PG/openGauss three-level structure
14. Mode: Registry center supports database-level distributed lock

Enhancement

1. Kernel: Support copy statement for PostgreSQL and openGauss
2. Kernel: Support alter/drop index statement for PostgreSQL
3. Kernel: Support update force index statement for MySQL
4. Kernel: Support create/alter/drop schema for openGauss
5. Kernel: Optimize RoundRobinReplicaLoadBalanceAlgorithm and RoundRobinTrafficLoadBalanceAlgorithm logic
6. Kernel: Optimize metadata loading logic when frontendDatabaseType and backendDatabaseType are different
7. Kernel: Refactor meta data load logic
8. Kernel: Optimize show processlist statement
9. Kernel: Improve performance about large tables loaded
10. Kernel: Support execute comment statement
11. Kernel: Support view in sharding rule
12. Kernel: Support parsing CREATE ROLLBACK SEGMENT in Oracle
13. Kernel: Support Parsing DROP TYPE in openGauss
14. Kernel: Support Parsing ALTER TYPE in openGauss
15. Kernel: Support parsing DROP DISKGROUP in Oracle
16. Kernel: Support parsing CREATE DISKGROUP in Oracle
17. Kernel: Support parsing DROP FLASHBACK ARCHIVE in Oracle
18. Kernel: Support Parsing CHECKPOINT in openGauss
19. Kernel: Support parsing CREATE FLASHBACK ARCHIVE in Oracle
20. Kernel: Add PostgreSQL Close Statement
21. Kernel: Support Parsing DROP CAST in openGauss
22. Kernel: Support parsing CREATE CAST in openGauss
23. Kernel: Support parsing CREATE CONTROL FILE in Oracle
24. Kernel: Support Parsing DROP DIRECTORY in openGauss
25. Kernel: Support parsing ALTER DIRECTORY in openGauss
26. Kernel: Support parsing CREATE DIRECTORY in openGauss
27. Kernel: Add PostgreSQL Checkpoint Statement
28. Kernel: Support parsing DROP SYNONYM in openGauss
29. Kernel: Support parsing CREATE SYNONYM in openGauss
30. Kernel: Support parsing ALTER SYNONYM in openGauss
31. Kernel: Add PostgreSQL CALL Statement
32. Kernel: Support parsing CREATE PFILE in Oracle
33. Kernel: Support parsing CREATE SPFILE in Oracle
34. Kernel: Support parsing ALTER SEQUENCE in Oracle
35. Kernel: Support parsing CREATE CONTEXT in Oracle
36. Kernel: Support Parsing ALTER PACKAGE in oracle
37. Kernel: Support parsing CREATE SEQUENCE in Oracle
38. Kernel: Support parsing ALTER ATTRIBUTE DIMENSION in Oracle
39. Kernel: Support parsing ALTER ANALYTIC VIEW in Oracle
40. Kernel: Use ShardingSphere SPI in SQLVisitorFacade
41. Kernel: Use ShardingSphere SPI in DatabaseTypedSQLParserFacade
42. Kernel: Support parsing ALTER OUTLINE in Oracle
43. Kernel: Support parsing DROP OUTLINE in Oracle
44. Kernel: Support parsing drop edition in oracle
45. Kernel: Support WITH Common Table Expression of SQLServer
46. Kernel: Exclude parenthesis from SubquerySegment's start and stop index in withClause
47. Kernel: Refactor JoinTableSegment
48. Kernel: Support parsing DROP SYNONYM in Oracle
49. Kernel: Support parsing CREATE DIRECTORY in Oracle
50. Kernel: Support parsing CREATE SYNONYM in Oracle
51. Kernel: Support for XmlNamespaces Clause of SQLServer SELECT Statement
52. Kernel: Support parsing Alter Database Dictionary in Oracle
53. Kernel: Support FOR Clause of SQLServer SELECT Statement
54. Kernel: Support Parsing ALTER DATABASE LINK in Oracle
55. Kernel: Support CREATE EDITION Parsing in Oracle
56. Kernel: Support parsing ALTER TRIGGER in Oracle
57. Kernel: Add SQLServer REVERT Statement
58. Kernel: Support Parsing DROP TEXT SEARCH in PostgreSQL
59. Kernel: Add drop server for PostgreSQL
60. Kernel: Support Parsing ALTER VIEW in Oracle
61. Kernel: Add drop access method for PostgreSQL
62. Kernel: Support Parsing DROP ROUTINE in PostgreSQL
63. Kernel: Proofread SQLServer DROP USER Statement
64. Kernel: Support parsing DROP TRIGGER in Oracle
65. Kernel: Support parsing Drop subscription in PostgreSQL
66. Kernel: Add drop operator class for PostgreSQL
67. Kernel: Support parsing DROP PUBLICATION in PostgreSQL
68. Kernel: Support Parsing DROP VIEW in Oracle
69. Kernel: Support Parsing DROP TRIGGER in PostgreSQL
70. Kernel: Support Parsing DROP DIRECTORY in Oracle
71. Kernel: Support Parsing DROP STATISTICS for PostgreSQL
72. Kernel: Add drop type SQL parser for PostgreSQL
73. Kernel: Support Parsing DROP RULE in PostgreSQL
74. Kernel: Proofread SQLServer ALTER LOGIN Statement
75. Kernel: Support parsing PostgreSQL DROP FOREIGN DATA WRAPPER
76. Kernel: Small changes to PostgreSQL DROP EVENT TRIGGER statement
77. Proxy: ShardingSphere-Proxy MySQL supports receiving MySQL packet more than 16 MB
78. Proxy: Supports netty parameter ChannelOption.SO_BACKLOG configurable in ShardingSphere-Proxy
79. Proxy: Optimize so-reuseaddr in netty to solve the problem of port occupied
80. Proxy: Docker image of ShardingSphere-Proxy supports aarch64 platform
81. Proxy: Make server version configurable in ShardingSphere-Proxy MySQL
82. Proxy: Supports more character sets in ShardingSphere-Proxy PostgreSQL/openGauss
83. Proxy: Make default port configurable in ShardingSphere-Proxy
84. Scaling: Compatible with HA ports for openGauss:3.0 when thread_pool enabled
85. Scaling: Optimize ZooKeeper event handling in PipelineJobExecutor to avoid blocking ZooKeeper events
86. Scaling: Make table name case-insensitive in whole process
87. Scaling: Improve replication slot cleanup for PostgreSQL and openGauss
88. Scaling: Improve lock protection for job preparation
89. Scaling: Support PostgreSQL insert on conflict do update
90. Scaling: Do not cache data source in GlobalDataSourceRegistry to avoid possible shared resource close issue
91. Scaling: Reuse data source pool as more as possible to reduce working database connections
92. DistSQL: REFRESH TABLE METADATA supports specifying PostgreSQL's schema
93. DistSQL: ALTER SHARDING TABLE RULE add validation of binding table
94. Mode: ShardingSphere-JDBC supports configuring database connection name
95. Distributed Transaction: DistSQL is prohibited from executing within a transaction
96. Distributed Transaction: autocommit = 0, DDL part of DML will automatically open the transaction

Bug Fix

1. Kernel: Fix parsing error about show statement for PostgreSQL and openGauss
2. Kernel: Fix parsing error about time extract function for for PostgreSQL and openGauss
3. Kernel: Fix parsing error about select mod function for for PostgreSQL and openGauss
4. Kernel: Fix PSQLException when execute join statement with multi schema in readwrite scenario
5. Kernel: Fix wrong route result when execute create schema statement in encrypt scenario
6. Kernel: Fix npe when execute drop schema if exist statement
7. Kernel: Fix wrong route result when execute SELECT LAST_INSERT_ID() AS id; statement
8. Kernel: Fix npe when execute use database when database doesn't contains datasource
9. Kernel: Fix create function with set var
10. Proxy: Fix NPE caused by column's case unmatched in PostgreSQLComDescribeExecutor
11. Proxy: Complete command tags for schema DDL in ShardingSphere-Proxy PostgreSQL / openGauss
12. Scaling: Fix MySQL unsigned type null value cause error during increment task
13. Scaling: Fix resource leak caused by error occurred when creating DataSource in ShardingSphere-Scaling
14. Scaling: Fix ShardingSphereDataSource creation ignoring other rules
15. Scaling: Fix on preparation job could not be stopped
16. Scaling: Fix data source property url and jdbcUrl compatibility
17. Scaling: Fix openGauss logical replication slot creation, avoid possible incremental data loss
18. Scaling: Update local job status before persisting job status to registry center, make sure it won't be overwritten later
19. Scaling: Handling null value in TestDecodingPlugin for PostgreSQL
20. DistSQL: Fix SET VARIABLE modification not taking effect in stand-alone and memory mode
21. DistSQL: Fix the inconsistency between SHOW INSTANCE LIST display data and actual data
22. DistSQL: Fix capitalization sensitivity in sharding DistSQL
23. Mode: Fix the new version metadata lost data after the Scaling changes the table sharding rules
24. Distributed Transaction: Fix getIndexInfo with catalog

API Changes

1. DistSQL: Change EXPORT SCHEMA CONFIG to EXPORT DATABASE CONFIG
2. DistSQL: Change IMPORT SCHEMA CONFIG to IMPORT DATABASE CONFIG
3. DistSQL: Change SHOW SCHEMA RESOURCES to SHOW DATABASE RESOURCES
4. DistSQL: Change COUNT SCHEMA RULES to COUNT DATABASE RULES
5. Mode: Adjust db-discovery algorithm configuration
6. Authority: Authority provider ALL_PRIVILEGES_PERMITTED updated to ALL_PERMITTED
7. Authority: Authority provider SCHEMA_PRIVILEGES_PERMITTED updated to DATABASE_PERMITTED

Refactor

1. Scaling: Refactor JobConfiguration, prepare for different types of jobs reuse and extension
2. Mode: Optimize compute node structure of the registry center
3. Mode: Use uuid instead of ip@port as instance id

Change Log

1. MILESTONE

5.1.1

New Feature

1. Kernel: support alter materialized view for PostgreSQL
2. Kernel: support declare for PostgreSQL
3. Kernel: support discard for PostgreSQL
4. Kernel: Add mode to parser to support $$ in PostgreSQL
5. Kernel: Support MySQL create tablespace statement parse
6. Scaling: Implement stop source writing and restore source writing
7. Scaling: Support partial tables scale-out
8. DistSQL: New DistSQL syntax: SHOW UNUSED RESOURCES
9. Mode: Added persistent XA Recovery Id to Governance Center
10. Mode: Database discovery adds delayed master-slave delay function
11. Distributed Transaction: Add savepoint support for ShardingSphere proxy
12. Distributed Transaction: Support auto rollback when report exception in transaction block for PostgreSQL and openGauss
13. Distributed Transaction: Make it is easy to use with Narayana
14. Distributed Transaction: Add savepoint support for ShardingSphere-JDBC

Enhancement

1. Kernel: Refactor kernel to improve performance
2. Proxy: Reduce Docker image size of ShardingSphere-Proxy
3. Proxy: ShardingSphere-Proxy supports set names statements
4. Proxy: ShardingSphere-Proxy MySQL supports multi statements
5. Scaling: Only one proxy node could do data consistency check in proxy cluster
6. Scaling: Replace scaling input and output config fields type from int to Integer
7. Scaling: Update MySQL checksum SQL
8. Scaling: Improve scaling job progress deletion in reset and progress check before starting job
9. Scaling: Improve FinishCheckJob data consistency check when target tables already have the same data as source tables
10. Scaling: Break scaling job ASAP when there is unsupported table since primary key
11. Scaling: Reuse ClusterPersistRepository of proxy in PipelineAPIFactory
12. Scaling: Update jobId generation algorithm, and make it support idempotency
13. DistSQL: Support configuration data type and length when CREATE/ALTER ENCRYPT RULE
14. DistSQL: Unify the display results of SHOW ALL VARIABLES and SHOW VARIABLE
15. DistSQL: Remove the effect of binding order when DROP BINDING TABLE RULES
16. DistSQL: Add column mode_type in the result of SHOW INSTANCE LIST
17. DistSQL: Add validation to the mode when ENABLE/DISABLE INSTANCE
18. DistSQL: Check if the rule is in used when DROP READWRITE_SPLITTING RULE
19. DistSQL: Check duplicate resource names when CREATE READWRITE_SPLITTING RULE
20. DistSQL: Add column delay_time to the result of SHOW READWRITE_SPLITTING READ RESOURCES
21. DistSQL: Support IF EXISTS when DROP RULE
22. DistSQL: Optimize the prompt information of connection failure when ADD/ALTER RESOURCE
23. Mode: Add schema-level global distributed locks
24. Mode: Add schema version number to support batch execution of DistSQL
25. Mode: Persistent metadata optimization in cluster mode
26. Mode: The database discovery add the schemaName identifier when create a JOB

Refactor

1. Kernel: Refactor test case for encrypt
2. Kernel: Refactor metadata to support PostgreSQL database and schema
3. Scaling: Remove HikariCP dependency in pipeline modules
4. Mode: Refactor governance center storage node structure
5. Mode: Refactor governance center meta data structure
6. Mode: Adjust the database discovery MGR module to MySQL module

Bug Fix

1. Kernel: Fix function with no parameter
2. Kernel: Fix InsertValueContext.getValue cast exception
3. Kernel: Fix aggregate distinct column error
4. Kernel: Fix NPE when rewrite parameter with schema
5. Kernel: Fix NPE caused by GeneratedKeysResultSet not return columnName in read-write splitting
6. Kernel: Fix show tables statement loses part of the single table
7. Kernel: Fix ModShardingAlgorithm wrong route result when exist same suffix table
8. Kernel: Fix sql parse error when contains key in assignment clause and optimize index parse
9. Kernel: Fix NumberFormatException when sharding algorithm config number props
10. Kernel: Fix wrong metadata when config single dataSource for read-write splitting
11. Kernel: Fix statement close exception when use BatchPreparedStatementExecutor
12. Kernel: Fix rewrite lowercase logic when sql contains shorthand projection
13. Kernel: Fix NullPointerException when start up proxy with memory mode
14. Proxy: Fix literals may be replaced by mistake in PostgreSQL/openGauss protocol
15. Proxy: Fix ShardingSphere-Proxy PostgreSQL with multi-schema cannot be connected by PostgreSQL JDBC Driver 42.3.x
16. Proxy: Fix timestamp nanos inaccurate in ShardingSphere-Proxy MySQL
17. Proxy: Complete ShardingSphere-Proxy PostgreSQL codec for numeric in binary format
18. Proxy: Potential performance issue and risk of OOM in ShardingSphere-JDBC
19. Proxy: Fix Operation not allowed after ResultSet closed occasionally happens in ShardingSphere-Proxy MySQL
20. Proxy: Fix NPE causes by ShardingSphere-JDBC executeBatch without addBatch
21. Scaling: Fix failed or stopped job could not be started by DistSQL except restarting proxy
22. DistSQL: Fix parsing exception for inline expression when CREATE SHARDING TABLE RULE
23. DistSQL: Fix parsing exception when password is keyword password in ADD RESOURCE statement
24. Mode: Fixed loss of compute nodes due to ZooKeeper session timeout
25. Mode: Fixed the case of the table name in the governance center
26. Mode: DistSQL enable disable instance refresh in-memory compute node status
27. Mode: Fixed database discovery unable to create Rule through DistSQL

Change Log

1. MILESTONE

5.1.0

New feature

1. Support SQL hint
2. New DistSQL syntax: SHOW AUTHORITY RULE
3. New DistSQL syntax: SHOW TRANSACTION RULE
4. New DistSQL syntax: ALTER TRANSACTION RULE
5. New DistSQL syntax: SHOW SQL_PARSER RULE
6. New DistSQL syntax: ALTER SQL_PARSER RULE
7. New DistSQL syntax: ALTER DEFAULT SHARDING STRATEGY
8. New DistSQL syntax: DROP DEFAULT SHARDING STRATEGY
9. New DistSQL syntax: CREATE DEFAULT SINGLE TABLE RULE
10. New DistSQL syntax: SHOW SINGLE TABLES
11. New DistSQL syntax: SHOW SINGLE TABLE RULES
12. New DistSQL syntax: SHOW SHARDING TABLE NODES
13. New DistSQL syntax: CREATE/ALTER/DROP SHARDING KEY GENERATOR
14. New DistSQL syntax: SHOW SHARDING KEY GENERATORS
15. New DistSQL syntax: REFRESH TABLE METADATA
16. New DistSQL syntax: PARSE SQL, Output the abstract syntax tree obtained by parsing SQL
17. New DistSQL syntax: SHOW UNUSED SHARDING ALGORITHMS
18. New DistSQL syntax: SHOW UNUSED SHARDING KEY GENERATORS
19. New DistSQL syntax: CREATE/DROP SHARDING SCALING RULE
20. New DistSQL syntax: ENABLE/DISABLE SHARDING SCALING RULE
21. New DistSQL syntax: SHOW SHARDING SCALING RULES
22. New DistSQL syntax: SHOW INSTANCE MODE
23. New DistSQL syntax: COUNT SCHEMA RULES
24. Scaling: Add rateLimiter configuration and QPS TPS implementation
25. Scaling: Add DATA_MATCH data consistency check
26. Scaling: Add batchSize configuration to avoid possible OOME
27. Scaling: Add streamChannel configuration and MEMORY implementation
28. Scaling: Support MySQL BINARY data type
29. Scaling: Support MySQL YEAR data type
30. Scaling: Support PostgreSQL BIT data type
31. Scaling: Support PostgreSQL MONEY data type
32. Database discovery adds support for JDBC Spring Boot
33. Database discovery adds support for JDBC Spring Namespace
34. Database discovery adds support for openGauss
35. Shadow DB adds support for logical data source transfer
36. Add data type validator for column matching shadow algorithm
37. Add support for xa start/end/prepare/commit/recover in encrypt case with only one data source

API Change

1. Redesign the database discovery related DistSQL syntax
2. In DistSQL, the keyword GENERATED_KEY is adjusted to KEY_GENERATE_STRATEGY
3. Native authority provider is marked as deprecated and will be removed in a future version
4. Scaling: Move scaling configuration from server.yaml to config-sharding.yaml
5. Scaling: Rename clusterAutoSwitchAlgorithm SPI to completionDetector and refactor method parameter
6. Scaling: Data consistency check API method rename and return type change
7. Database discovery module API refactoring
8. Readwrite-splitting supports static and dynamic configuration
9. Shadow DB remove the enable configuration
10. Shadow algorithm type modified

Enhancement

1. Improve load multi single table performance
2. Remove automatically added order by primary key clause
3. Optimize binding table route logic without sharding column in join condition
4. Support update sharding key when the sharding routing result keep the same
5. Optimize rewrite engine performance
6. Support select union/union all ... statements by federation engine
7. Support insert on duplicate key update sharding column when route context keep same
8. Use union all to merge sql route units for simple select to improve performance
9. Supports autocommit in ShardingSphere-Proxy
10. ShardingSphere openGauss Proxy supports sha256 authentication method
11. Remove property java.net.preferIPv4Stack=true from Proxy startup script
12. Remove the verification of null rules for JDBC
13. Optimize performance of executing openGauss batch bind
14. Disable Netty resource leak detector by default
15. Supports describe prepared statement in PostgreSQL / openGauss Proxy
16. Optimize performance of executing PostgreSQL batched inserts
17. Add instance_id to the result of SHOW INSTANCE LIST
18. Support to use instance_id to perform operations when enable/disable a proxy instance
19. Support auto creative algorithm when CREATE SHARDING TABLE RULE, reducing the steps of creating rule
20. Support specifying an existing KeyGenerator when CREATE SHARDING TABLE RULE
21. DROP DATABASE supports IF EXISTS option
22. DATANODES in SHARDING TABLE RULE supports enumerated inline expressions
23. CREATE/ALTER SHARDING TABLE RULE supports complex sharding algorithm
24. SHOW SHARDING TABLE NODES supports non-inline scenarios (range, time, etc.)
25. When there is only one read data source in the readwrite-splitting rule, it is not allowed to be disabled
26. Scaling: Add basic support of chunked streaming data consistency check
27. Shadow algorithm decision logic optimization to improve performance

Refactor

1. Refactor federation engine scan table logic
2. Avoid duplicated TCL SQL parsing when executing prepared statement in Proxy
3. Scaling: Add pipeline modules to redesign scaling
4. Scaling: Refactor several job configuration structure
5. Scaling: Precalculate tasks splitting and persist in job configuration
6. Scaling: Add basic support of pipeline-core code reuse for encryption job
7. Scaling: Add basic support of scaling job and encryption job combined running
8. Scaling: Add input and output configuration, including workerThread and rateLimiter
9. Scaling: Move blockQueueSize into streamChannel
10. Scaling: Change jobId type from integer to text
11. Optimize JDBC to load only the specified schema
12. Optimize meta data structure of the registry center
13. Rename Note shadow algorithm to HINT shadow algorithm

Bug Fix

1. Support parsing function
2. Fix alter table drop constrain
3. Fix optimize table route
4. Support Route resource group
5. Support parsing binlog
6. Support postgreSql/openGauss '&' and '|' operator
7. Support parsing openGauss insert on duplicate key
8. Support parse postgreSql/openGauss union
9. Support query which table has column contains keyword
10. Fix missing parameter in function
11. Fix sub query table with no alias
12. Fix utc timestamp function
13. Fix alter encrypt column
14. Support alter column with position encrypt column
15. Fix delete with schema for postgresql
16. Fix wrong route result caused by oracle parser ambiguity
17. Fix projection count error when use sharding and encrypt
18. Fix npe when using shadow and readwrite_splitting
19. Fix wrong metadata when actual table is case insensitive
20. Fix encrypt rewrite exception when execute multiple table join query
21. Fix parsing chinese
22. Fix encrypt exists sub query
23. Fix full route caused by the MySQL BINARY keyword in the sharding condition
24. Fix getResultSet method empty result exception when using JDBCMemoryQueryResult processing statement
25. Fix incorrect shard table validation logic when creating store function/procedure
26. Fix null charset exception occurs when connecting Proxy with some PostgreSQL client
27. Fix executing commit in prepared statement cause transaction status incorrect in MySQL Proxy
28. Fix client connected to Proxy may stuck if error occurred in PostgreSQL with non English locale
29. Fix file not found when path of configurations contains blank character
30. Fix transaction status may be incorrect cause by early flush
31. Fix the unsigned datatype problem when query with PrepareStatement
32. Fix protocol violation in implementations of prepared statement in MySQL Proxy
33. Fix caching too many connections in openGauss batch bind
34. Fix the problem of missing data in SHOW READWRITE_SPLITTING RULES when db-discovery and readwrite-splitting are used together
35. Fix the problem of missing data in SHOW READWRITE_SPLITTING READ RESOURCES when db-discovery and readwrite-splitting are used together
36. Fix the NPE when the CREATE SHARDING TABLE RULE statement does not specify the sub-database and sub-table strategy
37. Fix NPE when PREVIEW SQL by schema.table
38. Fix DISABLE statement could disable readwrite-splitting write data source in some cases
39. Fix DIABLE INSTANCE could disable the current instance in some cases
40. Fix the issue that user may query the unauthorized logic schema when the provider is SCHEMA_PRIVILEGES_PERMITTED
41. Fix NPE when authority provider is not configured
42. Scaling: Fix DB connection leak on XA initialization which triggered by data consistency check
43. Scaling: Fix PostgreSQL replication stream exception on multiple data sources
44. Scaling: Fix migrating updated record exception on PostgreSQL incremental phase
45. Scaling: Fix MySQL 5.5 check BINLOG_ROW_IMAGE option failure
46. Scaling: Fix PostgreSQL xml data type consistency check
47. Fix database discovery failed to modify cron configuration
48. Fix single read data source use weight loadbalance algorithm error
49. Fix create redundant data source without memory mode
50. Fix column value matching shadow algorithm data type conversion exception

Change Log

1. MILESTONE

5.0.0

New feature

1. Support parsing SQL comment
2. New DistSQL syntax: shadow rule management
3. New DistSQL syntax: scaling job management
4. New DistSQL syntax: disable proxy instance
5. New DistSQL syntax: disable readwrite-splitting read data source
6. New DistSQL syntax: DROP SHARDING ALGORITHM
7. New DistSQL syntax: ALTER RESOURCE
8. New DistSQL syntax: CREATE SHARDING ALGORITHM
9. New DistSQL syntax: CREATE DEFAULT SHARDING [TABLE | DATABASE] STRATEGY
10. New DistSQL syntax: SHOW ALL VARIABLE
11. New DistSQL syntax：SHOW VARIABLE variableName;
12. Support 3 modes, including Memory, Standalone and Cluster mode
13. Proxy supports for openGauss
14. Scaling: Add basic support for openGauss
15. Scaling: Add incremental task completion detect algorithm SPI interface
16. Scaling: Add data consistency check algorithm SPI interface
17. Scaling: Basic support of create table on target automatically for MySQL and openGauss
18. Scaling: Support auto switch cluster configuration when job finished
19. Scaling: Add more DistSQL support such as data consistency check, etc

API Change

1. Add schema name configuration for ShardingSphere-JDBC
2. Add default sharding column configuration
3. Change the default authority provider from NATIVE to ALL_PRIVILEGES_PERMITTED
4. SCTL syntax adjustment, merged with DistSQL RAL syntax
5. SHOW RESOURCES DistSQL is adjusted to SHOW SCHEMA RESOURCES
6. Remove shadow logical field, support shadow algorithm

Enhancement

1. Support parsing MySQL union/union all statement
2. Support PostgreSQL ABORT statement
3. Refactor and improve metadata loading process
4. Support PostgreSQL CREATE INDEX statement to generate index automatically when no index is specified
5. Support SQL statement execution with logical schema
6. Support binding tables configuration with different sharding columns
7. Optimize kernel performance
8. Proxy supports queries to part of information_schema tables to optimize client connection experience
9. DistSQL supports using quotation marks to use keywords as parameter names
10. The password in the ADD RESOURCE statement supports special characters
11. ADD RESOURCE supports custom JDBC parameters and connection pool properties
12. DROP RESOURCE supports optional parameter ignore single tables, used to ignore single table rule restrictions
13. Support the use of DistSQL to create sharding table rule based on readwrite-splitting rules
14. SHOW DATABASES statement supports like syntax
15. CREATE SHARDING TABLE RULE supports the use of inline expressions to specify resources
16. CREATE SHARDING TABLE RULE supports configuration using dataNodes
17. CREATE SHARDING TABLE RULE supports reuse of existing algorithms
18. SET VARIABLE, support to modify proxy configuration
19. PostgreSQL's protocol enhancements (Such as supports Portal, unspecified type)
20. Using Netty executor to optimize Proxy performance in specified scenarios
21. Make memory strictly fetch size configurable in Proxy
22. Scaling: Improve support for PostgreSQL
23. Scaling: Support concurrent data consistency check of source and target side

Refactor

1. Refactor the SingleTable feature to support Encrypt multiple data sources
2. Adjust the persistent data structure of the registry center state node
3. Remove the SQL rewrite of DML for Shadow
4. Support the SQL routing of DDL for Shadow
5. Scaling: Refactor default implementation of incremental task completion detect algorithm
6. Scaling: Refactor default implementation of data consistency check algorithm
7. Scaling: Remove HTTP API and binary distribution

Bug Fix

1. Fix sharding interval algorithm
2. Fix SHOW INDEX FROM TABLE FROM TABLE statement rewrite exception
3. Fix Encrypt multi tables join query rewrite exception
4. Fix subquery index out of range exception
5. Fix wrong result of Oracle paging query
6. Fix the rewrite exception when KeyGenerateStrategy is not configured in the Sharding feature
7. Fix federation executor engine exception caused by Oracle dialect case
8. Fix Sharding and Encrypt integration usage rewrite exception
9. Fix Oracle metadata loading exception
10. Fix the issue that SHOW RESOURCES statement cannot display custom attributes
11. Fix the issue that SQL execution exception is not thrown
12. Fix Etcd can not send node added event
13. Fix PostgreSQL rows contains null value may be missing in query result
14. Fix PostgreSQL metadata columns are out-of-order
15. Fix client character set may be incorrect in Proxy

Change Log

1. MILESTONE

5.0.0-beta

New feature

1. New DistSQL to load and show all the ShardingSphere configuration rules
2. Support join SQL from different database instances
3. Support multiple backend database types for new feature of database gateway
4. Support creating and updating the authentication online
5. Add new automated agent module

API Change

1. QueryReplica configuration item is replaced by read-write-splitting
2. Authentication configuration item of ShardingProxy is replaced by AUTHORITY
3. Optimize the datasource configuration for ShardingJDBC with dataSourceClassName
4. New API for automated sharding table configuration to provide standard and automated sharding tables
5. Remove configuration item acceptor-size from ShardingProxy
6. Create a built-in sharding algorithm SPI which allows users to directly configure the class name as in 4.x

Enhancement

1. Improve metadata loading process distinctly
2. Greatly enhance the SQL parsing for Oracle, SQLServer and PostgreSQL
3. Support loading privileges from MySQL/PostgreSQL/SQLServer/Oracle
4. Support DDL statement for encryption feature
5. Support rewrite owner table name of projection if using sharding and encrypt together
6. When using SELECT * from encrypt SQL, add quote char for rewritten column to avoid conflict with keyword
7. Support PostgreSQL JSON/JSONB/pattern matching operator parse
8. Support MySQL/PostgreSQL CREATE/ALTER/DROP TABLESPACE statement
9. Support PostgreSQL PREPARE, EXECUTE, DEALLOCATE statement
10. Support PostgreSQL EXPLAIN statement
11. Support PostgreSQL START/END TRANSACTION statement
12. Support PostgreSQL ALTER/DROP INDEX statement
13. Support CREATE TABLESPACE of PostgreSQL
14. Support MySQL CREATE LOADABLE FUNCTION statement
15. Support MySQL/PostgreSQL ALTER TABLE RENAME statement
16. Support PostgreSQL Close command

Refactor

1. New schema structure in registry center
2. Remove Nacos and Apollo config center support
3. ShardingScaling leverages elasticJob as its job distribution
4. Refactor the metadata content and its update process

Bug Fix

1. Bug fix for cannot use SELECT * wildcard when readwrite-splitting only
2. Fix instance error when the custom sharding algorithm does not match the configuration type.
3. Fix NoSuchTableException when execute drop table if exists statement
4. Fix wrong table rewrite in UPDATE ... SET ... statement
5. Fix wrong table rewrite in CREATE/ALTER TABLE statement foreign key reference table
6. Fix projection owner check exception when exist subquery temporary table
7. Fix Oracle/SQL92 SELECT ... WHERE ... LIKE statement class cast exception
8. Fix MySQL SELECT EXISTS ... FROM ... statement parse error
9. Fix wrong result of SHOW INDEX statement
10. Fix SELECT ... GROUP BY ... statement rewrite and merge result error
11. Fix CREATE TABLE statement rewrite error for encrypt
12. Fix exception occur in PostgreSQL Proxy when reading text format parameter values
13. Enhance the support of array object for PostgreSQL Proxy
14. Fix the bug of Datetype casting for ShardingProxy
15. PostgreSQL Proxy supports using numeric type
16. Fix PostgreSQL Proxy transaction command complete packet's tag incorrect
17. Fix PostgreSQL Proxy may return packet which is not expected by client

Change Log

1. MILESTONE

5.0.0-alpha

Build & Dependencies

1. Upgrade the minimum supported version of JDK to Java8
2. Update Zookeeper to version 3.6.x and curator to version 5.1.0
3. Update Google Guava to version 29.0-jre

New Feature

1. The pluggable architecture is available and support function extension through the SPI
2. Independent SQL parsing engine is available to get SQL parsed AST for the different database dialects
3. New RDL(Rule Definition Language) feature for ShardingSphere Proxy supports creating sharding rules and sharding tables
4. ShardingSphere-Scaling supports resuming data migration from break-point
5. ShardingSphere-Scaling supports using ShardingSphere JDBC to migrate data to new cluster
6. ShardingSphere shadow database is available

API Changes

1. New sharding/encryption/shadow/queryReplica API
2. New sharding algorithm and strategy API
3. New API for ShardingSphere Scaling to create task
4. Remove DefaultDataSourceName setting item
5. The separator of the parameter configuration item is changed from the dot ‘.’ to minus sign ‘-’
6. Change parameter allow.range.query.with.inline.sharding from global parameter to sharding algorithm parameter

Refactor

1. Refactor the architecture of SQL parsing module base on the database dialects
2. Use SPI mechanism to reconstruct online metadata change processing
3. Rename Orchestration module to Governance module
4. Rename MasterSlave module to QueryReplica module
5. Refactor the metadata structure in the governance registration center
6. Refactor GovernmentShardingSphereDataSource
7. ShardingSphere UI merges configuration center and registration center display layout

Enhancement

1. The enhancement for MySQL and PostgreSQL SQL syntax definition and parsing process
2. The enhancement for sub-queries in different database dialects
3. Support MySQL view operations for non-sharding tables
4. Support MySQL stored function and procedure operations for non-sharding tables
5. Support SQLServer Top syntax
6. Optimize the metadata loading to improve the startup speed
7. Optimize batch insert performance
8. Supports the use of Oracle RAC JDBC URL
9. XA transaction manager adds support for Oracle database
10. ShardingSphere Proxy supports the use of p6sy driver
11. Add help information to the ShardingSphere Proxy startup script

Bug Fixes

1. Fix alias rewriting error when processing OrderBy condition
2. Fix SQL rewriting error when MYSQL Insert statement contains expression
3. Fix parameter calculation error in update on duplicate SQL
4. Fix generatedKeys gets wrong when batch inserting
5. Fix the abnormal issue of multi-table verification in DML statement update operation
6. Fix the NPE problem caused by executing SQL when the table does not exist
7. Fix the exception when using the Show table command for an unconfigured table
8. Fix metadata loading error when Oracle database has multiple qualified users
9. Fix the issue that replica node cannot be enabled online
10. Fix the problem that ShardingSphere-JDBC does not support PostgreSQL array type
11. Fix the problem that ShardingSphere-Proxy does not respond when querying long blob data

Change Logs

1. MILESTONE

4.1.1

Enhancement

1. Add Sharding-Scaling & Sharding-UI dockerfile
2. update MySQL & PostgreSQL version for proxy

Bug Fixes

1. Fix parser syntax rule of SUBSTRING and CONVERT
2. Fix parser ColumnSegment ClassCastException
3. Fix TableMetaData load error when use Sharding-JDBC with oracle
4. Fix getSchema NPE when use Sharding-JDBC with oracle
5. Fix Sharding-JDBC parse sql NPE in PostgreSQL
6. Fix Sharding-Proxy receive error response for PostgreSQL JDBC client
7. Fix Sharding-Proxy response number of update is 0 for PostgreSQL JDBC client
8. Fix Sharding-Proxy receive null for PostgreSQL column meta data
9. Fix Sharding-Scaling NPE in MySQL incremental task

Change Logs

1. MILESTONE

4.1.0

New Features

1. Support scaling for ShardingSphere (alpha version)
2. Move etcd registry center from experiment repo to apache repo
3. Upgrade to Java 8

Enhancement

1. Optimization for Sharding Parser with ANTLR Visitor improving the parsing performance of long SQL by 100%~1000%
2. Use multiple threads to load metadata for different data sources
3. Support allow.range.query.with.inline.sharding option
4. The docker of ShardingSphere-Proxy supports loading external lib packages
5. Support integration with Spring using @ShardingSphereTransactionType
6. Enhance ShardingDataSource to compatible with Seata in micro-service distribution transaction

Refactor

1. Remove leaf key generator

Bug Fixes

1. Fix an exception caused by using a CHAR/VARCHAR type column as an order by item
2. Refine DataTypeName syntax rules of all database dialects
3. Fix an exception caused by executing BEGIN using prepared statement of MySQL C API
4. Fix the problem that ALTER TABLE fails to execute when the field type of the table contains Integer/Double/BigDecimal
5. Fix the problem of the stop index dislocation of segment with alias
6. Fix the problem that overwriting SQL SELECT * FROM tbl WHERE encrypt_col = ? AND (sharding_key = ? OR sharding_key = ?) when using sharding + encrypt would throw StringIndexOutOfBoundsException
7. Fix the problem of incorrect decoding after AES encoding when using ShardingSphere-Proxy in Spring Boot
8. Fix a long-time problem of adding schema dynamically in ShardingSphere-UI

Change Logs

1. MILESTONE

4.0.1

Bug Fixes

1. Using guava cache to fix parsing deadlock.
2. Oracle insert SQL could not work in encrypt mode.
3. Proxy for PostgreSQL decode parameters error in all types except String.
4. COM_STM_EXECUTE of proxy for MySQL could not support sysbench.
5. None sharding strategy could not config in spring-boot.
6. WasNull field was wrong in GroupByStreamMergeResult.
7. Metadata.getColumns could not work in JDBC.
8. IN operator contains space and \n \t \r could not supported by parser.

Enhancement

1. Optimize antlr performance using two-stage parsing strategy.
2. Add class filter constructor to restrict the illegal class from YAML.

Change Logs

1. MILESTONE

4.0.0

API Changes

1. Change package and maven groupId form io.shardingsphere to org.apache.shardingsphere.
2. Adjust ShardingSphere-JDBC configuration API.
3. Adjust persist structure for registry center.

New Features

1. SQL92 Syntax available.
2. ShardingSphere-Proxy for PostgreSQL protocol available.
3. SQL 100% compatible if route to single data node.
4. Less-than(<), greater-than(>) and Less-than-equal(<=), greater-than-equal(>=) for sharding key operator available.
5. DISTINCT SQL syntax available.
6. Broadcast table available.
7. LEAF key generator available.
8. XA Transaction available, Atomikos, Narayana and Bitronix integrated.
9. BASE Transaction available, Seata integrated.
10. Data encrypt available.
11. Skywalking plugin available.
12. ShardingSphere-UI available, an orchestration management platform.

Enhancement

1. MariaDB supported.
2. Improve the compatibility of SQL parsing.
3. SELECT FOR UPDATE route to primary data source only.
4. Hint in ShardingSphere-Proxy available.
5. Make configuration of orchestration consistent between ShardingSphere-JDBC and ShardingSphere-Proxy.
6. Renew modified data sources only, not renew all the data sources.
7. Vibrate configurable for Snowflake key generator.

Bug Fixes

1. Improve the compatibility of JDBC Driver URL.
2. Delete statement with alias available.
3. Check and disable updating sharding column.
4. Fix wrong type of TINYINT and SMALLINT as INTEGER.

Change Logs

1. MILESTONE #3
2. MILESTONE #4
3. MILESTONE #5
4. MILESTONE #6
5. MILESTONE #7
6. MILESTONE #8
7. MILESTONE #9

4.0.0.RC3

New Features

1. ShardingSphere-UI, an orchestration management platform for ShardingSphere comes online.
2. Not only SQLs from MySQL, PostgreSQL, SQLServer, Oracle, but any SQL92 Syntax can be parsed correctly and used in ShardingSphere.

Enhancement

1. Support using less-than character(<) and greater-than character(>) for sharding data.
2. When primary and replica dataSources exist, support executing SELECT FOR UPDATE on primary data source.
3. Support hint in ShardingSphere-Proxy.
4. Finish parsing DAL syntax for MySQL.
5. Make configuration of orchestration compatible between ShardingSphere-JDBC and ShardingSphere-Proxy.

Bug Fixes

1. Through Bug fix, the feature of encryption becomes much stable and applicable.
2. Support delete statement with alias.
3. Check and disable updating sharding column.
4. Fix wrong type of TINYINT and SMALLINT as INTEGER.

Refactor

1. Rename optimized module to preprocessor module.
2. Decouple rewrite core module and sharding/encrypt features.

Change Logs

1. MILESTONE

4.0.0.RC2

API Changes

1. Optimize and delete API and configuration item of sharding logic index.
2. Update the API of encryption to support the encrypted and plain data coexistence.

New Features

1. Integration of Seata for distributed transaction.
2. User can do data encryption by using ShardingProxy.
3. User can use Leaf-segment generator to get distributed ID.
4. Support Skywalking plugin for application performance management.

Enhancement

1. Renew modified dataSources, not all the datasources to improve performance for configuration orchestration.
2. Improve the compatibility of SQL parsing.

Refactor

1. Remove DatabaseType enum, use dynamic SPI to load data source type.
2. The parse engine upgrade from the 2nd generation to 3rd.
3. The Refactoring of SQL rewriting module.

Change Logs

1. MILESTONE

4.0.0.RC1

Merge all change logs of version 3.1.0.M1, 3.1.0, 3.1.0.1 and 4.0.0.M1. First apache release.

API Changes

1. Adjust persist structure for orchestration's registry center.
2. Adjust ShardingSphere-JDBC configuration API.
3. Change package and maven groupId form io.shardingsphere to org.apache.shardingsphere.
4. Adjust spring-boot-starter.

New Features

1. XA Transaction available.
2. Data encrypt available.
3. Use PostgreSQL protocol access ShardingSphere-Proxy available.
4. DISTINCT SQL syntax available.
5. Broadcast table.
6. All SQL 100% compatible if route to single data node (MySQL Only).

Change Logs

1. MILESTONE #3
2. MILESTONE #4
3. MILESTONE #5
4. MILESTONE #6

3.0.0

Milestones

1. ShardingSphere-Proxy launch. Support the use of ShardingSphere in the form of database to support for MySQL CLI and GUI client

New Features

Core

1. ISSUE #290 Support batch INSERT
2. ISSUE #501 Support OR
3. ISSUE #980 Support DCL
4. ISSUE #1111 Support MySQL DAL

ShardingSphere-Proxy

1. ISSUE #902 Support XA transaction
2. ISSUE #916 Support authorization
3. ISSUE #936 Support registry center
4. ISSUE #1046 Support multiple logic databases

Enhancements

Core

1. ISSUE #373 Support order by ?
2. ISSUE #610 Route unicast for DQL without table
3. ISSUE #701 Caching parsed results to improve performance
4. ISSUE #773 Support sharding and autoincrement key of INSERT without column names
5. ISSUE #935 Use YAML instead of JSON to store configurations in registry center
6. ISSUE #1004 Properties can configure for sharding and replica query independent
7. ISSUE #1205 Execute engine enhancement

ShardingSphere-JDBC

1. ISSUE #652 Support Spring Boot Starter 2.X
2. ISSUE #702 Support $->{..} for inline expression
3. ISSUE #719 Support inject key generator objects by spring namespace
4. ISSUE #720 Support inject sharding algorithm objects by spring namespace

Sharding-Opentracing

1. ISSUE #1172 Opentracing enhancement

API changes

1. ISSUE #1153 Adjust the maven artifactId for orchestration module
2. ISSUE #1203 Adjust Spring namespace xsd for sharding and replica query
3. ISSUE #1289 Adjust hint API
4. ISSUE #1302 Refine package structure
5. ISSUE #1305 Deprecated and remove sharding-jdbc-transaction-parent module
6. ISSUE #1382 Remove type configuration in orchestration module

Bug Fixes

Core

1. ISSUE #569 Failed to parse SQL for Oracle when ROWNUM is not at end
2. ISSUE #628 Support data type jsonb for PostgreSQL
3. ISSUE #646 When aliases in SELECT ITEMS correspond to the real column names of GROUP BY or ORDER BY, there is no need to generate derived columns
4. ISSUE #806 NOT IN parse exception
5. ISSUE #827 Endless loop for bad SQL like SELECT * FROM table WHERE id IN ()
6. ISSUE #919 Inline expression with groovy may cause memory leak
7. ISSUE #993 Fail to parsing PostgreSQL due to the quotation
8. ISSUE #1015 Support SQL like SELECT id, COUNT(*) FROM table GROUP BY 1,2
9. ISSUE #1120 Derived columns of GROUP BY / ORDER BY appear in query result
10. ISSUE #1186 Dead lock may occur on MEMORY_STRICTLY mode when get connection on concurrency environment
11. ISSUE #1265 RoundRobinReplicaLoadBalanceAlgorithm throw an ArrayIndexOutOfBoundsException when AtomicInteger overflow

ShardingSphere-JDBC

1. ISSUE #372 Reuse PreparedStatement cause cache of route result do not clean
2. ISSUE #629 Support transaction isolation on JDBC
3. ISSUE #735 Unexpected replica datasource routing result when using Round-robin load-balance algorithm in Mybatis
4. ISSUE #1011 Can't resolve placeholder in Spring Boot YAML configuration

2.0.3

New Features

Core

1. ISSUE #600 Support TCL

Bug Fixes

Core

1. ISSUE #540 Support SQL that alias is the keyword
2. ISSUE #577 Support new line for YAML configuration

ShardingSphere-JDBC

1. ISSUE #522 Replica database does not need to execute the DDL for replica query

2.0.2

Enhancements

Core

1. ISSUE #475 Support CREATE INDEX
2. ISSUE #525 Support DROP INDEX

Bug Fixes

Core

1. ISSUE #521 ShardingProperties is invalid in YAML configuration
2. ISSUE #529 Table name capitalization cannot be queried
3. ISSUE #541 IS NOT NULL parse error
4. ISSUE #557 When GROUP BY and ORDER BY aliases are inconsistent, stream merging should be used
5. ISSUE #559 Support parsing numbers beginning with minus and decimal (e.g. -.12)
6. ISSUE #567 Add escape char for derived columns or alias when using MySQL

ShardingSphere-JDBC

1. ISSUE #520 Exception is no longer DuplicateKeyException when the unique key conflict

2.0.1

Enhancements

Core

1. ISSUE #490 Using rownum greater than or equal to or less than or equal to the result of paging is incorrect in Oracle
2. ISSUE #496 Logical table names in sharding configuration can be case sensitive
3. ISSUE #497 Close registry center gracefully

Bug Fixes

ShardingSphere-JDBC

1. ISSUE #489 Uses RuntimeBeanReference to prevent the creation of InnerBean on spring namespace
2. ISSUE #491 Can't close connection by ResultSet.getStatement().getConnection().close()

2.0.0

Milestones

1. API adjust. Brand new groupId and artifactId for Maven, package name and spring namespace name. Simplify and enhance API configuration, inline expression fully configuration support
2. Support spring-boot-starter of ShardingSphere-JDBC
3. Dynamic configuration. ZooKeeper and etcd can be used as registry to dynamically modify data sources and sharding configurations
4. Database orchestration. Fusing database access procedures to access databases and disable access to replica databases
5. ConfigMap support. Predefined metadata can be obtained in the sharding and replica query strategy
6. Tracking system support. You can view the invocation chain of ShardingSphere-JDBC through sky-walking and other Opentracing based APM systems

Enhancements

Core

1. ISSUE #386 Support SQL that does not contain table names, such as SELECT 1

ShardingSphere-JDBC

1. ISSUE #407 Support Hyphen properties for sharding-jdbc-spring-boot-starter
2. ISSUE #424 Providing SQL overall execution events

Bug Fixes

Core

1. ISSUE #387 Prevent errors from keywords process when '`' exists in function + column name
2. ISSUE #419 When SQL is rewritten, it does not determine whether alias is a keyword without the escape character, which results in SQL exception
3. ISSUE #464 SQL if the varchar type is not closed due to the absence of matching single quotes, and the next varchar in SQL is the wrong SQL of Chinese characters, it will lead to higher use of CPU

ShardingSphere-JDBC

1. ISSUE #394 Can't only close statement
2. ISSUE #398 Use Hint routing to shield case sensitivity
3. ISSUE #404 Sharding-jdbc's spring-boot-starter does not support HikariDataSource
4. ISSUE #436 Replica query, when the RoundRobin algorithm is configured from the database and MyBatis is used, it can only be routed to the same replica database
5. ISSUE #452 Sharding of DDL statements to more than one table causes a connection leak
6. ISSUE #472 Before Connection executes createStatement, it calls getMetaData first and then setAutoCommit can not take effective connection to the database that was created later

1.5.4.1

Bug Fixes

1. ISSUE #382 The query cannot be completed without sharding rule

1.5.4

Bug Fixes

1. ISSUE #356 In the Where condition of SQL, the REGEXP operator is compatible with non sharding columns
2. ISSUE #362 Replica query using PreparedStatement does not invoke the setParameter method to cause errors
3. ISSUE #370 Error in calling getGeneratedKeys using native self increment primary key
4. ISSUE #375 Data can not be obtained after paging second pages route to a single node
5. ISSUE #379 When Mybatis is used to call Connection.getMetaData (), the connection is not close correct

1.5.3

Enhancements

1. ISSUE #98 Replica query load balancing strategy support configuration
2. ISSUE #196 Replica query and sharding configuration independence

Bug Fixes

1. ISSUE #349 Incorrect function of ResultSet.wasNull causes null numeric type in DB to zero
2. ISSUE #351 Tables that are included in the default data source but not in TableRule configuration are not properly executed
3. ISSUE #353 In the Where condition of SQL, it is compatible with non sharding columns !=, !> and !< operator
4. ISSUE #354 In the Where condition of SQL, NOT operators are compatible with non-sharding columns

1.5.2

Milestones

1. The test engine of quality assurance, each SQL can run 60 test cases of different dimensions

Enhancements

1. ISSUE #335 Support the GROUP BY + custom function SQL
2. ISSUE #341 Support ORDER BY xxx NULLS FIRST | LAST statement of Oracle

Bug Fixes

1. ISSUE #334 Parsing ORDER BY with functions will resolve the following ASC and DESC to the name attribute of OrderItem
2. ISSUE #335 JOIN parsing is incorrect using the full name association of the table
3. ISSUE #346 Parsing table name error of DDL statement DROP TABLE IF EXISTS USER

1.5.1

New Features

1. ISSUE #314 Support DDL type SQL

Changes

1. ISSUE #327 Close sql.show configuration by default

Bug Fixes

1. ISSUE #308 Invalid return of database native GeneratedKey
2. ISSUE #309 ORDER BY and GROUP BY in sub-queries are not included in the analytic context
3. ISSUE #313 Support <> operator
4. ISSUE #317 Parameter of LIMIT can not be type of Long
5. ISSUE #320 SQL rewriting error of GROUP BY + LIMIT
6. ISSUE #323 Parsing ORDER BY + Aggregation Expression error

1.5.0

Milestones

1. The new SQL parsing module removes the dependence on Druid. We only need to parse the sharding context, and adopt a "semi understanding" concept for SQL to further improve performance and compatibility, and reduce code complexity
2. The new SQL rewrite module adds an optimized rewrite module
3. The new SQL merge module is rebuilt into 3 merging engines: streaming, memory and decorator

New Features

1. Support Oracle, SQLServer and PostgreSQL
2. Non functional sub-query support

Enhancements

1. ISSUE #256 Configurable display sharding execute to SQL log
2. ISSUE #291 Processing SQL use stream mode that contains only GroupBy

Changes

1. Simplify the distributed self increasing sequence. Each table is supported by simplifying a multiple self increasing sequence to a single table supporting only a single distributed self increasing sequence, and no longer supporting workerID settings through environment variables
2. Remove support for OR

Bug Fixes

1. ISSUE #239 LIMIT routes to multiple query result sets. If there is only one result set that is not empty, the result of paging is incorrect
2. ISSUE #263 Sharding and logical table configuration can be case insensitive
3. ISSUE #292 When the memory mode handles GROUP BY statement, if there is paging information, it needs to be rewritten
4. ISSUE #295 LIMIT 0 does not filter the result set according to paging restrictions

1.4.2

Enhancements

1. ISSUE #219 Thread performance optimization
2. ISSUE #215 Aggregated result set of stream sort StreamingOrderByReducerResultSet performance optimization
3. ISSUE #161 When the result sets are merged, heap sort can be used to improve performance

Bug Fixes

1. ISSUE #212 More meaningful hints for missing data source rules
2. ISSUE #214 table_name.column_name in (?,?) in WHERE can't parser expression
3. ISSUE #180 Batch execution of Update return inaccuracy
4. ISSUE #225 The last character of automatic generation of Id is not 0

1.4.1

Enhancements

1. ISSUE #191 Generating KeyGenerator of workerId based on IP of host
2. ISSUE #192 Get workerId's KeyGenerator based on HOSTNAME's digital suffix
3. ISSUE #210 Routing to single database and single table to remove supplementary SQL statement fragments

Bug Fixes

1. ISSUE #194 Some component exceptions in the close method of Connection, Statement, ResultSet and other interfaces cause the close method of another component to be not invoked
2. ISSUE #199 Sharding and reuse PreparedStatement object cause route error
3. ISSUE #201 Event transmission missing before batch operation execution
4. ISSUE #203 Merge events sent by the batch operation
5. ISSUE #209 Parallel execution of multiple limit queries leads to IndexOutOfBoundsException

1.4.0

Enhancements

Automatic generation key implementation, including

1. ISSUE #162 Implementation of distributed primary key algorithm
2. ISSUE #163 Acquisition of a self increasing sequence of JDBC interfaces
3. ISSUE #171 Sharding-jdbc-core coordination automatic generation sequence transformation
4. ISSUE #172 The configuration of YAML and Spring supports the self increasing sequence

Bug Fixes

1. ISSUE #176 The wasNull flag of AbstractMemoryResultSet is not reset in time

1.3.3

Enhancements

1. ISSUE #59 PreparedStatement can call the correct underlying set method according to the parameter type when setting parameters

Bug Fixes

1. ISSUE #149 When INSERT IGNORE INTO, if the data is duplicated, the value returned to -1 when ignored, and it should be returned to 0
2. ISSUE #118 In the same thread, DQL is executed first, then DML is executed, and DML operation is executed from the replica database
3. ISSUE #122 In cases where connections are not available (such as network interruption), transactions should be interrupted rather than retry
4. ISSUE #152 PreparedStatement's cache causes an array out of bound
5. ISSUE #150 With the latest SQLServer jdbc driver compatibility problem, Product Name should be changed from SQLServer to Microsoft SQL Server
6. ISSUE #166 Druid data source stat filter multi-thread error reporting should be added to database connection level synchronization

1.3.2

Enhancements

1. ISSUE #79 Optimizes limit for only one target table, does not modify limit offset

Bug Fixes

1. ISSUE #36 ShardingPreparedStatement cannot set parameters repeatedly
2. ISSUE #114 When ShardingPreparedStatement performs batch tasks, it repeatedly analyzes SQL and leads to OOM
3. ISSUE #33 According to the MySQL document, queries similar to limit 100 and -1 format are not supported
4. ISSUE #124 The return value of com.dangdang.ddframe.rdb.sharding.jdbc.adapter.AbstractStatementAdapter.getUpdateCount does not conform to the JDBC specification
5. ISSUE #141 Multi-thread executor parameter setting failure

1.3.1

Enhancements

1. ISSUE #91 Open support for Statement.getGeneratedKeys can return the original database self increase primary key
2. ISSUE #92 Query DQL statement event sending

Bug Fixes

1. ISSUE #89 Use replica query with sharding hint leads to conflict
2. ISSUE #95 Write operations in the same thread read from the primary database changed to the same thread and within the same connection

1.3.0

New Features

1. ISSUE #85 New feature for replica query

Enhancements

1. ISSUE #82 TableRule can import the dataSourceName attribute to specify the data source corresponding to the TableRule
2. ISSUE #88 Release restrictions on other databases, support standard SQL, do not support personalized paging statements

Bug Fixes

1. ISSUE #81 Associative table query uses OR query condition to resolve the result exceptions

1.2.1

Refactor

1. ISSUE #60 API adjust, remove ShardingDataSource, use factory instead
2. ISSUE #76 ShardingRule and TableRule change to Builder pattern
3. ISSUE #77 ShardingRule and TableRule change to Builder pattern

Enhancements

1. ISSUE #61 Add the logical table name to the ShardingValue class
2. ISSUE #66 Statement on the JDBC tier supports get/set MaxFieldSize, MaxRows and QueryTimeout
3. ISSUE #72 Batch inserts supporting select union all
4. ISSUE #78 Simplifying sharding only configuration, without configuring logical table and real table correspondence
5. ISSUE #80 Simplifying the configuration that does not sharding, specifying the default data source, do not need configure TableRule

Bug Fixes

1. ISSUE #63 No table name or table alias is added to the ORDER BY and GROUP BY derivation columns
2. ISSUE #65 Performance enhancement for parsing condition context
3. ISSUE #67 The soft transaction log cannot be deleted when routed to multiple tables
4. ISSUE #71 Routing single sharding key by OFFSET of LIMIT error
5. ISSUE #75 MemoryTransactionLogStorage retry times update concurrency problem

1.2.0

New Features

1. ISSUE #53 The relationship between the real table and the logical table is not configured, and the real table is dynamically calculated by the sharding algorithm
2. ISSUE #58 Soft transaction: the initial version of the best effort type

Refactor

1. ISSUE #49 Adjust the property configuration
2. ISSUE #51 Refactor of Hint interface

Bug Fixes

1. ISSUE #43 The yaml file contains Chinese, and the operating system mode is not UTF-8 encoding, resulting in yaml can not be parsed
2. ISSUE #48 Yaml file is not closed after reading
3. ISSUE #57 At the analytic level, we can identify subqueries to ensure that the behavior of supplementary columns can be accurately positioned

1.1.0

New Features

1. ISSUE #40 Support YAML configuration
2. ISSUE #41 Support Spring namespace configuration
3. ISSUE #42 Support inline expression configuration

Bug Fixes

1. ISSUE #25 The problem of repeated results will appear under the OR expression

1.0.1

Enhancements

1. ISSUE #39 Support the use of Hint method to register the key value to SQL routing

Bug Fixes

1. ISSUE #11 The count function returns incorrectly without aliases
2. ISSUE #13 The Insert statement does not write column names or write column names but column names do not contain sharding fields, occur broadcast route
3. ISSUE #16 For now a new connection pool is executed every time SQL is executed. Instead, each ShardingDataSource object should be changed to share a pool
4. ISSUE #18 When query Count, getObject() throws Exception: Unsupported data type: Object
5. ISSUE #19 In SUM and AVG functions, merger is not executed if aliases are not added, and null pointer exception fired if aliases are added
6. ISSUE #38 The compatibility between JPA and ShardingSphere-JDBC. JPA automatically add the column aliases of SELECT, resulting in ORDER BY obtaining ResultSet data only by aliases rather than column names

1.0.0

1. Initial version