TiSpark 3.1.2

New Features

• Support write into partition table with new collations #2570
• Support read TiFlash load balancing with Round-Robin strategy #2576
• Support partition pruning with to_days function when read from TiKV/TiFlash #2594 #2600

Fixes

• Fix CBO to let TiSpark choose the min cost between TiKV table scan, TiKV index scan and TiFlash scan correctly #2568
• Fix the region not find error when reading from TiKV #2575
• Fix the bug that statistics is not collected which may affect the choice of plans #2589
• Compatible with TiDB v6.5.0 #2602

TiSpark 2.4.4

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460.

Fixes

• Fix the bug that single column condition is in the incorrect if branch #2393.
• Fix region may be missed with too many tables #2442.
• Fix the bug that count can not be pushed down #2468 #2483

TiSpark 3.1.1

Fixes

• Fix fail to read from some partition table #2553
• Bump spark version to avoid vulnerabilities #2555
• Fix error will be thrown when date col is not the first col in hash partition table #2556

TiSpark 3.1.0

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2457

New Features

• Add authorization check for datasource api #2366
• Make TiSpark's Explain clearer and easier to read #2439
• Support host mapping in TiSpark #2436
• Support bypass-TiDB write into partition table #2451
• Support insert sql #2471
• Support Spark 3.3 #2492
• Only do auth check for tables in TiDB #2489
• Support new Collation #2524

Fixes

• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw table not found exceptions #2433
• Fix the bug that count/avg can not push down #2445
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict. And the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2455
• Fix the bug that exception would through when the size of pdAddresse is > 1 #2473
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2476
• Upgraded Spark 3.1 support version from 3.0.2 to 3.0.3, upgraded Spark 3.1 support version from 3.1.1 to 3.1.3, upgraded Spark 3.2 support version from 3.2.1 to 3.2.2 #2486
• Fix the bug that exception will be throw when date col is not the first col ref #2538

TiSpark 3.0.2

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460

Fixes

• Fix the bug that single column condition is in the incorrect if branch #2395
• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2440
• Fix the bug that count/avg can not push down #2470
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2515
• Fix exception would through when the size of pdAddresse is > 1 #2478
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2485
• Upgraded Spark3.1 support version from 3.0.2 to 3.0.3, Upgraded Spark3.1 support version from 3.1.1 to 3.1.3, Upgraded Spark3.2 support version from 3.2.1 to 3.2.2 #2488
• Only do auth check for tables in TiDB #2500
• Changed profile #2518

TiSpark 2.5.2

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2461

Fixes

• Fix the bug that single column condition is incorrect if branch #2394
• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2441
• Fix the bug that count/avg can not push down #2469
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2516
• Fix exception would through when the size of pdAddresse is > 1 #2477
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2484
• Upgraded Spark3.1 support version from 3.0.2 to 3.0.3，Upgraded Spark3.1 support version from 3.1.1 to 3.1.3 #2487
• Only do auth check for tables in TiDB #2502
• Change spark profile #2517

TiSpark 3.0.1

Fixes

• Fix the bug that the single column condition is in the incorrect if branch #2395
• Fix the bug that the spark-shell is stuck when exiting #2402
• Fix the bug that the maven dependency of TiSpark 3.0.0 can not be imported #2401
• Shutdown recycler when closing ChannelFactory to avoid resource leak #2405

TiSpark 3.0.0

Compatibility Changes

• TiSpark without catalog plugin is no more supported. You must configure catalog configs and use tidb_catalog now #2252
• TiSpark's jar has a new naming rule like tispark-assembly-{$spark_version}_{$scala_version}-{$tispark_verison} #2370

New Feature

• Support DELETE statement #2276
• Support Spark 3.2 #2287
• Support telemetry to collect information #2316
• Support stale read to read historical versions of data #2322
• Support TLS with reload capability #2306 #2349 #2365 #2377

Fixes

• Fix the wrong result of _tidb_rowid when set spark.tispark.show_rowid=true #2270
• Fix sum not push down bug #2314
• Fix limit not push down bug #2329
• Avoid NoSuchElementException when setting catalog #2220
• Avoid ClassCastException when cluster index with type Timestamp and Date #2319
• Improve retry logic in write so that it will not retry in some scenarios which needn't retry #2279
• Delete unused configuration spark.tispark.statistics.auto_load #2300
• Upgrade jackson-databind from 2.9.10.8 to 2.12.6.1 #2285
• Upgrade guava from 26.0-android to 29.0-android #2340
• Upgrade mysql-connector-java from 5.1.44 to 5.1.49 #2367

Documents

• Update communication channels #2228
• Add limitation: new collations are not supported #2238

TiSpark 2.5.1

Fixes

• Fix limit not push down bug #2335
• Fix ClassCastException when cluster index with type Timestamp and Date #2323
• Upgrade jackson-databind from 2.9.10.8 to 2.12.6.1 #2288
• Fix the wrong result of _tidb_rowid #2278
• Fix set catalog throw NoSuchElementException #2254

Documents

• Add limitation: TLS is not supported #2281
• Add limitation: new collations are not supported #2251
• Update communication channels #2244

TiSpark 2.5.0

New feature

• Support Spark 3.1.x and 3.0.x version.
• Support Spark SQL's authentication and authorization through TiDB #2185.

Fixes

• Fix duplicate range in RowIDAllocator #2156.
• upgrade log4j to 2.17.1 #2197.
• upgrade jackson-databind to 2.9.10.8 #2161.

Documents

• Fix in user docs forget to use options while using BatchWrite #2169

Known issue

• Spark SQL's authentication and authorization through TiDB will affect other datasource such as hive. See details in this issue #2224.
• Batch write needs more tests and may be unstable. See details in this issue #2222.