[Spark] Implement Purge table for SparkCatalog#3015

Open

zhu3pang wants to merge 82 commits intoapache:masterfrom zhu3pang:PAIMON-1401

+122-12

Commits on Mar 15, 2024

[PAIMON-1401] [Feature] Implement Purge table for SparkCatalog
zhu3pang
committed

Commits on Mar 29, 2024

[hotfix] Fix unstable testRandomCdcEventsUnawareBucket (apache#3014 )

yuzelin
authored and
zhu3pang
committed
[flink] Refactor: flink tests use common CatalogITCaseBase (apache#3016 )

yuzelin
authored and
zhu3pang
committed
[core] Support compiled class cache in CodeGenUtils

xiangyuf
authored and
zhu3pang
committed
[core] Optimize compiled class cache in CodeGenUtils

JingsongLi
authored and
zhu3pang
committed
[core] AvroBulkFormat should not get file-size by file-io repeatly (apache#3019 )

leaves12138
authored and
zhu3pang
committed
[hive] Cache TableSchema into Configuration to avoid loading read scheme file in PaimonSerDe (apache#2946 )

wg1026688210
authored and
zhu3pang
committed
[core] Support configuring lock in paimon catalog (apache#2933 )

FangYongs
authored and
zhu3pang
committed
[codegen] EqualiserCodeGenerator supports ARRAY<ROW> (apache#3023 )

yuzelin
authored and
zhu3pang
committed
[core] Unified Naming of JdbcCatalog Fields (apache#3025 )

sunxiaojian
authored and
zhu3pang
committed
[hotfix] upgrade json-path and avro version (apache#2987 )

zhuangchong
authored and
zhu3pang
committed
[core] Dv table supports value filter pushdown (apache#3024 )

Zouxxyy
authored and
zhu3pang
committed
[hive] Fix hive writer in tez-mr (apache#2954 )

Alibaba-HZY
authored and
zhu3pang
committed
[hive] Add license for TezUtil

JingsongLi
authored and
zhu3pang
committed
[core] Reduce schema read for readers (apache#3021 )

JingsongLi
authored and
zhu3pang
committed
[core] Add more validation for sequence field during create table. (apache#3022 )

hzjhjjyy
authored and
zhu3pang
committed
[doc] Updated Spark quickstart documentation for using --packages (apache#2897 )

cxzl25
authored and
zhu3pang
committed
[core] Use var length encoding for row position (apache#3031 )

JingsongLi
authored and
zhu3pang
committed
[oss] use multi-release mechanism to ship jaxb (apache#2658 )

zhoulii
authored and
zhu3pang
committed
[core] Data files with delete records should not be upgraded directly to max level (apache#2962 )

tsreaper
authored and
zhu3pang
committed
[Flink] Introduce range strategy for sort compaction. (apache#2749 )

wg1026688210
authored and
zhu3pang
committed
[spark] spark sql support get/list function (apache#2880 )

waywtdcc
authored and
zhu3pang
committed
[docs] Trino docs update for sharing file system. (apache#3035 )

leaves12138
authored and
zhu3pang
committed
[core] Fix dv table with partial-update and aggregate (apache#3036 )

Zouxxyy
authored and
zhu3pang
committed
[core] Introduce BatchTableCommit.truncateTable (apache#3037 )

JingsongLi
authored and
zhu3pang
committed
[core] orc/parquet reader obtain the fileSize from metadata (apache#2918 )

wg1026688210
authored and
zhu3pang
committed
[core] Optimize write performance (apache#3039 )

JingsongLi
authored and
zhu3pang
committed
[core] Reduce getFileSize for avro reader (apache#3040 )

JingsongLi
authored and
zhu3pang
committed
[core] support create tag based on tag (apache#3044 )

Aitozi
authored and
zhu3pang
committed
[flink] Remove compatibility utils that are used for flink-1.14 (apache#3054 )

yuzelin
authored and
zhu3pang
committed
[core] Add validation for fields prefix (apache#3052 )

Zouxxyy
authored and
zhu3pang
committed
[core] add initial map size to solve load hash index slowly (apache#3051 )

Stephen0421
authored and
zhu3pang
committed
[core] Introduce Int2ShortHashMap.Builder to to accelerate init

JingsongLi
authored and
zhu3pang
committed
[core] support to drop partition when delete tag (apache#3042 )

Aitozi
authored and
zhu3pang
committed
[core] Optimize first_row batch read (apache#3055 )

JingsongLi
authored and
zhu3pang
committed
[flink] Bump flink version to 1.19 (apache#3049 )

yuzelin
authored and
zhu3pang
committed
[orc] ORC Support ZStandard compression using zstd-jni (apache#3056 )

zyl891229
authored and
zhu3pang
committed
[license] Add license to orc copied files

JingsongLi
authored and
zhu3pang
committed
[doc] Introduce orc options documentation page (apache#3061 )

JingsongLi
authored and
zhu3pang
committed
[doc] Add flink 1.19 to the documentation. (apache#3064 )

zhuangchong
authored and
zhu3pang
committed
[hotfix] Remove redundant and unused test code (apache#3066 )

yuzelin
authored and
zhu3pang
committed
[doc] Fix version in trino documentation (apache#3072 )

Zouxxyy
authored and
zhu3pang
committed
[core] Skip rewrite when delete row count is 0 with dv (apache#3070 )

Zouxxyy
authored and
zhu3pang
committed
[hotfix] Add project root path to files to facilitate compiling modules in child dirs (apache#3063 )

Zouxxyy
authored and
zhu3pang
committed
[flink] Compact procedure does not process 'ALL' word (apache#3065 )

leaves12138
authored and
zhu3pang
committed
[fix] Fix unstable test TagAutoCreationTest#testModifyTagPeriod (apache#3073 )

yuzelin
authored and
zhu3pang
committed
[spark] Minor refactor SparkWriter to separate commit (apache#3067 )

YannByron
authored and
zhu3pang
committed
[flink] Fix that compact actions haven't handled scan parallelism (apache#3077 )

yuzelin
authored and
zhu3pang
committed
[core] JdbcCatalog supports custom configuration for the length of lock table (apache#3069 )

sunxiaojian
authored and
zhu3pang
committed
[doc] Introduce the minimum version of Maven. (apache#3082 )

wg1026688210
authored and
zhu3pang
committed
[doc] Apache Paimon became top level project. (apache#3085 )

leaves12138
authored and
zhu3pang
committed
[incubating] Remove incubator disclaimer

JingsongLi
authored and
zhu3pang
committed
[flink] Compact procedure supports named arguments (apache#3078 )

yuzelin
authored and
zhu3pang
committed
[cdc] Update the latest dynamic options to the table schema file when synchronizing an existing paimon table using the cdc action (apache#3062 )

zhuangchong
authored and
zhu3pang
committed
[flink] Fix performance issue in ContinuousFileSplitEnumerator (apache#3071 )

yuzelin
authored and
zhu3pang
committed
[Refactor] Minor refactor update the latest dynamic options using the cdc action. (apache#3090 )

yuzelin
authored and
zhu3pang
committed
[cdc] Extract common code (apache#3050 )

zhangjun0x01
authored and
zhu3pang
committed
[cdc] Fix Mysql sync_table unable to synchronize newly added tables with regular expression (apache#2851 )

huyuanfeng2018
authored and
zhu3pang
committed
[hotfix] Apply spotless to fix checkstyle violation in flink-cdc module (apache#3092 )

yuzelin
authored and
zhu3pang
committed
[test] Append only table can't test failing IO re-insert yet (apache#3089 )

leaves12138
authored and
zhu3pang
committed
[core] Introduce SplitGroup for SplitGenerator to optimize more rawFiles (apache#3059 )

Zouxxyy
authored and
zhu3pang
committed
[hotfix] Fix unstable test MySqlSyncTableActionITCase#testSyncShards (apache#3096 )

yuzelin
authored and
zhu3pang
committed
[core] Refactor FileStorePathFactory to clean test methods (apache#3093 )

JingsongLi
authored and
zhu3pang
committed
[core] Add TableFormatBenchmark to test format only

JingsongLi
authored and
zhu3pang
committed
[spark][core] Support input_file_name UDF (apache#3094 )

YannByron
authored and
zhu3pang
committed
[core] Unify FileRecordReader and reduce file access (apache#3098 )

JingsongLi
authored and
zhu3pang
committed
[core] Support common jdbc catalog lock for filesystem catalog. (apache#3076 )

sunxiaojian
authored and
zhu3pang
committed
[core] Fix OrphanFilesClean has a chance to delete normal file (apache#3075 )

wxplovecc
authored and
zhu3pang
committed
[catalog] Refactor Catalog Factory and revert jdbc lock in FileSystemCatalog (apache#3099 )

JingsongLi
authored and
zhu3pang
committed
[core] Shade roaringbitmap dependency into paimon-common (apache#3100 )

Zouxxyy
authored and
zhu3pang
committed
[cdc] Fix cdc job mistakenly changes immutable options of existing table (apache#3095 )

zhuangchong
authored and
zhu3pang
committed
[core] Support dv with avro format (apache#3105 )

Zouxxyy
authored and
zhu3pang
committed
[core] Introduce File index format for data skipping (apache#3068 )

leaves12138
authored and
zhu3pang
committed
[test] Fix flaky test in KeyValueFileStoreScanTest.testWithValueFilter (apache#3109 )

Zouxxyy
authored and
zhu3pang
committed
[doc] Introduce deletion vector documentation page (apache#3107 )

Zouxxyy
authored and
zhu3pang
committed
[core] Rename to compactDvIndexFiles in FileStoreCommitImpl

JingsongLi
authored and
zhu3pang
committed
[test] Fix test in KeyValueFileStoreScanTest, make it more accurate (apache#3111 )

leaves12138
authored and
zhu3pang
committed
[format] Set orc.compression.zstd.level to 3 by default (apache#3114 )

zyl891229
authored and
zhu3pang
committed
[core] add deletedFiles in NewFilesIncrement (apache#3117 )

YannByron
authored and
zhu3pang
committed
[cdc] Fix that kafka message value might be null which causes NPE (apache#3113 )

yuzelin
authored and
zhu3pang
committed
[PAIMON-1401] [Feature] Implement Purge table for SparkCatalog, add some UTs
zhu3pang
committed
Merge remote-tracking branch 'origin/master' into PAIMON-1401
zhu3pang
committed