Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge branch-24.10 into main [skip ci] #11578

Merged
merged 108 commits into from
Oct 14, 2024
Merged
Show file tree
Hide file tree
Changes from 103 commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
1b064c6
Init version 24.10.0-SNAPSHOT
NvTimLiu Jul 24, 2024
eadc6db
Merge pull request #11252 from NVIDIA/branch-24.08
nvauto Jul 25, 2024
bce9d51
Merge pull request #11253 from NVIDIA/branch-24.08
nvauto Jul 25, 2024
5d1b82b
Merge pull request #11257 from NVIDIA/branch-24.08
nvauto Jul 25, 2024
469bab4
Merge pull request #11261 from NVIDIA/branch-24.08
nvauto Jul 26, 2024
c5059a7
Merge pull request #11262 from NVIDIA/branch-24.08
nvauto Jul 26, 2024
09b2dd5
Merge pull request #11271 from NVIDIA/branch-24.08
nvauto Jul 30, 2024
97f6d4d
Merge pull request #11272 from NVIDIA/branch-24.08
nvauto Jul 30, 2024
2e0e22b
Update the rapids JNI and private dependency version to 24.10.0-SNAPS…
NvTimLiu Jul 30, 2024
99c2d12
Merge pull request #11274 from NVIDIA/branch-24.08
nvauto Jul 30, 2024
079cf38
Merge pull request #11275 from NVIDIA/branch-24.08
nvauto Jul 30, 2024
6779154
Explicitly disable ANSI mode for ast_test.py [databricks] (#11258)
razajafri Jul 30, 2024
e53f234
Merge pull request #11279 from NVIDIA/branch-24.08
nvauto Jul 31, 2024
dfcff71
Asynchronously copy table data to the host during shuffle (#11280)
jlowe Jul 31, 2024
6c93a1b
remove spark31x tools supported files (#11285)
cindyyuanjiang Aug 2, 2024
cfac0b9
Move easy unshimmed classes to sql-plugin-api (#11288)
gerashegalov Aug 2, 2024
2af9c0e
Use distinct count to estimate join magnification factor (#11284)
jlowe Aug 2, 2024
96d32f0
Merge pull request #11298 from NVIDIA/branch-24.08
nvauto Aug 5, 2024
e7f218a
Remove redundant classes from the dist jar and unshimmed list (#11295)
gerashegalov Aug 5, 2024
93cdae1
Use the new chunked API fro multi-get_json_object (#11289)
revans2 Aug 5, 2024
a0cbd1d
Enable get_json_object by default and remove legacy version (#11299)
revans2 Aug 6, 2024
18980f7
Skip deploying non-critical intermediate artifacts (#11301)
yinqingh Aug 7, 2024
8ef2e41
Fix display issue of lore.md (#11302)
liurenjie1024 Aug 7, 2024
a54c9b3
Add jihoonson as an authorized user for blossom-ci (#11312)
jihoonson Aug 8, 2024
8827bd8
Merge branch-24.08 into branch-24.10
NvTimLiu Aug 9, 2024
a4bafa7
To skip this '*'
NvTimLiu Aug 9, 2024
de39a94
Merge pull request #11313 from NvTimLiu/fix-auto-merge-conflict-11310
nvauto Aug 9, 2024
05152f7
Safely close multiple resources in RapidsBufferCatalog (#11307)
jihoonson Aug 9, 2024
1101427
Merge pull request #11315 from NVIDIA/branch-24.08
nvauto Aug 10, 2024
36b9e2c
Update passing JSON tests after list support added in CUDF (#11319)
revans2 Aug 13, 2024
48d8b3d
Merge branch-24.08 into branch-24.10
NvTimLiu Aug 14, 2024
a85d101
Merge pull request #11325 from NvTimLiu/fix-auto-merge-conflict-11317
jlowe Aug 14, 2024
a876df0
Append ustcfy to blossom-ci whitelist [skip ci] (#11324)
ustcfy Aug 15, 2024
25be396
Make hive column matches not case-sensitive (#11327)
revans2 Aug 15, 2024
bc8c577
Fix the mismatching default configs in integration tests (#11283)
jihoonson Aug 15, 2024
6283029
Merge pull request #11336 from NVIDIA/branch-24.08
nvauto Aug 16, 2024
2d90ec8
Merge pull request #11338 from NVIDIA/branch-24.08
nvauto Aug 16, 2024
c0c93b3
Audit script - Check commits from sql-hive directory [skip ci] (#11340)
nartal1 Aug 16, 2024
5302651
replace inputFiles with location.rootPaths.toString (#11323)
zpuller Aug 16, 2024
a61b834
Revert "Fix the mismatching default configs in integration tests (#1…
jihoonson Aug 17, 2024
c412433
Merge remote-tracking branch 'upstream/branch-24.08' into fix-auto-me…
pxLi Aug 18, 2024
2133b7d
Merge pull request #11357 from pxLi/fix-auto-merge-conflict-11354
pxLi Aug 18, 2024
6fb3205
Fix 24.08 download doc (#11358)
pxLi Aug 18, 2024
15aa45c
Swap build side for outer joins when natural build side is explosive …
jlowe Aug 19, 2024
92d9bbb
Add string escaping JSON tests to the test_json_matrix (#10604)
revans2 Aug 19, 2024
b23e3fa
conform dependency list in 341db to previous versions style [databric…
zpuller Aug 19, 2024
2495b72
Add tests for repeated columns (#11362)
revans2 Aug 19, 2024
6924404
Fix failing test compile for Spark 4.0.0 (#11363)
revans2 Aug 20, 2024
9fcd8c7
Set numRows for the ColumnBatch created in GpuBringBackToHost (#11365)
jihoonson Aug 20, 2024
3023625
Change reference to `MapUtils` into `JSONUtils` (#11329)
ttnghia Aug 21, 2024
eb7b927
Move SparkRapidsBuildInfoEvent to its own file (#11368)
tgravescs Aug 22, 2024
0f5254b
Fix nightly snapshots being downloaded in premerge build (#11383)
jlowe Aug 22, 2024
35d2163
Create a PrioritySemaphore to back the GpuSemaphore (#11376)
zpuller Aug 22, 2024
d53de06
Fix spark400 build in datagen and tests (#11375)
jlowe Aug 22, 2024
6148257
JSON tests for corrected date, timestamp, and mixed types (#11388)
revans2 Aug 23, 2024
ac49393
Add support for Spark 3.5.2 [databricks] (#11334)
razajafri Aug 23, 2024
35c593c
Revert work-around for empty split-string (#11393)
revans2 Aug 26, 2024
eb5735c
Drop cudf-py python 3.9 support (#11396)
NvTimLiu Aug 27, 2024
3e04c1f
Add distinct join support for right outer joins (#11291)
jlowe Aug 27, 2024
549daf6
prevent duplicate queueing in the prio semaphore (#11389)
zpuller Aug 27, 2024
46057a4
stop using copyWithBooleanColumnAsValidity (#11399)
revans2 Aug 27, 2024
61f0cca
Merge pull request #11406 from NVIDIA/branch-24.08
nvauto Aug 28, 2024
dbd92d2
Support MinBy and MaxBy for non-float ordering (#11371)
thirtiseven Aug 29, 2024
db1d580
Fix a Pandas UDF slowness issue (#11395)
firestarman Aug 30, 2024
ee2049a
Fix asymmetric join crash when stream side is empty (#11411)
jlowe Aug 30, 2024
6d4c4ef
Stop using `copyWithBooleanColumnAsValidity` [databricks] (#11418)
res-life Sep 3, 2024
d2abcd9
Add in array_join support (#11420)
revans2 Sep 4, 2024
53bf03a
Update signoff usage (#11425)
pxLi Sep 5, 2024
5d7dc39
Update CI scripts (#11427)
NvTimLiu Sep 5, 2024
4c2203e
Dynamic Shim Detection for `build` Process [databricks] (#11308)
razajafri Sep 6, 2024
77615e6
remove the redundant archive link (#11421)
nvliyuan Sep 6, 2024
7de3fc4
Add companion metrics for all nsTiming metrics without semaphore (#11…
binmahone Sep 6, 2024
9fc0667
xfail array and map cast to string tests (#11451)
jlowe Sep 9, 2024
a92bfbf
Log DBR BuildInfo [databricks] (#11455)
gerashegalov Sep 10, 2024
502f5a3
Fixed some of the failing parquet_tests [databricks] (#11429)
razajafri Sep 10, 2024
5beeba8
Skip test_hash_groupby_approx_percentile byte and double tests tempor…
abellina Sep 12, 2024
938d4df
Replace scala.util.Try with a try statement in the DBR buildinfo [dat…
gerashegalov Sep 14, 2024
65f0095
Revert "Skip test_hash_groupby_approx_percentile byte and double test…
abellina Sep 16, 2024
00cd422
Enable tests after string_split was fixed (#11474)
revans2 Sep 16, 2024
2589976
Use improved CUDF JSON validation (#11464)
revans2 Sep 17, 2024
f4119c1
Fix a json test for non utc time zone (#11482)
revans2 Sep 18, 2024
7c13383
Use reusable auto-merge workflow (#11483)
pxLi Sep 19, 2024
4b26190
Enable tests for all JSON white space normalization (#11456)
revans2 Sep 20, 2024
0f5d510
Support yyyyMMdd in GetTimestamp operator for LEGACY mode [databricks…
res-life Sep 23, 2024
ebcc146
Support non-UTC timezone for casting from date type to timestamp type…
res-life Sep 23, 2024
510ee83
Install cuDF-py against python 3.10 on Databricks (#11477)
NvTimLiu Sep 24, 2024
a34f33e
Enable parquet suites from Spark UT (#11366)
Feng-Jiang28 Sep 24, 2024
5725e2e
Optimzing Expand+Aggregate in sqls with many count distinct (#10798)
binmahone Sep 24, 2024
6a9731f
Use UnaryLike instead of UnaryExpression (#11490)
abellina Sep 24, 2024
10eaa22
Download artifacts via wget (#11503)
NvTimLiu Sep 26, 2024
b113c46
Replace libmamba-solver with mamba command [skip ci] (#11507)
pxLi Sep 26, 2024
c047707
GPU device watermark metrics (#11457)
zpuller Sep 26, 2024
41351c0
Fix FileAlreadyExistsException in LORE dump process (#11484)
ustcfy Sep 27, 2024
9625e58
Deploy all submodules for default sparkver (#11516)
pxLi Sep 27, 2024
fbd4db9
Update from_json to use new cudf features (#11497)
revans2 Sep 27, 2024
f87ccb9
Fixed buildall script (#11515)
razajafri Sep 27, 2024
c74e2dd
Only test years after 1900 for LEGACY mode (#11545)
res-life Sep 30, 2024
5fae883
Fix negative rs. shuffle write time (#11548)
abellina Oct 1, 2024
8207f7b
Update test now that code is fixed (#11496)
revans2 Oct 1, 2024
b9b7629
Fix test case unix_timestamp(col, 'yyyyMMdd') failed for Africa/Casab…
res-life Oct 8, 2024
b715ef2
Have "dump always" dump input files before trying to decode them (#11…
revans2 Oct 9, 2024
670d8ec
Merge branch-24.10 into main
nvauto Oct 9, 2024
85db666
Change version to 24.10.0
nvauto Oct 9, 2024
ec9d008
backport fixes of #11573 to branch 24.10 (#11588)
binmahone Oct 11, 2024
8254f63
update doc for 2410 release (#11582)
nvliyuan Oct 14, 2024
3c9bb8b
Update rapids JNI and private dependency to 24.10.0 (#11576)
nvauto Oct 14, 2024
b535f2d
Update latest changelog [skip ci] (#11577)
nvauto Oct 14, 2024
4007f2c
Merge branch 'branch-24.10' into main
NvTimLiu Oct 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
24 changes: 8 additions & 16 deletions .github/workflows/auto-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,16 @@ name: auto-merge HEAD to BASE
on:
pull_request_target:
branches:
- branch-24.08
- branch-*
types: [closed]

jobs:
auto-merge:
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
ref: branch-24.08 # force to fetch from latest upstream instead of PR ref

- name: auto-merge job
uses: ./.github/workflows/auto-merge
env:
OWNER: NVIDIA
REPO_NAME: spark-rapids
HEAD: branch-24.08
BASE: branch-24.10
AUTOMERGE_TOKEN: ${{ secrets.AUTOMERGE_TOKEN }} # use to merge PR
uses: NVIDIA/spark-rapids-common/.github/workflows/auto-merge.yml@main
with:
owner: ${{ github.repository_owner }}
repo: spark-rapids
branch: ${{ github.event.pull_request.base.ref }}
secrets:
token: ${{ secrets.AUTOMERGE_TOKEN }}
22 changes: 0 additions & 22 deletions .github/workflows/auto-merge/Dockerfile

This file was deleted.

20 changes: 0 additions & 20 deletions .github/workflows/auto-merge/action.yml

This file was deleted.

137 changes: 0 additions & 137 deletions .github/workflows/auto-merge/automerge

This file was deleted.

4 changes: 3 additions & 1 deletion .github/workflows/blossom-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ jobs:
github.actor == 'zpuller' ||
github.actor == 'pxLi' ||
github.actor == 'Feng-Jiang28' ||
github.actor == 'SurajAralihalli'
github.actor == 'SurajAralihalli' ||
github.actor == 'jihoonson' ||
github.actor == 'ustcfy'
)
steps:
- name: Check if comment is issued by authorized person
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,7 @@ elif [[ $SCALA_VER == '2.13' ]]; then
fi
while true; do
{
mvn $COMMON_MVN_FLAGS --file $pom help:evaluate -pl dist \
-Dexpression=included_buildvers \
-DforceStdout -PnoSnapshots -q | tr -d ',' | \
python build/get_buildvers.py "no_snapshots.buildvers" $pom | tr -d ',' | \
xargs -n 1 -I {} bash -c \
"mvn $COMMON_MVN_FLAGS --file $pom -Dbuildver={} de.qaware.maven:go-offline-maven-plugin:resolve-dependencies"

Expand Down
16 changes: 7 additions & 9 deletions .github/workflows/signoff-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,10 @@ jobs:
signoff-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: sigoff-check job
uses: ./.github/workflows/signoff-check
env:
OWNER: NVIDIA
REPO_NAME: spark-rapids
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PULL_NUMBER: ${{ github.event.number }}
- name: signoff
uses: NVIDIA/spark-rapids-common/signoff-check@main
with:
owner: ${{ github.repository_owner }}
repo: spark-rapids
pull_number: ${{ github.event.number }}
token: ${{ secrets.GITHUB_TOKEN }}
22 changes: 0 additions & 22 deletions .github/workflows/signoff-check/Dockerfile

This file was deleted.

19 changes: 0 additions & 19 deletions .github/workflows/signoff-check/action.yml

This file was deleted.

71 changes: 0 additions & 71 deletions .github/workflows/signoff-check/signoff-check

This file was deleted.

3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,5 @@ scalastyle-on-compile.generated.xml
scalastyle-output.xml
scalastyle.txt
target/
cufile.log
cufile.log
build/*.class
Loading