Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release 24.12 [skip ci] #822

Merged
merged 34 commits into from
Jan 13, 2025
Merged

release 24.12 [skip ci] #822

merged 34 commits into from
Jan 13, 2025

Conversation

YanxuanLiu
Copy link
Collaborator

Merge to main for 24.12 release

Release notes as follows:

  • Enables saving models to cloud storage and precomputed k-NN argument in UMAP.
  • Uses improved precision GPU kernels for mean and variance in logistic regression.
  • Updates RAPIDS dependencies to 24.12.
  • Updates Dataproc notebook and benchmark examples.
  • Multiple bug fixes for multi-gpu nodes, ivf_pq with cagra build, logistic regression training and estimator copy.

NOTE: this PR must be merged as create a merge commit

nvauto and others added 30 commits September 26, 2024 05:21
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
[auto-merge] branch-24.10 to branch-24.12 [skip ci] [bot]
* add example for current support for no import change in notebook and some patches

Signed-off-by: Erik Ordentlich <[email protected]>

* fix notebook and clarify and link readme(s)

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
)

* update etl plugin to 24.10 to pull in some benchmark related fixes, turn off mlflow for benchmarks

Signed-off-by: Erik Ordentlich <[email protected]>

* use import error excpetion

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
…793)

* add cosine to ann docstring about metric.

* rephrase the docstring of ANN to be more accurate

* explain default value euclidean of metric

* improved phrasing

* add helpful information on metric restriction for cagra, with a test case

---------

Signed-off-by: Jinfeng <[email protected]>
…#797)

* update own package versions to 24.12 and pull in 24.12 nightly for ci

Signed-off-by: Erik Ordentlich <[email protected]>

* copyright date

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
…ittent … (#787)

* avoid reinitializing rmm multiple times to resolve some intermittent memory issuespin numpy < 1 in readme

Signed-off-by: Erik Ordentlich <[email protected]>

* move memory resource setting to function and invoke in predict as well

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
…er a hanging issue has been fixed in cuml 24.12 (#799)

* add catching cuda error in knn.py

support test_logistic_regression_nl.py

add environmental variable NCC_DEBUG and CUDA_LAUNCH_BLOCKING

add a test case for triggering hanging issue

add skip testing not expect failure

* revert conftest.py, knn.py, utils.py and get mypy checking passed

* refine test case

---------

Signed-off-by: Jinfeng <[email protected]>
…#804)

* add repeat running to reproduce error locally

Signed-off-by: Jinfeng <[email protected]>

enable pre-merge ci to reproducing nightly ci failed test case

add print statement and remove pytest-xdist to get print out results

fix the failure issue that alignment hasn't be done in cuml 24.12

* fix non-deterministic indexing + inconsistent output of ivfpq refine

* revert run_test.sh

* revise type per comment
* remove standardization_with_cupy for dense case as latest raft stats kernel are working good for calculating mean and stddev

revise cpu memory requirement for tests_large to 256g

revise test_compat to reproduce error, passed on single gpu fails on two gpus due to variance incorrect

* revert changes for PR creation

refine test cases

relax tolerance from 1e-4 to 1e-3 for comparing intercept_ when standardization is true in test_compat

relax tolerance for the second place comparing intercept

---------

Signed-off-by: Jinfeng <[email protected]>
…alty cuml parameter (#807)

* fix LogisticRegression copy ignores penalty

Signed-off-by: Jinfeng <[email protected]>

* demo for general solution per comment

* get copy and its supoort of verbose work for dbscan, lr

* move copy to base class

* remove clutter and checked nightly passed locally

* get copy and its support of verbose passed in kmeans

* linear_regression

* get it work for knn, ann

* get it works for full lr params except float32_inputs and num_workers

* get it work for pca

* get copy works for random_forest

* support umap

* clean

* move a test function to common and get mypy passed

* Update python/tests/test_kmeans.py

Co-authored-by: Rishi C. <[email protected]>

---------

Signed-off-by: Jinfeng <[email protected]>
Co-authored-by: Rishi C. <[email protected]>
* update rapids dependencies

Signed-off-by: Erik Ordentlich <[email protected]>

* update copyright year

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
eordentlich and others added 4 commits December 24, 2024 23:27
* Update notebook, test precomputed_knn, bump threshold

* bump trust_diff

* comments

* type checking

* typo

* signoff

Signed-off-by: Rishi Chandra <[email protected]>

* cleanup

* formatting

* Address comments

* add row_id in pdf

---------

Signed-off-by: Rishi Chandra <[email protected]>
* backwards compat for createDataFrame

* sign

Signed-off-by: Rishi Chandra <[email protected]>

* formatting

* 2025

---------

Signed-off-by: Rishi Chandra <[email protected]>
keep up with spark-rapids init script changes and update to newer
dataproc version

---------

Signed-off-by: Erik Ordentlich <[email protected]>
@YanxuanLiu
Copy link
Collaborator Author

build

@YanxuanLiu YanxuanLiu requested review from eordentlich, rishic3, lijinf2 and pxLi and removed request for eordentlich and rishic3 January 13, 2025 02:30
@YanxuanLiu YanxuanLiu self-assigned this Jan 13, 2025
Copy link
Collaborator

@pxLi pxLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! please merge as create a merge commit

@YanxuanLiu YanxuanLiu merged commit 7c4686c into main Jan 13, 2025
8 checks passed
YanxuanLiu added a commit that referenced this pull request Jan 15, 2025
Additional changes to fix issue of 24.12 RC based on
#822

NOTE: this PR must be merged as `create a merge commit`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants