Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jvm-packages] Supports external memory #11186

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

wbo4958
Copy link
Contributor

@wbo4958 wbo4958 commented Jan 27, 2025

Supporting ext memory which is based on #11181

@wbo4958
Copy link
Contributor Author

wbo4958 commented Jan 27, 2025

Hi @trivialfis, please help review

@trivialfis
Copy link
Member

trivialfis commented Feb 8, 2025

Note:

  • The external memory is only supported by GPU at the moment; we should make this clear.
  • Must enable RMM.
  • Need global configuration.

@trivialfis
Copy link
Member

We need to let XGBoost access as many CPU threads as available.

@trivialfis
Copy link
Member

  • The default nthread parameter in the spark package XGBoost estimator is 1.
  • There are still other factors limiting the openmp threads in my test run. Perhaps Spark is setting environment variables for executors?

@trivialfis
Copy link
Member

For reference, the --conf spark.task.cpus=2 affects the global OpenMP runtime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants