Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compare data,table with polars #26

Open
tdhock opened this issue Jan 29, 2024 · 0 comments
Open

compare data,table with polars #26

tdhock opened this issue Jan 29, 2024 · 0 comments

Comments

@tdhock
Copy link
Owner

tdhock commented Jan 29, 2024

for multi-threaded operations, the number of threads used by polars must be set as an option, before loading polars.
pola-rs/r-polars#720 says

#' - Set the `polars.limit_max_threads` option to `FALSE` with
#'   the [options()] function. Same as setting the `POLARS_MAX_THREADS` environment
#'   variable, this option must be set before loading the package.
...
    Sys.setenv(POLARS_MAX_THREADS = 2)
...
pl$threadpool_size() 

in contrast data.table can change number of threads interactively (without having to restart R) using setDTthreads().
Are there situations where a different number of threads is optimal?
say for operation A, 2 threads would be best, but for operation B, 4 threads would be best? in that case data.table could have an advantage. @DorisAmoakohene can you please investigate?
how can we determine optimal number of threads for polars?

  • run atime, save results to disk, quit R, restart R with different number of threads, etc.
  • is it possible to use atime_versions on different versions of polars?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant