Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accidental data sending via closures #333

Open
mschubert opened this issue Sep 26, 2024 · 0 comments
Open

Accidental data sending via closures #333

mschubert opened this issue Sep 26, 2024 · 0 comments
Labels

Comments

@mschubert
Copy link
Owner

Users may inadvertently send hundreds of gigabytes of data via the network by using closures. Consider the following:

do_my_stuff = function() {
    huge_object = runif(1e9)
    my_parallel = function(i) {
    }
    Q(my_parallel, i=1:1000, n_jobs=1000)
}

This will send 7.6 Gb to 1000 workers (= 7.6 Tb total), without any of the workers requiring it.

It should not be that easy to make this mistake.

Related: #200, likely re-introduced with globals package changes close to 9f01bbc

@mschubert mschubert added the bug label Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant