Skip to content

Accidental data sending via closures #333

Open
@mschubert

Description

@mschubert

Users may inadvertently send hundreds of gigabytes of data via the network by using closures. Consider the following:

do_my_stuff = function() {
    huge_object = runif(1e9)
    my_parallel = function(i) {
    }
    Q(my_parallel, i=1:1000, n_jobs=1000)
}

This will send 7.6 Gb to 1000 workers (= 7.6 Tb total), without any of the workers requiring it.

It should not be that easy to make this mistake.

Related: #200, likely re-introduced with globals package changes close to 9f01bbc

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions