Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local processing reports errors only after all calls complete #305

Open
mschubert opened this issue Aug 23, 2023 · 1 comment
Open

Local processing reports errors only after all calls complete #305

mschubert opened this issue Aug 23, 2023 · 1 comment

Comments

@mschubert
Copy link
Owner

Consider the following function

fx = function(x) {
    if (x %% 2 == 0)
        warning(paste("warning on", x))
    if (x %% 3 == 0)
        stop(paste("error on", x))
    x
}

clustermq::Q(fx, 1:10, n_jobs=1)

This fails correctly after 3 calls:

Starting 1 cores ...
Running 10 calculations (4 objs/4.3 Kb common; 1 calls/chunk) ...
Error: 1/3 jobs failed (1 warnings). Stopping.
(Error #3) error on 3
(#2) warning on 2

However, if running with n_jobs=0, all calls are evaluated before the error is evaluated:

Running sequentially ('LOCAL') ...
Error: 3/10 jobs failed (5 warnings). Stopping.
(Error #3) error on 3
(Error #6) error on 6
(Error #9) error on 9
(#2) warning on 2
(#4) warning on 4
(#6) warning on 6
(#8) warning on 8
(#10) warning on 10

This should probably exit early if fail_on_error=TRUE. Ideally, we would also support a custom error handling function supplied by the user (related: #164)

@mschubert
Copy link
Owner Author

Related to that, (also parallel) chunks are currently finished even if they encounter an error; may be worth reconsidering

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant