Don't use a restart to handle stop_on_error = 2L #232

cderv · 2024-12-17T16:00:04Z

After discussing with @lionel- , our conclusion was that invokeRestart("error") may not be necessary.
When stop_on_error = 2L, the code needs to stop on error, which will happen if the handler called in withCallingHandler() returns

So in this case, the handler could simply return NULL.

This solve the traceback problem encountered with latest evaluate and current rlang and knitr logic for entracing.

It also solve the traceback trimming problem that is created when invokeRestart() is used. With current invokeRestart("error", cnd), the sys.frames() that is handled by the traceback is not containing the same environments it seems.

Also calling signalCondition(cnd) in the restart handler was triggering some global handlers registered in Positron or in RStudio IDE - which led to differences in the context where rendering happened.

Which this change, rmarkdown::render() shows a rlang traceback with only the cell's function in all context: Positron, RStudio IDE and R console.

However, the snapshot test has changed, as output is now captured. I think it is ok though.

Regarding #231 and the idea of auto entracing, this was supposed to be done in knitr already. The change to withRestart() in recent evaluate demands an adaptation in knitr. I have a PR to come on knitr side to help with this.

In this case, the handler could simply return NULL

hadley

Thanks so much for diving into this problem and figuring out a fix!

I think my only suggestion would be to make the snapshot a little clearer by doing:

ev <- evaluate("stop(\"1\")\n2", stop_on_error = 2L)

hadley · 2024-12-17T19:04:49Z

Do you have any thoughts on how we might test this to make sure we don't accidentally break RMarkdown tracebacks again in the future?

cderv · 2024-12-18T13:58:59Z

Do you have any thoughts on how we might test this to make sure we don't accidentally break RMarkdown tracebacks again in the future?

I tried a few things, but I am struggling to get a reproducible test for issue, let alone with evaluate only. It seems highly dependant on the context. So I am still unto this.

hadley · 2024-12-18T14:04:45Z

@cderv I think it would be taking taking a suggests dependency on knitr, rmarkdown etc if we could get a good reprex for this. Have you tried callr + rmarkdown::render()? Maybe we could do a snapshot test of the output?

cderv · 2024-12-18T15:14:10Z

Have you tried callr + rmarkdown::render()?

I was trying with evaluate() only by setting calling.handlers as would knitr or rlang do right now.

Maybe we could do a snapshot test of the output?

This is what I am trying, but with testthat using rlang and entracing by default error it seems, I can get something to reproduce the issue we have when using expect_snapshot() or expect_error() + printing.

I think the issue is that with current evaluate logic before this PR, stop() is used in the restart, and this hide the backtrace even if the condition is correctly a rlang_error with one. Anything that catch the condition and print it will no show the problem.

Even the trace cleaning problem is not an issue because it seems expect_error() does the right thing to clean trace.

If we can use callr + rmarkdown or knitr, I'll try that.

Initially, I tried to make a test in knitr instead directly (as this is knitr setting the logic of calling.handlers now) but it is not using testthat so I did not have snaphot available.

tests/testthat/_snaps/conditions/abort-error.txt

tests/testthat/ressources/with-abort-error.Rmd

`options(rlang_backtrace_on_error_report = "full")` is only useful when error is not thrown by render() and kept as cell output (e.g. error = TRUE)

…r = 2L behavior - Using evaluate only - In rmarkdown + knitr context

tests/testthat/_snaps/conditions/abort-error.txt

- Use knitr instead of rmarkdown - Test error thrown and capture in cell output

cderv · 2024-12-20T14:16:27Z

It seems something is different on CI than locally - looking into that.

cderv · 2024-12-20T14:33:19Z

New change reduced the difference, but it seems to be related to some reproducibility on character used

❯ difft.exe .\stop-error-trace-trimmed.txt .\stop-error-trace-trimmed.new.txt        
.\stop-error-trace-trimmed.new.txt --- Text
1 Error in `h()`:             1 Error in `h()`:
2 ! !                         2 ! !
3 Backtrace:                  3 Backtrace:
4     ▆                       4     x
5  1. └─global f()            5  1. \-global f()
6  2.   └─global g()          6  2.   \-global g()
7  3.     └─global h()        7  3.     \-global h()
8 Execution halted            8 Execution halted

I probably need to set some reproducibilty setting when passing to rscript execution 🤔

testthat::local_reproducible_output() needs to be called in the background Rscript process too

because g() was missing

R/evaluate.R

hadley

I've kind of forgotten where we are on this, but this looks like a big improvement so I think we should merge and get on to CRAN soon.

hadley · 2025-01-07T20:50:35Z

tests/testthat/_snaps/conditions/rmd-abort-error.txt

+Error in `h()`:
+! !
+Backtrace:
+ 1. global f()


What do we need to do to display a tree here?

I'm surprised to see the simplified backtrace which is no longer the default in rlang. Do we have a print(err, simplify = "branch") parameter somewhere?

Found the culprit! https://github.com/r-lib/rlang/blob/628f4e4b0d645da22731733e237a979c8295eb08/R/cnd-abort.R#L1477

@cderv Could you test with r-lib/rlang#1769 please? I'm hoping it helps in non-interactive sessions without adverse effects in interactive ones (calling knit() at the console).

Regarding the two snapshot, they are "fixed"

diff --git a/tests/testthat/_snaps/conditions/rmd-abort-error.txt b/tests/testthat/_snaps/conditions/rmd-abort-error.txt index 114acb6..6b87b65 100644 --- a/tests/testthat/_snaps/conditions/rmd-abort-error.txt +++ b/tests/testthat/_snaps/conditions/rmd-abort-error.txt @@ -4,9 +4,11 @@ processing file: ressources/with-abort-error.Rmd Error in `h()`: ! ! Backtrace: - 1. global f() - 2. global g() - 3. global h() + ▆ + 1. └─global f() + 2. └─global g() + 3. └─global h() + 4. └─rlang::abort("!") Quitting from lines 6-10 [unnamed-chunk-1] (ressources/with-abort-error.Rmd) Execution halted

diff --git a/tests/testthat/_snaps/conditions/rmd-stop-error-auto-entrace.txt b/tests/testthat/_snaps/conditions/rmd-stop-error-auto-entrace.txt index c2ea7b0..a1867ee 100644 --- a/tests/testthat/_snaps/conditions/rmd-stop-error-auto-entrace.txt +++ b/tests/testthat/_snaps/conditions/rmd-stop-error-auto-entrace.txt @@ -4,9 +4,10 @@ processing file: ressources/with-stop-error-auto-entrace.Rmd Error in `h()`: ! ! Backtrace: - 1. global f() - 2. global g() - 3. global h() + ▆ + 1. └─global f() + 2. └─global g() + 3. └─global h() Quitting from lines 6-10 [unnamed-chunk-1] (ressources/with-stop-error-auto-entrace.Rmd) Execution halted

I then tested manually and I think I covered most of cases. It does not seem to have adverse affect on interactive session. It looks good to merge.

Maybe I'll update the snapshot once next rlang is release so that it does not prevent evaluate release.

hadley · 2025-01-07T20:52:15Z

tests/testthat/_snaps/conditions/stop-error-no-trace.txt

@@ -0,0 +1,4 @@
+Error in h() : !
+Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> f -> g -> h


I think this is the error you see when a vignette has an error, so that might suggest if we can entrace in a vignette we can get better errors?

Good idea. I'll see if I can adapt knitr for this.

lionel-

LGTM! Letting the error escape feels like the right approach here.

lionel- · 2025-01-08T09:40:22Z

tests/testthat/_snaps/conditions/rmd-abort-error.txt

+Error in `h()`:
+! !
+Backtrace:
+ 1. global f()


I'm surprised to see the simplified backtrace which is no longer the default in rlang. Do we have a print(err, simplify = "branch") parameter somewhere?

for different rlang top env

To allow reproducibility on CI

cderv added 2 commits December 17, 2024 12:01

Don't use a restart to handle stop_on_error = 2L

df10cdf

In this case, the handler could simply return NULL

stop_on_error = 2L make evaluate error with output captured

86325bb

hadley reviewed Dec 17, 2024

View reviewed changes

assign result to value for cleaner snapshot

24e30b6

cderv added 2 commits December 19, 2024 17:26

Add some test about render error and backtrace showing

8774fe2

Add missing Suggests packages

385365f

hadley reviewed Dec 19, 2024

View reviewed changes

tests/testthat/_snaps/conditions/abort-error.txt Outdated Show resolved Hide resolved

hadley reviewed Dec 19, 2024

View reviewed changes

tests/testthat/ressources/with-abort-error.Rmd Outdated Show resolved Hide resolved

cderv added 3 commits December 19, 2024 18:58

when testing rlang::abort(), rlang::global_entrace() is not needed

26c3a68

Remove unneeded option to clarify the test case

7bf2d80

`options(rlang_backtrace_on_error_report = "full")` is only useful when error is not thrown by render() and kept as cell output (e.g. error = TRUE)

Add more test cases for complete coverage and example of stop_on_erro…

6b31084

…r = 2L behavior - Using evaluate only - In rmarkdown + knitr context

hadley reviewed Dec 20, 2024

View reviewed changes

tests/testthat/_snaps/conditions/abort-error.txt Outdated Show resolved Hide resolved

Use knitr and test sewed output

880b664

- Use knitr instead of rmarkdown - Test error thrown and capture in cell output

cderv force-pushed the fix/rlang-traceback branch from bc3e3d7 to 880b664 Compare December 20, 2024 14:04

indicate to rlang top env for trimmed traceback

c9d7a69

cderv added 2 commits December 20, 2024 15:36

Ensure reproducible output in snapshot

9b2a12b

testthat::local_reproducible_output() needs to be called in the background Rscript process too

Fix wrong error in test file

c6ce2c9

because g() was missing

cderv commented Dec 20, 2024

View reviewed changes

R/evaluate.R Show resolved Hide resolved

cderv requested review from hadley and lionel- December 20, 2024 14:59

hadley approved these changes Jan 7, 2025

View reviewed changes

lionel- approved these changes Jan 8, 2025

View reviewed changes

lionel- mentioned this pull request Jan 8, 2025

Simplify reporting of bracktraces during knitting r-lib/rlang#1769

Merged

Evaluate trim by default so test opting out

3a95542

for different rlang top env

cderv added 3 commits January 8, 2025 15:21

scrub traceback source file data

82557b4

To allow reproducibility on CI

remove previous snapshot

d7e3fa8

correctly scrub possible source file path

98ff231

hadley merged commit fa1403c into main Jan 8, 2025
14 checks passed

hadley deleted the fix/rlang-traceback branch January 8, 2025 15:01

kelly-sovacool mentioned this pull request Jan 13, 2025

traceback for rendering R Markdown non-interactively no longer works hadley/adv-r#1792

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't use a restart to handle stop_on_error = 2L #232

Don't use a restart to handle stop_on_error = 2L #232

cderv commented Dec 17, 2024

hadley left a comment

hadley commented Dec 17, 2024

cderv commented Dec 18, 2024

hadley commented Dec 18, 2024

cderv commented Dec 18, 2024

cderv commented Dec 20, 2024

cderv commented Dec 20, 2024

hadley left a comment

hadley Jan 7, 2025

lionel- Jan 8, 2025

lionel- Jan 8, 2025

lionel- Jan 8, 2025 •

edited

Loading

cderv Jan 8, 2025

hadley Jan 7, 2025

cderv Jan 8, 2025

lionel- left a comment

lionel- Jan 8, 2025

		@@ -0,0 +1,4 @@
		Error in h() : !
		Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> f -> g -> h

Don't use a restart to handle stop_on_error = 2L #232

Don't use a restart to handle stop_on_error = 2L #232

Conversation

cderv commented Dec 17, 2024

hadley left a comment

Choose a reason for hiding this comment

hadley commented Dec 17, 2024

cderv commented Dec 18, 2024

hadley commented Dec 18, 2024

cderv commented Dec 18, 2024

cderv commented Dec 20, 2024

cderv commented Dec 20, 2024

hadley left a comment

Choose a reason for hiding this comment

hadley Jan 7, 2025

Choose a reason for hiding this comment

lionel- Jan 8, 2025

Choose a reason for hiding this comment

lionel- Jan 8, 2025

Choose a reason for hiding this comment

lionel- Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

cderv Jan 8, 2025

Choose a reason for hiding this comment

hadley Jan 7, 2025

Choose a reason for hiding this comment

cderv Jan 8, 2025

Choose a reason for hiding this comment

lionel- left a comment

Choose a reason for hiding this comment

lionel- Jan 8, 2025

Choose a reason for hiding this comment

lionel- Jan 8, 2025 •

edited

Loading