Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Code with very long string gets modified during highlight() #128

Closed
cderv opened this issue Mar 1, 2022 · 2 comments · Fixed by #136
Closed

Code with very long string gets modified during highlight() #128

cderv opened this issue Mar 1, 2022 · 2 comments · Fixed by #136

Comments

@cderv
Copy link
Contributor

cderv commented Mar 1, 2022

This comes from a report in rstudio/distill#442 where someone has a chunk with a very long string (more than 1000 chars).

It seems R has a limitation in getParseData() where string with more than 1000 chars are not stored in the result and replace by something like [1011 chars quoted with ''']. This means that the highlighted code by downlit::highlight() is not the correct one.

See below a reprex:

val <- paste0(rep('very', 200), collapse = " ")
text <- sprintf(" x <- '%s long string'", val)
downlit::highlight(text)
#> [1] " <span class='nv'>x</span> <span class='o'>&lt;-</span> <span class='s'>[1011 chars quoted with ''']</span>"
utils::getParseData(parse(text = text))
#>   line1 col1 line2 col2 id parent       token terminal
#> 7     1    2     1 1019  7      0        expr    FALSE
#> 1     1    2     1    2  1      3      SYMBOL     TRUE
#> 3     1    2     1    2  3      7        expr    FALSE
#> 2     1    4     1    5  2      7 LEFT_ASSIGN     TRUE
#> 4     1    7     1 1019  4      6   STR_CONST     TRUE
#> 6     1    7     1 1019  6      7        expr    FALSE
#>                           text
#> 7                             
#> 1                            x
#> 3                             
#> 2                           <-
#> 4 [1011 chars quoted with ''']
#> 6

val <- paste0(rep('very', 100), collapse = " ")
text <- sprintf(" x <- '%s long string'", val)
downlit::highlight(text)
#> [1] " <span class='nv'>x</span> <span class='o'>&lt;-</span> <span class='s'>'very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very long string'</span>"
utils::getParseData(parse(text = text))
#>   line1 col1 line2 col2 id parent       token terminal
#> 7     1    2     1  519  7      0        expr    FALSE
#> 1     1    2     1    2  1      3      SYMBOL     TRUE
#> 3     1    2     1    2  3      7        expr    FALSE
#> 2     1    4     1    5  2      7 LEFT_ASSIGN     TRUE
#> 4     1    7     1  519  4      6   STR_CONST     TRUE
#> 6     1    7     1  519  6      7        expr    FALSE
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                text
#> 7                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
#> 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 x
#> 3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
#> 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <-
#> 4 'very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very long string'
#> 6

Created on 2022-03-01 by the reprex package (v2.0.1)

This could be an edge case of the usage of downlit because such long string are not encounter often, but this is now a known limitation documented in this issue.

Maybe this could be fixed somehow if highlight() was not using the parsed$data$text but directly the parsed$text or providing text. I did not try to look deeper in why one text is used and not the other.

Probably not high priority, deactivating downlit in distill is the workaround that can be used to avoid the initial issue.
@dmurdoch
Copy link
Contributor

I just came across this in rgl, which uses very long strings as OpenGL shaders. It's not hard to fix: the utils::getParseText() function doesn't have the same length limitation, but you don't want to use it in every case, because it's slow.

I put together a patch and will submit it after a bit more testing.

dmurdoch added a commit to dmurdoch/downlit that referenced this issue Apr 28, 2022
@dmurdoch
Fixes r-lib#128, truncation of long tokens, by using utils::getParseT…
dd3fbc9 
…ext when necessary.
@dmurdoch dmurdoch mentioned this issue Apr 28, 2022
Merged
@cderv
Copy link
Contributor Author

cderv commented Apr 29, 2022

Oh great thanks! 👍

hadley added a commit that referenced this issue Jun 9, 2022
@dmurdoch @hadley
Fixes truncation of long tokens by using utils::getParseText() (#136)
8434f36 
Fixes #128

Co-authored-by: Hadley Wickham <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes truncation of long tokens by using utils::getParseText dmurdoch/downlit
2 participants
@dmurdoch @cderv