Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

write_qsf() currently enables only encoding UTF-8 that doesn't handle Hebrew and Arabic #336

Open
saritpery opened this issue Dec 25, 2023 · 16 comments
Open

write_qsf() currently enables only encoding UTF-8 that doesn't handle Hebrew and Arabic #336

saritpery opened this issue Dec 25, 2023 · 16 comments

Comments

@saritpery
Copy link

Hi,
Thank you very much for the great write_qsf().
Can you add other encoding options?
Most of our surveys are in Hebrew and Arabic. When I save a qsf file manually from Qualtrics, it's saved and restored very well, but using the write_qsf() doesn't handle it well.
Is there an option to add an encoding option?
Here is how it looks in the qsf file generated by write_qsf(). Look in the description to see the bad encoding :(
{
"Type": "Branch",
"FlowID": "FL_8",
"Description": "New Branch",
"BranchLogic": {
"0": {
"0": {
"LogicType": "Question",
"QuestionID": "QID49",
"QuestionIsInLoop": "no",
"ChoiceLocator": "q://QID49/SelectableChoice/1",
"Operator": "Selected",
"QuestionIDFromLocator": "QID49",
"LeftOperand": "q://QID49/SelectableChoice/1",
"Type": "Expression",
"Description": "<span class="ConjDesc">If</span> <span class="QuestionDesc"><U+05D4><U+05E0><U+05D9><U+05E1><U+05D5><U+05D9> <U+05E0><U+05D2><U+05DE><U+05E8>, <U+05D0><U+05DA> <U+05D1><U+05DE><U+05D9><U+05D3><U+05D4> <U+05D5><U+05D0><U+05EA>/<U+05D4> <U+05E8><U+05D5><U+05E6><U+05D4> <U+05DC><U+05E4><U+05EA><U+05D5><U+05E8> <U+05D0><U+05EA> <U+05D4><U+05E7><U+05D5><U+05E0><U+05E4><U+05DC><U+05D9><U+05E7><U+05D8> <U+05D1><U+05D3><U+05E8><U+05DA> <U+05D9><U+05E6><U+05D9><U+05E8><U+05EA><U+05D9><U+05EA> <U+05D0><U+05D7><U+05E8><U+05EA>, <U+05D0><U+05E0><U+05D4> <U+05DC><U+05D7><U+05E6>/<U+05D9> <U+05E2><U+05DC> <U+05DB><U+05E4><U+05EA><U+05D5><U+05E8> <U+05D4><U+05DE><U+05E9><U+05DA>, <U+05D1>...</span> <span class="LeftOpDesc"><U+05D4><U+05DE><U+05E9><U+05DA></span> <span class="OpDesc">Is Selected</span> "
},
"Type": "If"
},
"Type": "BooleanExpression"
},
@juliasilge
Copy link
Collaborator

Do I remember correctly that you installed from #319 to use write_qsf()?

I see that we are using encoding = "UTF-8" within that new function. Maybe instead we can have an argument like encoding = "UTF-8" so we have a default but folks can pass in their own encoding when needed? What do you think about that @jmobrien?

@juliasilge juliasilge mentioned this issue Dec 28, 2023
Open
@saritpery
Copy link
Author

saritpery commented Dec 28, 2023 via email

@jmobrien
Copy link
Collaborator

jmobrien commented Dec 29, 2023 •
edited
Loading

Hi @saritpery & @juliasilge , responding here to this, as well as to pick back up with a response to #335 now that the holidays have passed.

If I can, @saritpery, would it be possible for me to again look at pairs of qsfs (from the web and from the API) using this new write_qsf() framework from PR #319? I think that could help me get clear on what's going on.

That said--after starting to write this response, I looked at a few things that, I think, have made what I'm seeing click into place for me, or at least give me a working theory of what's going on. I'm now thinking that, rather than this being an encoding issue per se, it may center around how Qualtrics themselves is choosing to do things under the JSON standard (QSF files are JSON internally). We still have something to fix, but it may be different than providing encoding choices.

https://en.wikipedia.org/wiki/JSON#Character_encoding

Officially, JSON files are supposed to use UTF-8 for their overall encoding. In theory, this means it should be possible for JSON (QSF) files to directly represent any Unicode character, including Hebrew and/or Arabic text. That said, the JSON standard also includes syntax for an escaping sequence \u[xxxx] that allows any Unicode character to be represented in the ASCII subset as its UTF-16 notation. So, for example, the Hebrew letter He (ה) could validly be expressed in JSON as "\u05d4".

In the working, web-downloaded qsf files @saritpery provided in #335, that's exactly what I saw: a file that overall appeared to be in UTF-8 (and so which, again, in theory could have included Hebrew & Arabic text natively), but which seemed to consistently use this ASCII escaping notation for the text of questions in Hebrew or Arabic.

In other words, I think this is Qualtrics's doing--they're choosing to write out their (JSON) QSF's using escape sequences for any Unicode characters.

By comparison, looking at the copied output of write_qsf() @saritpery provided just above, the first character in the question is, once again, the He/ה--but it written as "<U+05D4>" rather than the JSON-compliant "/u05d4". In the non-working files from the older fetch_qsf() provided in #335, I saw the same thing.

So, I'm now thinking our central problem might actually be that the tools we're using to get the data from the API endpoint (R packages httr and jsonlite) are somehow producing non-compliant JSON. I don't know why yet, but at least I know what to try and fix.

If we can get that, it may be that we just want to stick with UTF-8 for overall encoding, since that's the JSON standard and should respect most all use cases (?).

A separate question is whether we would want to stick with what Qualtrics seems to do with web downloads using these \uXXXX escape sequences, or whether we'd want to find a way to output native Unicode characters so the QSF's are more human-readable (taking into consideration, of course, whether Qualtrics themselves will accept a QSF upload that directly incorporates Unicode characters. I don't yet know.)

@juliasilge
Copy link
Collaborator

Ah, that's very interesting @jmobrien 👀 and it sounds like we may well want to stick with UTF-8. Hopefully if @saritpery can provide another pair of QSFs, that will help shine some light on why we are not getting correct escape sequences.

@jmobrien
Copy link
Collaborator

jmobrien commented Jan 4, 2024

Stranger still--I reuploaded @saritpery 's "s1 manual WORKING.qsf" to a different account, then redownloaded the qsf using the web interface and our new write_qsf(). I also did another API download, using the post-download "pretty" argument to restructure things, just to test that. Files are uploaded below with new extension JSON so that github will accept them.

With the web download, it looks like I got the same thing as before, using the JSON-style escape codes for the Hebrew and Arabic characters. So, that part is somewhat confirmed--this is how Qualtrics structures their qsf's, I guess.

On the other hand, in the API downloaded files, I'm now seeing the actual Hebrew and Arabic text, not the <U+XXXX> codes that @saritpery saw. I also tried to re-upload these using the web interface again, and they seemed to work fine.

Possibly a Unix-style vs. Windows thing? My tests was on a Mac so I'm on the Unix side. @saritpery what OS have you been using here?

qsf_test_api_pretty.qsf.json
qsf_test_api.qsf.json
qsf_test_webdownload.qsf.json

@saritpery
Copy link
Author

saritpery commented Jan 15, 2024 via email

@saritpery
Copy link
Author

saritpery commented Jan 15, 2024 via email

@saritpery
Copy link
Author

saritpery commented Jan 15, 2024 via email •
edited
Loading

@saritpery
Copy link
Author

Web generated _Hebrew_questions_example.qsf.json
Screenshot_questions
API generated _Hebrew_questions_example.qsf.json

@jmobrien
Copy link
Collaborator

Okay, I'm now thinking this has something to do with the spot where httr::content() makes a call to iconv() (inside parse_text()) (see the info for argument sub at ?iconv).

@saritpery, to help me figure this out, could you run this reprex after replacing the parts in "<<>>" with the appropriate information? You'll have to install the package reprex if you don't already have it, but then this should run and copy the result directly to your clipboard.

Do make sure to delete the API credentials before posting here.

# Run this reprex:
reprex::reprex({
  
  # Initial setup:
  require(qualtRics)
  require(stringr)
  require(httr)
  Extract <- `[`
  
  # Locale info:
  l10n_info()
  Sys.getlocale("LC_CTYPE")
  .Platform
  
  # Use if not saved, else can skip.  Also, REMOVE BEFORE POSTING THE REPREX OUTPUT:
  qualtrics_api_credentials("<<YOUR_API_KEY>>", "<<YOUR_BASE_URL>>")
  
  # Replace with a relevant surveyID of your own: 
  description_url <-
    generate_url(query = "fetchdescription", surveyID = "<<YOUR_SURVEYID>>")
  
  headers <-
    qualtRics:::construct_header(Sys.getenv("QUALTRICS_API_KEY"))
  
  # Send request to Qualtrics API:
  res <-
   httr::RETRY(
      verb = "GET",
      url = description_url,
      config = httr::add_headers(headers),
      query = list(format = "qsf"),
      times = 4,
      terminate_on = 400:451,
      quiet = TRUE
    )
  
  # Get survey text out the normal way, httr::content:
  text_normal <- content(x = res, as = "text", encoding = "UTF-8")
  
  # Get it out w/o pre-specifying the encoding (shouldn't make a difference):
  text_unspecified <- content(x = res, as = "text")
  
  # Just read the returned binary manually:
  text_manual <- readBin(res$content, character())
  
  # See what the encodings are:
  Encoding(text_normal)
  Encoding(text_unspecified)
  Encoding(text_manual)
  
  # Set up a regex to extract the last question (which should contain some 
  # Arabic/Hebrew characters

  # Take a look at the last question's text for each:
  text_normal |> str_match(rgx) |> Extract(1,2)
  text_unspecified |> str_match(rgx)|> Extract(1,2)
  text_manual |> str_match(rgx)|> Extract(1,2)
 
  # Trying to work out what was happening with iconv():
  
  # This is, in theory, what is done, and shouldn't cause issues:
  iconv_normal <- iconv(text_manual, from = "UTF-8", to = "UTF-8") 
  iconv_normal |> str_match(rgx) |> Extract(1,2)

  # This one should cause the observed issues on my machine:
  iconv_break <- iconv(text_manual, from = "UTF-8", to = "Latin1", sub = "Unicode") 
  iconv_break |> str_match(rgx) |> Extract(1,2)
  
  # Others that might affect Windows?
  iconv_alt1 <- iconv(text_manual, from = "", to = "Latin1") 
  iconv_alt1 |> str_match(rgx) |> Extract(1,2)
  iconv_alt2 <- iconv(text_manual, from = "Latin1", to = "UTF-8") 
  iconv_alt2 |> str_match(rgx) |> Extract(1,2)
  iconv_alt3 <- iconv(text_manual, from = "", to = "UTF-8") 
  iconv_alt3 |> str_match(rgx) |> Extract(1,2)
})

Here's my output from the same. Notice where I can reproduce the observed issue with iconv using the sub = "Unicode" argument. However, internally sub isn't being used by default, so I'm trying to work out how we're getting that kind of replacement.

Might have other ideas depending on what I see above back from @saritpery

 require(qualtRics)
#> Loading required package: qualtRics
 require(stringr)
#> Loading required package: stringr
 require(httr)
#> Loading required package: httr
 Extract <- `[`
 
 # Locale info:
 l10n_info()
#> $MBCS
#> [1] TRUE
#> 
#> $`UTF-8`
#> [1] TRUE
#> 
#> $`Latin-1`
#> [1] FALSE
#> 
#> $codeset
#> [1] "UTF-8"
 Sys.getlocale("LC_CTYPE")
#> [1] "en_US.UTF-8"
 .Platform
#> $OS.type
#> [1] "unix"
#> 
#> $file.sep
#> [1] "/"
#> 
#> $dynlib.ext
#> [1] ".so"
#> 
#> $GUI
#> [1] "X11"
#> 
#> $endian
#> [1] "little"
#> 
#> $pkgType
#> [1] "mac.binary.big-sur-x86_64"
#> 
#> $path.sep
#> [1] ":"
#> 
#> $r_arch
#> [1] ""

# DELETED UNNEEDED STUFF FROM HERE 
 
 # Get survey text out the normal way, httr::content:
 text_normal <- content(x = res, as = "text", encoding = "UTF-8")
 
 # Get it out w/o pre-specifying the encoding (shouldn't make a difference)
 text_unspecified <- content(x = res, as = "text")
#> No encoding supplied: defaulting to UTF-8.
 
 # Just read the returned binary manually:
 text_manual <- readBin(res$content, character())
 
 # See what the encodings are:
 Encoding(text_normal)
#> [1] "UTF-8"
 Encoding(text_unspecified)
#> [1] "UTF-8"
 Encoding(text_manual)
#> [1] "unknown"
 
 # Take a look at the last question's text (which should contain some 
 # Arabic/Hebrew characters)
 rgx <- '.*("QuestionText_Unsafe":".*"),"DataExportTag"'
 text_normal |> str_match(rgx) |> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"<div><span lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">سنسألك الان عن انطباعك من الشخص المتعارف عليه بالحرف&nbsp;</span><span dir=\\\"LTR\\\" lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">&nbsp;E</span></div>\""
 text_unspecified |> str_match(rgx)|> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"<div><span lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">سنسألك الان عن انطباعك من الشخص المتعارف عليه بالحرف&nbsp;</span><span dir=\\\"LTR\\\" lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">&nbsp;E</span></div>\""
 text_manual |> str_match(rgx)|> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"<div><span lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">سنسألك الان عن انطباعك من الشخص المتعارف عليه بالحرف&nbsp;</span><span dir=\\\"LTR\\\" lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">&nbsp;E</span></div>\""

 # Trying to work out what was happening with iconv():
 
 # This is, in theory, what is done, and shouldn't cause issues:
 iconv_normal <- iconv(text_manual, from = "UTF-8", to = "UTF-8") 
 iconv_normal |> str_match(rgx) |> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"<div><span lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">سنسألك الان عن انطباعك من الشخص المتعارف عليه بالحرف&nbsp;</span><span dir=\\\"LTR\\\" lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">&nbsp;E</span></div>\""

 # This one should cause the observed issues on Joseph's machine:
 iconv_break <- iconv(text_manual, from = "UTF-8", to = "Latin1", sub = "Unicode") 
 iconv_break |> str_match(rgx) |> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"<div><span lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\"><U+0633><U+0646><U+0633><U+0623><U+0644><U+0643> <U+0627><U+0644><U+0627><U+0646> <U+0639><U+0646> <U+0627><U+0646><U+0637><U+0628><U+0627><U+0639><U+0643> <U+0645><U+0646> <U+0627><U+0644><U+0634><U+062E><U+0635> <U+0627><U+0644><U+0645><U+062A><U+0639><U+0627><U+0631><U+0641> <U+0639><U+0644><U+064A><U+0647> <U+0628><U+0627><U+0644><U+062D><U+0631><U+0641>&nbsp;</span><span dir=\\\"LTR\\\" lang=\\\"AR-SA\\\" style=\\\"font-size: 19px; font-weight: 700; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; line-height: 21.85px; color: rgb(34, 34, 34);\\\">&nbsp;E</span></div>\""
 
 # Others that might affect Windows?
 iconv_alt1 <- iconv(text_manual, from = "", to = "Latin1") 
 iconv_alt1 |> str_match(rgx) |> Extract(1,2)
#> [1] NA
 iconv_alt2 <- iconv(text_manual, from = "Latin1", to = "UTF-8") 
 iconv_alt2 |> str_match(rgx) |> Extract(1,2)
#> [1] "\"QuestionText_Unsafe\":\"שפת ×\u0090×\u009d\""

Created on 2024-01-17 with reprex v2.0.2

@saritpery
Copy link
Author

saritpery commented Jan 18, 2024 via email

@saritpery
Copy link
Author

reprex output:

# Initial setup:
require(qualtRics)
#> Loading required package: qualtRics
require(stringr)
#> Loading required package: stringr
require(purrr)
#> Loading required package: purrr
require(httr)
#> Loading required package: httr

# Replace with own surveyID:
description_url <-
  generate_url(query = "fetchdescription", surveyID = "SV_38YiuYodnW7N3L0")

headers <- qualtRics:::construct_header(Sys.getenv("QUALTRICS_API_KEY"))

# Send request to Qualtrics API:
res <- httr::RETRY(
  verb = "GET", url = description_url,
  config = httr::add_headers(headers), query = list(format = "qsf"),
  times = 4, terminate_on = 400:451, quiet = TRUE
)

# Possible match for your encoding: 
local.enc <- iconvlist() |> str_subset("1252") |> pluck(1)

# Just read the returned binary manually:
text_manual <- readBin(res$content, character())
# use iconv() in the problematic way
text_normal <- iconv(text_manual, from = "UTF-8", to = "UTF-8")
# Try an alternative
text_alt1 <- iconv(text_manual, from = local.enc, to = "UTF-8")
# Try another alternative
text_alt2 <- iconv(text_manual, from = "Latin1", to = "UTF-8")

# Look at them (if at all possible, please paste on Github web--this will be a huge mess if it's pasted and sent via email)
# (Also, look at these printing directly to the console.  Again, will be massive, but the stuff we saw last time 
# should be near the end of each's printed output:
text_manual
#> [1] "{\"meta\":{\"httpStatus\":\"200 - OK\",\"requestId\":\"adc16732-aaeb-4e59-a668-d616b5d69dca\"},\"result\":{\"SurveyEntry\":{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"SurveyName\":\"Single Heb Question\",\"SurveyDescription\":null,\"SurveyOwnerID\":\"UR_8lcoxFgxRCldQwd\",\"SurveyBrandID\":\"huji\",\"DivisionID\":null,\"SurveyLanguage\":\"EN\",\"SurveyActiveResponseSet\":\"RS_0JMTBtDXgjwqFGS\",\"SurveyStatus\":\"Inactive\",\"SurveyStartDate\":\"0000-00-00 00:00:00\",\"SurveyExpirationDate\":\"0000-00-00 00:00:00\",\"SurveyCreationDate\":\"2024-01-18 14:20:55\",\"CreatorID\":\"UR_8lcoxFgxRCldQwd\",\"LastModified\":\"2024-01-18 14:29:56\",\"LastAccessed\":\"0000-00-00 00:00:00\",\"LastActivated\":\"0000-00-00 00:00:00\",\"Deleted\":null},\"SurveyElements\":[{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"BL\",\"PrimaryAttribute\":\"Survey Blocks\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":[{\"Type\":\"Default\",\"Description\":\"Default Question Block\",\"ID\":\"BL_d7jUfflEUreYha6\",\"BlockElements\":[{\"Type\":\"Question\",\"QuestionID\":\"QID1\"}],\"Options\":{\"BlockLocking\":\"false\",\"RandomizeQuestions\":\"false\",\"BlockVisibility\":\"Expanded\"}},{\"Type\":\"Trash\",\"Description\":\"Trash / Unused Questions\",\"ID\":\"BL_3qn3tQbCogHFQBE\"}]},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"FL\",\"PrimaryAttribute\":\"Survey Flow\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"Flow\":[{\"ID\":\"BL_d7jUfflEUreYha6\",\"Type\":\"Block\",\"FlowID\":\"FL_2\"}],\"Properties\":{\"Count\":2},\"FlowID\":\"FL_1\",\"Type\":\"Root\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PL\",\"PrimaryAttribute\":\"Preview Link\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"PreviewType\":\"Brand\",\"PreviewID\":\"7bc492cc-7577-4fae-a270-bed9316342e9\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SO\",\"PrimaryAttribute\":\"Survey Options\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"BackButton\":\"false\",\"SaveAndContinue\":\"true\",\"SurveyProtection\":\"PublicSurvey\",\"BallotBoxStuffingPrevention\":\"false\",\"NoIndex\":\"Yes\",\"SecureResponseFiles\":\"true\",\"SurveyExpiration\":\"None\",\"SurveyTermination\":\"DefaultMessage\",\"Header\":\"\",\"Footer\":\"\",\"ProgressBarDisplay\":\"None\",\"PartialData\":\"+1 week\",\"ValidationMessage\":\"\",\"PreviousButton\":\"\",\"NextButton\":\"\",\"SurveyTitle\":\"Qualtrics Survey | Qualtrics Experience Management\",\"SkinLibrary\":\"Qualtrics\",\"SkinType\":\"templated\",\"Skin\":{\"brandingId\":null,\"templateId\":\"*base\",\"overrides\":null},\"NewScoring\":1,\"SurveyMetaDescription\":\"The most powerful, simple and trusted way to gather experience data. Start your journey to experience management and try a free account today.\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SCO\",\"PrimaryAttribute\":\"Scoring\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"ScoringCategories\":[],\"ScoringCategoryGroups\":[],\"ScoringSummaryCategory\":null,\"ScoringSummaryAfterQuestions\":0,\"ScoringSummaryAfterSurvey\":0,\"DefaultScoringCategory\":null,\"AutoScoringCategory\":null}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PROJ\",\"PrimaryAttribute\":\"CORE\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":\"1.1.0\",\"Payload\":{\"ProjectCategory\":\"CORE\",\"SchemaVersion\":\"1.1.0\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"STAT\",\"PrimaryAttribute\":\"Survey Statistics\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"MobileCompatible\":true,\"ID\":\"Survey Statistics\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"QC\",\"PrimaryAttribute\":\"Survey Question Count\",\"SecondaryAttribute\":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך<br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"

text_normal
#> [1] "{\"meta\":{\"httpStatus\":\"200 - OK\",\"requestId\":\"adc16732-aaeb-4e59-a668-d616b5d69dca\"},\"result\":{\"SurveyEntry\":{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"SurveyName\":\"Single Heb Question\",\"SurveyDescription\":null,\"SurveyOwnerID\":\"UR_8lcoxFgxRCldQwd\",\"SurveyBrandID\":\"huji\",\"DivisionID\":null,\"SurveyLanguage\":\"EN\",\"SurveyActiveResponseSet\":\"RS_0JMTBtDXgjwqFGS\",\"SurveyStatus\":\"Inactive\",\"SurveyStartDate\":\"0000-00-00 00:00:00\",\"SurveyExpirationDate\":\"0000-00-00 00:00:00\",\"SurveyCreationDate\":\"2024-01-18 14:20:55\",\"CreatorID\":\"UR_8lcoxFgxRCldQwd\",\"LastModified\":\"2024-01-18 14:29:56\",\"LastAccessed\":\"0000-00-00 00:00:00\",\"LastActivated\":\"0000-00-00 00:00:00\",\"Deleted\":null},\"SurveyElements\":[{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"BL\",\"PrimaryAttribute\":\"Survey Blocks\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":[{\"Type\":\"Default\",\"Description\":\"Default Question Block\",\"ID\":\"BL_d7jUfflEUreYha6\",\"BlockElements\":[{\"Type\":\"Question\",\"QuestionID\":\"QID1\"}],\"Options\":{\"BlockLocking\":\"false\",\"RandomizeQuestions\":\"false\",\"BlockVisibility\":\"Expanded\"}},{\"Type\":\"Trash\",\"Description\":\"Trash / Unused Questions\",\"ID\":\"BL_3qn3tQbCogHFQBE\"}]},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"FL\",\"PrimaryAttribute\":\"Survey Flow\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"Flow\":[{\"ID\":\"BL_d7jUfflEUreYha6\",\"Type\":\"Block\",\"FlowID\":\"FL_2\"}],\"Properties\":{\"Count\":2},\"FlowID\":\"FL_1\",\"Type\":\"Root\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PL\",\"PrimaryAttribute\":\"Preview Link\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"PreviewType\":\"Brand\",\"PreviewID\":\"7bc492cc-7577-4fae-a270-bed9316342e9\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SO\",\"PrimaryAttribute\":\"Survey Options\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"BackButton\":\"false\",\"SaveAndContinue\":\"true\",\"SurveyProtection\":\"PublicSurvey\",\"BallotBoxStuffingPrevention\":\"false\",\"NoIndex\":\"Yes\",\"SecureResponseFiles\":\"true\",\"SurveyExpiration\":\"None\",\"SurveyTermination\":\"DefaultMessage\",\"Header\":\"\",\"Footer\":\"\",\"ProgressBarDisplay\":\"None\",\"PartialData\":\"+1 week\",\"ValidationMessage\":\"\",\"PreviousButton\":\"\",\"NextButton\":\"\",\"SurveyTitle\":\"Qualtrics Survey | Qualtrics Experience Management\",\"SkinLibrary\":\"Qualtrics\",\"SkinType\":\"templated\",\"Skin\":{\"brandingId\":null,\"templateId\":\"*base\",\"overrides\":null},\"NewScoring\":1,\"SurveyMetaDescription\":\"The most powerful, simple and trusted way to gather experience data. Start your journey to experience management and try a free account today.\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SCO\",\"PrimaryAttribute\":\"Scoring\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"ScoringCategories\":[],\"ScoringCategoryGroups\":[],\"ScoringSummaryCategory\":null,\"ScoringSummaryAfterQuestions\":0,\"ScoringSummaryAfterSurvey\":0,\"DefaultScoringCategory\":null,\"AutoScoringCategory\":null}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PROJ\",\"PrimaryAttribute\":\"CORE\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":\"1.1.0\",\"Payload\":{\"ProjectCategory\":\"CORE\",\"SchemaVersion\":\"1.1.0\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"STAT\",\"PrimaryAttribute\":\"Survey Statistics\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"MobileCompatible\":true,\"ID\":\"Survey Statistics\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"QC\",\"PrimaryAttribute\":\"Survey Question Count\",\"SecondaryAttribute\":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA>\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA><br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA>\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"

text_alt1
#> [1] "{\"meta\":{\"httpStatus\":\"200 - OK\",\"requestId\":\"adc16732-aaeb-4e59-a668-d616b5d69dca\"},\"result\":{\"SurveyEntry\":{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"SurveyName\":\"Single Heb Question\",\"SurveyDescription\":null,\"SurveyOwnerID\":\"UR_8lcoxFgxRCldQwd\",\"SurveyBrandID\":\"huji\",\"DivisionID\":null,\"SurveyLanguage\":\"EN\",\"SurveyActiveResponseSet\":\"RS_0JMTBtDXgjwqFGS\",\"SurveyStatus\":\"Inactive\",\"SurveyStartDate\":\"0000-00-00 00:00:00\",\"SurveyExpirationDate\":\"0000-00-00 00:00:00\",\"SurveyCreationDate\":\"2024-01-18 14:20:55\",\"CreatorID\":\"UR_8lcoxFgxRCldQwd\",\"LastModified\":\"2024-01-18 14:29:56\",\"LastAccessed\":\"0000-00-00 00:00:00\",\"LastActivated\":\"0000-00-00 00:00:00\",\"Deleted\":null},\"SurveyElements\":[{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"BL\",\"PrimaryAttribute\":\"Survey Blocks\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":[{\"Type\":\"Default\",\"Description\":\"Default Question Block\",\"ID\":\"BL_d7jUfflEUreYha6\",\"BlockElements\":[{\"Type\":\"Question\",\"QuestionID\":\"QID1\"}],\"Options\":{\"BlockLocking\":\"false\",\"RandomizeQuestions\":\"false\",\"BlockVisibility\":\"Expanded\"}},{\"Type\":\"Trash\",\"Description\":\"Trash / Unused Questions\",\"ID\":\"BL_3qn3tQbCogHFQBE\"}]},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"FL\",\"PrimaryAttribute\":\"Survey Flow\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"Flow\":[{\"ID\":\"BL_d7jUfflEUreYha6\",\"Type\":\"Block\",\"FlowID\":\"FL_2\"}],\"Properties\":{\"Count\":2},\"FlowID\":\"FL_1\",\"Type\":\"Root\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PL\",\"PrimaryAttribute\":\"Preview Link\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"PreviewType\":\"Brand\",\"PreviewID\":\"7bc492cc-7577-4fae-a270-bed9316342e9\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SO\",\"PrimaryAttribute\":\"Survey Options\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"BackButton\":\"false\",\"SaveAndContinue\":\"true\",\"SurveyProtection\":\"PublicSurvey\",\"BallotBoxStuffingPrevention\":\"false\",\"NoIndex\":\"Yes\",\"SecureResponseFiles\":\"true\",\"SurveyExpiration\":\"None\",\"SurveyTermination\":\"DefaultMessage\",\"Header\":\"\",\"Footer\":\"\",\"ProgressBarDisplay\":\"None\",\"PartialData\":\"+1 week\",\"ValidationMessage\":\"\",\"PreviousButton\":\"\",\"NextButton\":\"\",\"SurveyTitle\":\"Qualtrics Survey | Qualtrics Experience Management\",\"SkinLibrary\":\"Qualtrics\",\"SkinType\":\"templated\",\"Skin\":{\"brandingId\":null,\"templateId\":\"*base\",\"overrides\":null},\"NewScoring\":1,\"SurveyMetaDescription\":\"The most powerful, simple and trusted way to gather experience data. Start your journey to experience management and try a free account today.\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SCO\",\"PrimaryAttribute\":\"Scoring\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"ScoringCategories\":[],\"ScoringCategoryGroups\":[],\"ScoringSummaryCategory\":null,\"ScoringSummaryAfterQuestions\":0,\"ScoringSummaryAfterSurvey\":0,\"DefaultScoringCategory\":null,\"AutoScoringCategory\":null}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PROJ\",\"PrimaryAttribute\":\"CORE\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":\"1.1.0\",\"Payload\":{\"ProjectCategory\":\"CORE\",\"SchemaVersion\":\"1.1.0\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"STAT\",\"PrimaryAttribute\":\"Survey Statistics\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"MobileCompatible\":true,\"ID\":\"Survey Statistics\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"QC\",\"PrimaryAttribute\":\"Survey Question Count\",\"SecondaryAttribute\":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך<br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"

text_alt2
#> [1] "{\"meta\":{\"httpStatus\":\"200 - OK\",\"requestId\":\"adc16732-aaeb-4e59-a668-d616b5d69dca\"},\"result\":{\"SurveyEntry\":{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"SurveyName\":\"Single Heb Question\",\"SurveyDescription\":null,\"SurveyOwnerID\":\"UR_8lcoxFgxRCldQwd\",\"SurveyBrandID\":\"huji\",\"DivisionID\":null,\"SurveyLanguage\":\"EN\",\"SurveyActiveResponseSet\":\"RS_0JMTBtDXgjwqFGS\",\"SurveyStatus\":\"Inactive\",\"SurveyStartDate\":\"0000-00-00 00:00:00\",\"SurveyExpirationDate\":\"0000-00-00 00:00:00\",\"SurveyCreationDate\":\"2024-01-18 14:20:55\",\"CreatorID\":\"UR_8lcoxFgxRCldQwd\",\"LastModified\":\"2024-01-18 14:29:56\",\"LastAccessed\":\"0000-00-00 00:00:00\",\"LastActivated\":\"0000-00-00 00:00:00\",\"Deleted\":null},\"SurveyElements\":[{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"BL\",\"PrimaryAttribute\":\"Survey Blocks\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":[{\"Type\":\"Default\",\"Description\":\"Default Question Block\",\"ID\":\"BL_d7jUfflEUreYha6\",\"BlockElements\":[{\"Type\":\"Question\",\"QuestionID\":\"QID1\"}],\"Options\":{\"BlockLocking\":\"false\",\"RandomizeQuestions\":\"false\",\"BlockVisibility\":\"Expanded\"}},{\"Type\":\"Trash\",\"Description\":\"Trash / Unused Questions\",\"ID\":\"BL_3qn3tQbCogHFQBE\"}]},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"FL\",\"PrimaryAttribute\":\"Survey Flow\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"Flow\":[{\"ID\":\"BL_d7jUfflEUreYha6\",\"Type\":\"Block\",\"FlowID\":\"FL_2\"}],\"Properties\":{\"Count\":2},\"FlowID\":\"FL_1\",\"Type\":\"Root\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PL\",\"PrimaryAttribute\":\"Preview Link\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"PreviewType\":\"Brand\",\"PreviewID\":\"7bc492cc-7577-4fae-a270-bed9316342e9\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SO\",\"PrimaryAttribute\":\"Survey Options\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"BackButton\":\"false\",\"SaveAndContinue\":\"true\",\"SurveyProtection\":\"PublicSurvey\",\"BallotBoxStuffingPrevention\":\"false\",\"NoIndex\":\"Yes\",\"SecureResponseFiles\":\"true\",\"SurveyExpiration\":\"None\",\"SurveyTermination\":\"DefaultMessage\",\"Header\":\"\",\"Footer\":\"\",\"ProgressBarDisplay\":\"None\",\"PartialData\":\"+1 week\",\"ValidationMessage\":\"\",\"PreviousButton\":\"\",\"NextButton\":\"\",\"SurveyTitle\":\"Qualtrics Survey | Qualtrics Experience Management\",\"SkinLibrary\":\"Qualtrics\",\"SkinType\":\"templated\",\"Skin\":{\"brandingId\":null,\"templateId\":\"*base\",\"overrides\":null},\"NewScoring\":1,\"SurveyMetaDescription\":\"The most powerful, simple and trusted way to gather experience data. Start your journey to experience management and try a free account today.\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SCO\",\"PrimaryAttribute\":\"Scoring\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"ScoringCategories\":[],\"ScoringCategoryGroups\":[],\"ScoringSummaryCategory\":null,\"ScoringSummaryAfterQuestions\":0,\"ScoringSummaryAfterSurvey\":0,\"DefaultScoringCategory\":null,\"AutoScoringCategory\":null}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"PROJ\",\"PrimaryAttribute\":\"CORE\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":\"1.1.0\",\"Payload\":{\"ProjectCategory\":\"CORE\",\"SchemaVersion\":\"1.1.0\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"STAT\",\"PrimaryAttribute\":\"Survey Statistics\",\"SecondaryAttribute\":null,\"TertiaryAttribute\":null,\"Payload\":{\"MobileCompatible\":true,\"ID\":\"Survey Statistics\"}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"QC\",\"PrimaryAttribute\":\"Survey Question Count\",\"SecondaryAttribute\":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A>\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A><br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A>\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"

# An attempt to shorten them w/o changing the encoding (which str_match() does implicitly):

substring(text_manual, nchar(text_manual) - 1000, nchar(text_manual)) 
#> [1] ":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך<br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"
substring(text_normal, nchar(text_normal) - 1000, nchar(text_normal)) 
#> [1] "YiuYodnW7N3L0\",\"Element\":\"QC\",\"PrimaryAttribute\":\"Survey Question Count\",\"SecondaryAttribute\":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA>\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA><br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1><U+05E2><U+05D1><U+05E8><U+05D9><U+05EA> <U+05D5><U+05E2><U+05D5><U+05D3> <U+05D8><U+05E7><U+05E1><U+05D8> <U+05E9><U+05D9><U+05D4><U+05D9><U+05D4> <U+05D9><U+05D5><U+05EA><U+05E8> <U+05D0><U+05E8><U+05D5><U+05DA>\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"
substring(text_alt1, nchar(text_alt1) - 1000, nchar(text_alt1)) 
#> [1] ":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך<br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220לה בעבר×\231ת ועוד ×\230קס×\230 ש×\231×”×\231×” ×\231ותר ×\220רוך\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"
substring(text_alt2, nchar(text_alt2) - 1000, nchar(text_alt2))
#> [1] ":\"1\",\"TertiaryAttribute\":null,\"Payload\":null},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"SQ\",\"PrimaryAttribute\":\"QID1\",\"SecondaryAttribute\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A>\",\"TertiaryAttribute\":null,\"Payload\":{\"QuestionText\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A><br>\",\"DefaultChoices\":false,\"DataExportTag\":\"Q1\",\"QuestionID\":\"QID1\",\"QuestionType\":\"TE\",\"Selector\":\"SL\",\"Configuration\":{\"QuestionDescriptionOption\":\"UseText\"},\"QuestionDescription\":\"A question in Hebrew ש×\220×<U+009C>×<U+0094> ×<U+0091>×¢×<U+0091>ר×<U+0099>ת ×<U+0095>×¢×<U+0095>×<U+0093> ×<U+0098>קס×<U+0098> ש×<U+0099>×<U+0094>×<U+0099>×<U+0094> ×<U+0099>×<U+0095>תר ×\220ר×<U+0095>×<U+009A>\",\"Validation\":{\"Settings\":{\"ForceResponse\":\"OFF\",\"Type\":\"None\"}},\"GradingData\":[],\"Language\":[],\"NextChoiceId\":4,\"NextAnswerId\":1,\"SearchSource\":{\"AllowFreeResponse\":\"false\"}}},{\"SurveyID\":\"SV_38YiuYodnW7N3L0\",\"Element\":\"RS\",\"PrimaryAttribute\":\"RS_0JMTBtDXgjwqFGS\",\"SecondaryAttribute\":\"Default Response Set\",\"TertiaryAttribute\":null,\"Payload\":null}]}}"

Created on 2024-01-18 by the reprex package (v2.0.1)

Session info 
sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value
#>  version  R version 4.1.3 (2022-03-10)
#>  os       Windows 10 x64 (build 22621)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_Israel.1252
#>  ctype    English_Israel.1252
#>  tz       Asia/Jerusalem
#>  date     2024-01-18
#>  pandoc   3.1.1 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> - Packages -------------------------------------------------------------------
#>  ! package     * version    date (UTC) lib source
#>    assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.3)
#>    cli           3.6.1      2023-03-23 [1] CRAN (R 4.1.3)
#>    curl          4.3.2      2021-06-23 [1] CRAN (R 4.0.5)
#>    DBI           1.1.2      2021-12-20 [1] CRAN (R 4.0.5)
#>    digest        0.6.29     2021-12-01 [1] CRAN (R 4.0.5)
#>    dplyr         1.0.8      2022-02-08 [1] CRAN (R 4.0.5)
#>    ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.0.5)
#>    evaluate      0.15       2022-02-18 [1] CRAN (R 4.0.5)
#>    fansi         0.4.1      2020-01-08 [1] CRAN (R 4.0.3)
#>    fastmap       1.1.1      2023-02-24 [1] CRAN (R 4.1.3)
#>    fs            1.5.2      2021-12-08 [1] CRAN (R 4.0.5)
#>    generics      0.1.3      2022-07-05 [1] CRAN (R 4.1.3)
#>    glue          1.6.2      2022-02-24 [1] CRAN (R 4.1.3)
#>    highr         0.9        2021-04-16 [1] CRAN (R 4.0.5)
#>    hms           1.1.1      2021-09-26 [1] CRAN (R 4.0.5)
#>    htmltools     0.5.6      2023-08-10 [1] CRAN (R 4.1.3)
#>    httr        * 1.4.2      2020-07-20 [1] CRAN (R 4.0.3)
#>    insight       0.19.3     2023-06-29 [1] CRAN (R 4.1.3)
#>    jsonlite      1.8.0      2022-02-22 [1] CRAN (R 4.0.5)
#>    knitr         1.38       2022-03-25 [1] CRAN (R 4.0.5)
#>    lifecycle     1.0.3      2022-10-07 [1] CRAN (R 4.1.3)
#>    lubridate     1.8.0      2021-10-07 [1] CRAN (R 4.0.5)
#>    magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.1.3)
#>    pillar        1.9.0      2023-03-22 [1] CRAN (R 4.1.3)
#>    pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.3)
#>    purrr       * 0.3.4      2020-04-17 [1] CRAN (R 4.0.3)
#>    qualtRics   * 3.1.7.9000 2024-01-15 [1] Github (ropensci/qualtRics@05b4b7c)
#>    R.cache       0.15.0     2021-04-30 [1] CRAN (R 4.0.5)
#>    R.methodsS3   1.8.1      2020-08-26 [1] CRAN (R 4.0.3)
#>    R.oo          1.24.0     2020-08-26 [1] CRAN (R 4.0.3)
#>    R.utils       2.11.0     2021-09-26 [1] CRAN (R 4.0.5)
#>    R6            2.5.1      2021-08-19 [1] CRAN (R 4.0.5)
#>    readr         2.1.2      2022-01-30 [1] CRAN (R 4.0.5)
#>    reprex        2.0.1      2021-08-05 [1] CRAN (R 4.0.5)
#>    rlang         1.1.1      2023-04-28 [1] CRAN (R 4.1.3)
#>    rmarkdown     2.13       2022-03-10 [1] CRAN (R 4.0.5)
#>    rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.0.3)
#>    sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.0.5)
#>    sjlabelled    1.2.0      2022-04-10 [1] CRAN (R 4.1.3)
#>  D stringi       1.7.12     2023-01-11 [1] CRAN (R 4.1.3)
#>    stringr     * 1.5.0      2022-12-02 [1] CRAN (R 4.1.3)
#>    styler        1.10.2     2023-08-29 [1] CRAN (R 4.1.3)
#>    tibble        3.1.2      2021-05-16 [1] CRAN (R 4.0.5)
#>    tidyr         1.2.0      2022-02-01 [1] CRAN (R 4.0.5)
#>    tidyselect    1.2.0      2022-10-10 [1] CRAN (R 4.1.3)
#>    tzdb          0.2.0      2021-10-27 [1] CRAN (R 4.0.5)
#>    utf8          1.2.1      2021-03-12 [1] CRAN (R 4.0.5)
#>    vctrs         0.6.3      2023-06-14 [1] CRAN (R 4.1.3)
#>    withr         2.5.0      2022-03-03 [1] CRAN (R 4.0.5)
#>    xfun          0.30       2022-03-02 [1] CRAN (R 4.0.5)
#>    yaml          2.3.5      2022-02-21 [1] CRAN (R 4.0.5)
#> 
#>  [1] C:/Users/sarit/Dropbox/Documents/R/win-library/4.0
#>  [2] C:/Program Files/R/R-4.1.3/library
#> 
#>  D -- DLL MD5 mismatch, broken installation.
#> 
#> ------------------------------------------------------------------------------

Console output:
And again the console is different in text_normal.
In the console I see the Hebrew characters, whereas in the reprex I see the symbols with the plus sign.
"A question in Hebrew <U+05E9><U+05D0><U+05DC><U+05D4> <U+05D1>..."
If you need me to add more of the console output, please just say so...

substring(text_manual, nchar(text_manual) - 1000, nchar(text_manual))

[1] ":"1","TertiaryAttribute":null,"Payload":null},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"SQ","PrimaryAttribute":"QID1","SecondaryAttribute":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך","TertiaryAttribute":null,"Payload":{"QuestionText":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך
","DefaultChoices":false,"DataExportTag":"Q1","QuestionID":"QID1","QuestionType":"TE","Selector":"SL","Configuration":{"QuestionDescriptionOption":"UseText"},"QuestionDescription":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך","Validation":{"Settings":{"ForceResponse":"OFF","Type":"None"}},"GradingData":[],"Language":[],"NextChoiceId":4,"NextAnswerId":1,"SearchSource":{"AllowFreeResponse":"false"}}},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"RS","PrimaryAttribute":"RS_0JMTBtDXgjwqFGS","SecondaryAttribute":"Default Response Set","TertiaryAttribute":null,"Payload":null}]}}"

substring(text_normal, nchar(text_normal) - 1000, nchar(text_normal))

[1] "YiuYodnW7N3L0","Element":"QC","PrimaryAttribute":"Survey Question Count","SecondaryAttribute":"1","TertiaryAttribute":null,"Payload":null},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"SQ","PrimaryAttribute":"QID1","SecondaryAttribute":"A question in Hebrew שאלה בעברית ועוד טקסט שיהיה יותר ארוך","TertiaryAttribute":null,"Payload":{"QuestionText":"A question in Hebrew שאלה בעברית ועוד טקסט שיהיה יותר ארוך
","DefaultChoices":false,"DataExportTag":"Q1","QuestionID":"QID1","QuestionType":"TE","Selector":"SL","Configuration":{"QuestionDescriptionOption":"UseText"},"QuestionDescription":"A question in Hebrew שאלה בעברית ועוד טקסט שיהיה יותר ארוך","Validation":{"Settings":{"ForceResponse":"OFF","Type":"None"}},"GradingData":[],"Language":[],"NextChoiceId":4,"NextAnswerId":1,"SearchSource":{"AllowFreeResponse":"false"}}},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"RS","PrimaryAttribute":"RS_0JMTBtDXgjwqFGS","SecondaryAttribute":"Default Response Set","TertiaryAttribute":null,"Payload":null}]}}"

substring(text_alt1, nchar(text_alt1) - 1000, nchar(text_alt1))

[1] ":"1","TertiaryAttribute":null,"Payload":null},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"SQ","PrimaryAttribute":"QID1","SecondaryAttribute":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך","TertiaryAttribute":null,"Payload":{"QuestionText":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך
","DefaultChoices":false,"DataExportTag":"Q1","QuestionID":"QID1","QuestionType":"TE","Selector":"SL","Configuration":{"QuestionDescriptionOption":"UseText"},"QuestionDescription":"A question in Hebrew ש×\u0090לה בעברית ועוד טקסט שיהיה יותר ×\u0090רוך","Validation":{"Settings":{"ForceResponse":"OFF","Type":"None"}},"GradingData":[],"Language":[],"NextChoiceId":4,"NextAnswerId":1,"SearchSource":{"AllowFreeResponse":"false"}}},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"RS","PrimaryAttribute":"RS_0JMTBtDXgjwqFGS","SecondaryAttribute":"Default Response Set","TertiaryAttribute":null,"Payload":null}]}}"

substring(text_alt2, nchar(text_alt2) - 1000, nchar(text_alt2))

[1] ":"1","TertiaryAttribute":null,"Payload":null},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"SQ","PrimaryAttribute":"QID1","SecondaryAttribute":"A question in Hebrew ש×\u0090×\u009c×\u0094 ×\u0091×¢×\u0091ר×\u0099ת ×\u0095×¢×\u0095×\u0093 ×\u0098קס×\u0098 ש×\u0099×\u0094×\u0099×\u0094 ×\u0099×\u0095תר ×\u0090ר×\u0095×\u009a","TertiaryAttribute":null,"Payload":{"QuestionText":"A question in Hebrew ש×\u0090×\u009c×\u0094 ×\u0091×¢×\u0091ר×\u0099ת ×\u0095×¢×\u0095×\u0093 ×\u0098קס×\u0098 ש×\u0099×\u0094×\u0099×\u0094 ×\u0099×\u0095תר ×\u0090ר×\u0095×\u009a
","DefaultChoices":false,"DataExportTag":"Q1","QuestionID":"QID1","QuestionType":"TE","Selector":"SL","Configuration":{"QuestionDescriptionOption":"UseText"},"QuestionDescription":"A question in Hebrew ש×\u0090×\u009c×\u0094 ×\u0091×¢×\u0091ר×\u0099ת ×\u0095×¢×\u0095×\u0093 ×\u0098קס×\u0098 ש×\u0099×\u0094×\u0099×\u0094 ×\u0099×\u0095תר ×\u0090ר×\u0095×\u009a","Validation":{"Settings":{"ForceResponse":"OFF","Type":"None"}},"GradingData":[],"Language":[],"NextChoiceId":4,"NextAnswerId":1,"SearchSource":{"AllowFreeResponse":"false"}}},{"SurveyID":"SV_38YiuYodnW7N3L0","Element":"RS","PrimaryAttribute":"RS_0JMTBtDXgjwqFGS","SecondaryAttribute":"Default Response Set","TertiaryAttribute":null,"Payload":null}]}}"

@ropensci ropensci deleted a comment from saritpery Jan 18, 2024
@jmobrien
Copy link
Collaborator

(Good catch, thanks; I've deprecated that one)

Okay, that's informative. Last one for now, before I think I can look into this via other ways--what does this do? Again, both programmatic and copied console output.

Also, I may have found a solution for you, see below. That said, if you decide to try out the potential fix, would you mind running this reprex first? Could still give us some important information for other users who might also be in your situation.

reprex::reprex({

test_raw_windows <- 
  charToRaw("שלום")

test_raw_windows

# See if it makes the round-trip:
readBin(test_raw_windows, character())

# From Joseph's, in UTF-8
test_raw_utf8 <- 
  c("d7", "a9", "d7", "9c", "d7", "95", "d7", "9d") |> 
  as.hexmode() |> 
  as.raw()

readBin(test_raw_utf8, character())

})

As far as a fix for you, are you and/or someone able to update your R setup, specifically R itself? I noticed you're using R version 4.1.3. Starting with 4.2.0, CRAN made some changes that allows using UTF-8 more natively within Windows. Here's a post I found that describes that change, plus some of the further updates/bugfixes since then (the clipboard/console issue may also have been fixed? And the current version is 4.3.2, so there's likely more, though I haven't inspected the changelogs closely):

https://blog.r-project.org/2022/06/16/upcoming-changes-in-r-4.2.1-on-windows/

Might want to try that. (Do note, though, that it seems like it was a fundamental enough change on the Windows side that you'll need to fully rebuild your R setup & packages. Just a heads-up in case you have critical projects that mean you can't risk temporarily breaking your setup).

@saritpery
Copy link
Author

Hi @jmobrien ,
Thank you so much for pursuing this. I'm on board in assisting in any way I can.

Running it on console:

test_raw_windows <-

  • charToRaw("שלום")

test_raw_windows
[1] d7 a9 d7 9c d7 95 d7 9d

See if it makes the round-trip:

readBin(test_raw_windows, character())
[1] "שלו×\u009d"

From Joseph's, in UTF-8

test_raw_utf8 <-

  • c("d7", "a9", "d7", "9c", "d7", "95", "d7", "9d") |> 

  • as.hexmode() |> 

  • as.raw()

readBin(test_raw_utf8, character())
[1] "שלו×\u009d"

Clipboard paste from reprex:

test_raw_windows <- 
  charToRaw("\u5e9\u5dc\u5d5\u5dd")

test_raw_windows
#> [1] d7 a9 d7 9c d7 95 d7 9d

# See if it makes the round-trip:
readBin(test_raw_windows, character())
#> [1] "שלו×\235"

# From Joseph's, in UTF-8
test_raw_utf8 <- 
  c("d7", "a9", "d7", "9c", "d7", "95", "d7", "9d") |> 
  as.hexmode() |> 
  as.raw()

readBin(test_raw_utf8, character())
#> [1] "שלו×\235"

Created on 2024-01-21 by the reprex package (v2.0.1)

@jmobrien
Copy link
Collaborator

Thanks, this is useful.

As far as your situation, did updating the R version help?

@saudiwin
Copy link

I don't have any offers in terms of ideas to fix this, but just wanted to say that Arabic (and I assume Hebrew) has been a headache in R for a long time. Newer R versions though should be better than old ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jmobrien @saudiwin @juliasilge @saritpery