overwrite-modify-beta substring is not handling UTF-8 Characters correctly #1064
Comments
sodre
added a commit
to sodre/jolt
that referenced
this issue
Apr 12, 2021
sodre
added a commit
to sodre/jolt
that referenced
this issue
Apr 12, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
First, thank you for creating Jolt. It is an extremely helpful library and we use I use it constantly in my NiFi dataflows.
Unfortunately, I found a bug in modify-overwrite-beta while handling text with UTF-8 characters.
Please see the example below:
Input
{ "text" : "Video length 2:19👇 authentication \n\nThe_Castle_Runs_RED_yes🔴 https://t.co/MZWmLJLggL", "entities" : { "UrlTxT" : [ { "indices" : [ 61, 73 ] } ], "TimeTxT" : [ { "indices" : [ 13, 17 ] } ] } }modify-overwrite-beta Transform
[ { "operation": "modify-overwrite-beta", "spec": { "entities": { "*": { "*": { "text": "=substring(@(4,text), @(1,indices[0]), @(1,indices[1]))" } } } } } ]Expected Output
{ "text" : "Video length 2:19👇 authentication \n\nThe_Castle_Runs_RED_yes🔴 https://t.co/MZWmLJLggL", "entities" : { "UrlTxT" : [ { "indices" : [ 61, 73 ], "text" : "https://t.co/MZWmLJLggL" } ], "TimeTxT" : [ { "indices" : [ 13, 17 ], "text" : "2:19" } ] } }Jolt Output from https://jolt-demo.appspot.com/#inception
{ "text" : "Video length 2:19?? authentication \n\nThe_Castle_Runs_RED_yes?? https://t.co/MZWmLJLggL", "entities" : { "UrlTxT" : [ { "indices" : [ 61, 73 ], "text" : "? https://t." } ], "TimeTxT" : [ { "indices" : [ 13, 17 ], "text" : "2:19" } ] } }The text was updated successfully, but these errors were encountered: