Can we provide better error messages for jsoniter?

[adamw]

Currently when a JSON body fails to decode, when using jsoniter, a generic error is returned. Can we somehow extract information from jsoniter, when parsing fails, which would point to the path of the failure? Something like that is already available e.g. for circe.

[kciesielski]

This may be tricky, see here: plokhotnyuk/jsoniter-scala#992
An example jsoniter error looks like:

missing required field "student_take_units", offset: 0x00007255, buf:
+----------+-------------------------------------------------+------------------+
|          |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f | 0123456789abcdef |
+----------+-------------------------------------------------+------------------+
| 00007230 | 73 74 75 64 65 6e 74 5f 74 61 6b 65 5f 75 6e 69 | student_take_uni |
| 00007240 | 74 73 58 58 58 22 3a 20 30 2e 31 33 0a 20 20 20 | tsXXX": 0.13.    |
| 00007250 | 20 20 20 20 20 7d 2c 0a 20 20 20 20 20 20 20 20 |      },.         |
| 00007260 | 7b 0a 20 20 20 20 20 20 20 20 20 20 20 20 22 73 | {.            "s |
| 00007270 | 75 62 6a 65 63 74 5f 63 6f 64 65 22 3a 20 22 57 | ubject_code": "W |
+----------+-------------------------------------------------+------------------+

We could catch jsoniter's JsonReaderException and copy its message until the , offset part (if present).
I'm now testing it with iron support and getting decent errors like

Invalid value for: body (name should not be empty and only made of alphanumeric characters)

[plokhotnyuk]

Most JSON parsers (e.g. circe, zio-json, smithy4s-json) do tracking of the current JSON path during parsing, but it burns a lot of CPU cycles and reduce memory bandwidth.

From the beginning jsoniter-scala was designed to parse UTF-8 input from byte arrays and buffers (not strings) on servers for matured ad-tech protocols like OpenRTB where most of fields are optional or have meaningful default values. Pointing with an exact byte location and a textual dump helped a lot to fix JSON/UTF-8 encoding errors (like malformed bytes or wrong escaping sequences) on the client side.

For some services that works with personal data having those errors logged could be insecure, so in a runtime configuration for parser users can switch off dumping while keeping hex offset printed in the error message.

Also, for debugging purposes the size of hex dumps could be increased using this option.

I vote to preserve hex offset, because it helps to point on problematic places without relaying on uniqueness of field names or developer intuition.

Also, please consider an ability for user to pass the config parameter for readFromString in tapir codecs.

[kciesielski]

Keeping the offset indeed sounds safe and may be helpful. We could use ReaderConfig.withAppendHexDumpToParseException(false) and copy the full message from JsonReaderException. This would make the error look like

Invalid value for: body (name should not be empty and only made of alphanumeric characters, offset: 0x00000016)

Right now, Tapir uses just the default ReaderConfig. Additionally, we could allow passing custom ReaderConfig if anyone needs to enable the hex dump, increase it, etc. for deeper debugging.