char16_t support

[AndreyAlifanov]

Hello!

I try to use char16_t for json with boost::iostreams. I need it for compatibility with Windows.

using JSONImpl = jsoncons::basic_json<char16_t, jsoncons::order_preserving_policy, std::allocator<char>>;

And get a vast number of errors with jsoncons methods.
For example (it's a very small part of full log):

[build] In file included from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_reader.hpp:20,
[build]                  from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/basic_json.hpp:36,
[build]                  from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json.hpp:10,
[build]                  from /home/user/Dev/AP/Mod/PiLib/Pi/PiJsonImpl.cpp:10:
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp: In instantiation of ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::end_fraction_value(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’:
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:575:21:   required from ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::parse_some_(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:531:9:   required from ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::parse_some(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_reader.hpp:300:35:   required from ‘void jsoncons::basic_json_reader<CharT, Source, TempAllocator>::read_next(std::error_code&) [with CharT = char16_t; Source = jsoncons::stream_source<char16_t>; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_reader.hpp:273:22:   required from ‘void jsoncons::basic_json_reader<CharT, Source, TempAllocator>::read_next() [with CharT = char16_t; Source = jsoncons::stream_source<char16_t>; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/basic_json.hpp:2530:29:   required from ‘static jsoncons::basic_json<CharT, Policy, Allocator> jsoncons::basic_json<CharT, Policy, Allocator>::parse(std::basic_istream<charT>&, const jsoncons::basic_json_decode_options<CharT>&) [with CharT = char16_t; Policy = jsoncons::order_preserving_policy; Allocator = std::allocator<char>]’
[build] /home/user/Dev/AP/Mod/PiLib/Pi/PiJsonImpl.cpp:205:37:   required from ‘bool CPiJsonElementImpl::DoRead(T&) [with T = std::__cxx11::basic_istringstream<char16_t>]’
[build] /home/user/Dev/AP/Mod/PiLib/Pi/PiJsonImpl.cpp:238:15:   required from here
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:2659:38: error: no match for call to ‘(jsoncons::detail::chars_to) (const char16_t*, std::__cxx11::basic_string<char16_t>::size_type)’
[build]  2659 |                 double d = to_double_(string_buffer_.c_str(), string_buffer_.length());
[build]       |                            ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[build] In file included from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/detail/write_number.hpp:20,
[build]                  from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_encoder.hpp:25,
[build]                  from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/pretty_print.hpp:19,
[build]                  from /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/basic_json.hpp:29:
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/detail/parse_number.hpp:1019:5: note: candidate: ‘template<class CharT> typename std::enable_if<std::is_same<XChar, char>::value, double>::type jsoncons::detail::chars_to::operator()(const CharT*, std::size_t) const’
[build]  1019 |     operator()(const CharT* s, std::size_t /*length*/) const
[build]       |     ^~~~~~~~
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/detail/parse_number.hpp:1019:5: note:   template argument deduction/substitution failed:
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/detail/parse_number.hpp: In substitution of ‘template<class CharT> typename std::enable_if<std::is_same<XChar, char>::value, double>::type jsoncons::detail::chars_to::operator()(const CharT*, std::size_t) const [with CharT = char16_t]’:
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:2659:38:   required from ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::end_fraction_value(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:575:21:   required from ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::parse_some_(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_parser.hpp:531:9:   required from ‘void jsoncons::basic_json_parser<CharT, TempAllocator>::parse_some(jsoncons::basic_json_visitor<CharT>&, std::error_code&) [with CharT = char16_t; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_reader.hpp:300:35:   required from ‘void jsoncons::basic_json_reader<CharT, Source, TempAllocator>::read_next(std::error_code&) [with CharT = char16_t; Source = jsoncons::stream_source<char16_t>; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/json_reader.hpp:273:22:   required from ‘void jsoncons::basic_json_reader<CharT, Source, TempAllocator>::read_next() [with CharT = char16_t; Source = jsoncons::stream_source<char16_t>; TempAllocator = std::allocator<char>]’
[build] /home/user/Dev/AP/build/vcpkg_installed/x64-linux/include/jsoncons/basic_json.hpp:2530:29:   required from ‘static jsoncons::basic_json<CharT, Policy, Allocator> jsoncons::basic_json<CharT, Policy, Allocator>::parse(std::basic_istream<charT>&, const jsoncons::basic_json_decode_options<CharT>&) [with CharT = char16_t; Policy = jsoncons::order_preserving_policy; Allocator = std::allocator<char>]’
[build] /home/user/Dev/AP/Mod/PiLib/Pi/PiJsonImpl.cpp:205:37:   required from ‘bool CPiJsonElementImpl::DoRead(T&) [with T = std::__cxx11::basic_istringstream<char16_t>]’

How to use jsoncons with char16_t in Linux?
I have gcc-12.2 with C++17 support.

[danielaparker]

We currently only support char and wchar_t. While most of the code on char type is generic, there is still some char type specific code in traits classes and number conversion functions.

I had originally thought to extend support for u16json etc., but there seems to be exceptionally little interest in it, over the 12 years this project has been around, this is the first request we've had for char16_t. And apart from yourself, almost nobody seems to be using the wchar_t specializions. Our most popular extension, json_schema, doesn't even support wide characters, and we haven't received a single request to add that. No recent competing project appears to be supporting wide characters at all, you have to go back to rapid json for that. Almost everybody seems to have settled on char and UTF-8 encoding.

There are a couple of projects that support detecting the encoding of a JSON source, and transparently converting it into UTF-8 if necessary before reading it into dom's and sax style handlers. We could do that. We could also provide alternative encoders that would write it out as UTF-16, if desired. Would that solve your problem?

[AndreyAlifanov]

Thank you for detailed information.
I think, we'll redesign our code in near future and will use UTF-8.
So, support of wchar_t in jsoncons will become redundant.