More BytesRef/UnicodeUtil UTF-8-to-string fixes, #1024

[paulirwin]

• You've read the Contributor Guide and Code of Conduct.
• You've included unit or integration tests for your change, where applicable.
• You've included inline docs for your change, where applicable.
• There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Some fixes to BytesRef/UnicodeUtil UTF-8 ToString behavior.

Related #1024

Description

• Change UnicodeUtil's UTF8toUTF16 to throw a DecoderFallbackException instead of FormatException.
• Make BytesRef's DebuggerDisplay use Utf8ToStringWithFallback

[NightOwl888]

What I originally meant by "fallback" was the same logic we use for the formatter:

lucenenet/src/Lucene.Net/Util/BytesRef.cs

Lines 607 to 614 in 9638258

	if (bytesRef.TryUtf8ToString(out var utf8String))
	{
	return utf8String;
	}
	else
	{
	return bytesRef.ToString();
	}

That is, when there is invalid UTF-8 in the BytesRef, it would fall back to ToString(), which shows the raw bytes. This would give the developer a clear indication that the bytes were invalid. It also seems like displaying the bytes before the string is reversed. The string is the important bit, so it should go first.

IMO, whatever the solution, there should be a very clear indication that the bytes are invalid. Showing the fallback characters looks a bit strange, but it is hard to tell when developing whether they are actually coming from the code you are debugging, or the debugger itself.

So I propose:

1. Show the string at the beginning and replace it with the text "Invalid UTF-8" if is invalid.
2. Show the bytes afterward.
3. Ideally, show each on separate lines (which requires writing a small custom visualizer).

A custom visualizer for this might look like:

using System;
using System.Diagnostics;
using System.Text;
using Microsoft.VisualStudio.DebuggerVisualizers;

public class MyClassVisualizer : DialogDebuggerVisualizer
{
    protected override void Show(IDialogVisualizerService windowService, IVisualizerObjectProvider objectProvider)
    {
        // Get the object being visualized
        BytesRef bytesRef = objectProvider.GetObject() as BytesRef;
        if (bytesRef is null)
        {
            return;
        }

        // Prepare the display content (multi-line)
        StringBuilder sb = new StringBuilder();
        sb.Append("Content = ");
        if (bytesRef.TryUtf8ToString(out var utf8String))
            sb.AppendLine(utf8String);
        else
            sb.AppendLine("Invalid UTF-8");
        sb.Append("Bytes = ");
        sb.AppendLine(bytesRef .ToString());

        // Show the content in a dialog window
        windowService.ShowDialog(sb.ToString());
    }
}