I have reviewed usages of Encoding (most commonly Encoding.UTF8), and determined that most usages do not need to be changed. The following cases do not result in a BOM being generated:

• Encoding.UTF8.GetBytes(string)
• J2N.Text.StringExtensions.GetBytes(this string, Encoding)
• Use of Encoding.Default or Encoding.GetEncoding(0) which uses a BOM-less UTF8 encoding on modern .NET, or the system current code page on .NET Framework

The following cases ignore a BOM if present, and do not fail if there is not a BOM, and thus do not need to be changed to a BOM-less Encoding:

• Any TextReader use (such as StreamReader)
• IOUtils.GetDecodingReader(...)
• Encoding.UTF8.GetString(byte[])
• FileStream with FileAccess.Read

So you'll see in the PR that the amount of changes to address BOM issues are not very many; that's because most fall into those buckets above.

Looks like you missed OfflineSorter. The tests specifically failed when it was configured to use a BOM, although I didn't analyze it at a high level to find out why that was the case. No objections if you wish to investigate this, but it definitely makes a difference as far as the tests are concerned.

It has gone through several rounds of refactoring since then, but currently it has a DEFAULT_ENCODING field that we added to ensure the tests pass. So, we have a couple of options:

1. Remove the DEFAULT_ENCODING field and replace it with IOUtils.CHARSET_UTF_8. Update the OfflineSorter documentation for ByteSequencesReader and ByteSequencesWriter to indicate that constructor overloads that accept BinaryReader and BinaryWriter should use IOUtils.CHARSET_UTF_8.
2. Initialize the DEFAULT_ENCODING field with the same instance as IOUtils.CHARSET_UTF_8.

Given the fact that we added this field specifically because OfflineSorter requires there to be no BOM (which difers from the .NET default), this could go either way. Given that we recently changed IOUtils.CHARSET_UTF_8 to remove the BOM, using it wasn't an option when the DEFAULT_ENCODING field was added. If it were, it would have been reused in this case and the field wouldn't have been added.

Side note: perhaps we should also rename IOUtils.CHARSET_UTF_8 because it is public and "CharSet" is Java nomenclature. ENCODING_UTF8_NO_BOM would be a better name.