Fix more Unicode bugs

[fmeum]

• Use Latin-1 in many native file write rules for consistency with the internal encoding.
• Use Latin-1 for the resolved repository file and the JSON profile.
• Fix unused_input_list handling of non-ASCII characters in file names.
• Flip the legacy_utf8 parameter of repository_ctx.file to False and make it a no-op. With the previous default, any non-ASCII characters would be written out as double encoded UTF-8, which is not a useful choice.
• Change repository_ctx.template to operate on raw bytes for consistency with repository_ctx.read and to fix substitution with non-ASCII keys/values.
• Move some usages of UTF_8 closer to their usage site to clarify why they are correct.
• Fixes parsing of dependency files with Unicode character contents (/showIncludes and .d files)

[fmeum]

@tjgq This is stacked on #24010, but the last commit is new and ready for review.

[tjgq]

@fmeum Mind rebasing before I review?

[fmeum]

@tjgq Thanks for the review, it helped me find and fix more issues.

I noticed that there are a bunch of FileSystemUtils methods that read and write files in different encodings. I could clean that up in a follow-up PR or as part of this PR, as you prefer.

[tjgq]

@tjgq Thanks for the review, it helped me find and fix more issues.

I noticed that there are a bunch of FileSystemUtils methods that read and write files in different encodings. I could clean that up in a follow-up PR or as part of this PR, as you prefer.

I'd prefer a separate PR, thanks!

[tjgq]

@fmeum it looks like there's an issue with .netrc parsing? (see failed tests)

[fmeum]

@fmeum it looks like there's an issue with .netrc parsing? (see failed tests)

Should be fixed now

[fmeum]

@meteorcloudy Compiling a cc_library that depends on a header file with non-ASCII characters in its name results in this kind of error:

ERROR: C:/b/absmfft5/execroot/_main/_tmp/925c5432c7427d85ec55d631e57c838b/workspace/pkg/BUILD:5:10: Compiling pkg/bin.cc failed: undeclared inclusion(s) in rule '//pkg:bin':
this rule is missing dependency declarations for the following files included by 'pkg/bin.cc':
  'pkg/„”�Ž™šá??.h'

The raw bytes of the file name are 112, 107, 103, 47, -124, -108, -127, -114, -103, -102, -31, 63, 63, 46, 104, which indicates that the output of MSVC is in the OEM codepage 437. This is hinted at here. Do you know of a way to force cl.exe to output Unicode when it's not running in a console (or, alternatively, to have it think it's running in a console)? I guess we could wrap cl.exe in a .bat script that runs chcp 65001 prior to invoking the compiler, but that seems pretty intrusive.

[meteorcloudy]

Maybe this one https://learn.microsoft.com/en-gb/cpp/build/reference/utf-8-set-source-and-executable-character-sets-to-utf-8?view=msvc-170?

[fmeum]

Maybe this one https://learn.microsoft.com/en-gb/cpp/build/reference/utf-8-set-source-and-executable-character-sets-to-utf-8?view=msvc-170?

That didn't help :-(

[fmeum]

I have given up on getting cl.exe to work. Users could presumably still do this themselves by switching their system code page to UTF-8. CI should be green now that I am skipping the test on Windows.

[fmeum]

@tjgq Rebased onto master.

[fmeum]

@bazel-io fork 8.0.0