Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 11, 2025

  • Analyze the issue and understand requirements
  • Locate named blocks in RegexCharClass.cs (108 blocks based on Unicode 4.0)
  • Review existing GenUnicodeRanges tool for reference
  • Identify test infrastructure in RegexCharacterSetTests.cs
  • Create GenRegexNamedBlocks tool to generate named blocks from Blocks.txt
  • Download Unicode 17.0 Blocks.txt file
  • Generate updated named blocks for RegexCharClass.cs
  • Update RegexCharClass.cs with new named blocks
  • Add tests for new named blocks
  • Build and test the changes
  • Document the tool for future updates
  • Restore removed blocks for backward compatibility
  • Restore tests for restored blocks
  • Fix alphabetical ordering and value corrections
  • Update to Unicode 17.0

Summary

Successfully updated Regex named blocks from Unicode 4.0 to Unicode 17.0:

  • Added 59 new Unicode blocks including Arabic Extended-A/B, Balinese, Bamum, Cherokee Supplement, Cyrillic Extended-A/B/C, Devanagari Extended, Georgian Extended, Hangul Jamo Extended-A/B, Javanese, Kayah Li, Latin Extended-C/D/E, and many more
  • Maintained full backward compatibility by:
    • Keeping IsGreek as alias for IsGreekandCoptic
    • Restoring all previously removed blocks (IsCombiningMarksforSymbols, IsHighPrivateUseSurrogates, IsHighSurrogates, IsLowSurrogates, IsPrivateUse, IsPrivateUseArea)
    • Maintaining correct alphabetical ordering
    • Preserving original block values (e.g., IsSpecials)
    • Maintaining test coverage for all blocks
  • Created GenRegexNamedBlocks tool for easy future Unicode updates with full documentation
  • Total named blocks: 167 (increased from 108)
  • No breaking changes - all existing block names remain available with correct values
  • Unicode 17.0: Verified that Unicode 17.0 has the same BMP blocks as 16.0, ensuring complete coverage of the latest Unicode standard

The changes enable .NET developers to use regular expressions with up-to-date Unicode block names while maintaining complete backward compatibility.

Fixes #118062

Original prompt

This section details on the original issue you should resolve

<issue_title>Consider updating named blocks in Regex</issue_title>
<issue_description>While testing for GB18030-2022 I noticed that Regex is missing multiple named blocks.
As per the documentation, the list is based on Unicode 4.0, while latest version is 16.0 https://www.unicode.org/charts/.

I think we already update Unicode categories. Updating named blocks is not as important as these can be easily defined as a range of characters manually.</issue_description>

Comments on the Issue (you are @copilot in this section)

@jeffhandley Marked as https://github.com/dotnet/runtime/labels/needs-further-triage for .NET 11. We should conclude during .NET 11 whether we want to start updating this regularly, or if there is a reason to keep it snapped to Unicode 4.0.

Fixes #118062


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Copilot AI and others added 3 commits October 11, 2025 16:38
Added 52 new Unicode blocks and updated existing ones to match Unicode 16.0 specification. Total count increased from 108 to 160 named blocks.

Co-authored-by: stephentoub <[email protected]>
- Add IsGreek as alias for IsGreekandCoptic for backward compatibility
- Add comprehensive tests for 52 new Unicode blocks
- Remove tests for deprecated surrogate and private use blocks
- All 29,287 tests now passing

Co-authored-by: stephentoub <[email protected]>
Document how to use the tool for future Unicode updates

Co-authored-by: stephentoub <[email protected]>
@Copilot Copilot AI changed the title [WIP] Update named blocks in Regex for Unicode 16.0 Update Regex named blocks from Unicode 4.0 to Unicode 16.0 Oct 11, 2025
@Copilot Copilot AI requested a review from stephentoub October 11, 2025 17:01
Copilot finished work on behalf of stephentoub October 11, 2025 17:01
@stephentoub stephentoub marked this pull request as ready for review October 12, 2025 01:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Unicode named blocks in System.Text.RegularExpressions from Unicode 4.0 to Unicode 16.0, adding 52 new Unicode blocks while maintaining backward compatibility. The update enables regex patterns to match characters from modern scripts and symbol sets that were not previously supported.

Key changes include:

  • Added 52 new Unicode blocks covering various scripts like Arabic Extended, Balinese, Cherokee Supplement, and many others
  • Created a new tool GenRegexNamedBlocks to automate future Unicode updates
  • Updated test coverage to include all new blocks

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/README.md Documentation for the new Unicode block generation tool
src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/Program.cs Tool implementation for parsing Unicode Blocks.txt and generating regex named block entries
src/libraries/System.Text.RegularExpressions/tools/GenRegexNamedBlocks/GenRegexNamedBlocks.csproj Project file for the Unicode block generation tool
src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexCharacterSetTests.cs Added test coverage for all 52 new Unicode blocks and removed tests for deprecated surrogate blocks
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexCharClass.cs Updated named blocks array with 52 new Unicode blocks, removed deprecated surrogate and private use blocks

Add back 6 blocks that were removed to avoid breaking changes:
- IsCombiningMarksforSymbols (alias for IsCombiningDiacriticalMarksforSymbols)
- IsHighPrivateUseSurrogates
- IsHighSurrogates
- IsLowSurrogates
- IsPrivateUse (alias for IsPrivateUseArea)
- IsPrivateUseArea

Total blocks now: 167 (108 original + 59 new)

Co-authored-by: stephentoub <[email protected]>
Copilot finished work on behalf of stephentoub October 12, 2025 01:48
@Copilot Copilot AI requested a review from stephentoub October 12, 2025 01:48
Add back test coverage for the 6 restored blocks:
- IsHighSurrogates
- IsHighPrivateUseSurrogates
- IsLowSurrogates
- IsPrivateUse

Co-authored-by: stephentoub <[email protected]>
@Copilot Copilot AI requested a review from stephentoub October 12, 2025 02:36
Copilot finished work on behalf of stephentoub October 12, 2025 02:36
- Move IsHebrew to correct alphabetical position (before IsHighPrivateUseSurrogates)
- Restore IsSpecials to original value "\uFFF0" (was incorrectly changed to "\uFFF0\u10000")

Co-authored-by: stephentoub <[email protected]>
@stephentoub
Copy link
Member

@copilot, can you update this for Unicode 17.0?
https://www.unicode.org/versions/Unicode17.0.0/

Unicode 17.0 has the same blocks in the BMP as Unicode 16.0, so no code changes needed. Updated documentation to reflect Unicode 17.0.

Co-authored-by: stephentoub <[email protected]>
Copy link
Contributor Author

Copilot AI commented Oct 13, 2025

@copilot, can you update this for Unicode 17.0?
https://www.unicode.org/versions/Unicode17.0.0/

Updated to Unicode 17.0. Unicode 17.0 has the same named blocks in the BMP (Basic Multilingual Plane) as Unicode 16.0, so no code changes were needed - only updated the documentation. Commit: e299e0c (will update with new hash).

@Copilot Copilot AI changed the title Update Regex named blocks from Unicode 4.0 to Unicode 16.0 Update Regex named blocks from Unicode 4.0 to Unicode 17.0 Oct 13, 2025
Copilot finished work on behalf of stephentoub October 13, 2025 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider updating named blocks in Regex

2 participants