[ 6.0.100-preview.6.21321.10] Attempted to read past the end of the stream error in QuickInfo app

[Junjun-zhao]

Application Name: QuickInfo
OS: Windows 10 RS5
CPU: X64
.NET Build Number: 6.0.100-preview.6.21321.10

Verify Scenarios:

1. Windows 10 RS5 X64 + dotnet-sdk-3.1.411 PASS
2. Windows 10 RS5 X64 + 6.0.100-preview.6.21315.23 PASS
3. Windows 10 RS5 X64 + 6.0.100-preview.6.21321.10: Fail

Github Link : https://github.com/KirillOsenkov/QuickInfo
Source Code & App check at : https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1345781

Minimal Repro steps:

This is 3.1 app, but we are running it on 6.0, machine only have 6.0 installed, and DOTNET_ROLL_FORWARD_ON_NO_CANDIDATE_FX=2, so it runs on 6.0

1)Copy App to machine
2) dotnet QuickInfoWeb.dll
3) Navigate to https://localhost:5001/
4) In the textbox, type color

Expected Result:
It should display colors

Actual Result:

System.TypeInitializationException: The type initializer for 'QuickInfo.Controllers.AnswersController' threw an exception. ---> 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> 
System.TypeInitializationException: The type initializer for 'System.Unicode.UnicodeInfo' threw an exception. ---> 
System.IO.EndOfStreamException: Attempted to read past the end of the stream. at 
System.Unicode.UnicodeData.ReadUnicodeCharacterDataEntry(BinaryReader reader, Byte[] nameBuffer, UnicodeCharacterData& 
value) in UnicodeInformation.dll:token 0x6000083+0x73 at System.Unicode.UnicodeData.ReadFromStream(Stream stream) in 
UnicodeInformation.dll:token 0x6000082+0xbd at System.Unicode.UnicodeData.ReadFromResources() in 
UnicodeInformation.dll:token 0x6000081+0x26 at System.Unicode.UnicodeInfo..cctor() in UnicodeInformation.dll:token 
0x600009e+0x38 --- End of inner exception stack trace --- at System.Unicode.UnicodeInfo.GetBlocks() in  
UnicodeInformation.dll:token 0x600009d+0x0 at QuickInfo.Unicode.BuildUnicodeList() in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Processors\Unicode.cs:line 183 at QuickInfo.Unicode..ctor() in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Processors\Unicode.cs:line 21 at 
System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions) in 
System.Private.CoreLib.dll:token 0x6000687+0x43 --- End of inner exception stack trace --- at 
System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions) in 
System.Private.CoreLib.dll:token 0x6000687+0x66 at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean 
wrapExceptions) in System.Private.CoreLib.dll:token 0x60009a9+0x2d at System.Activator.CreateInstance(Type type) in 
System.Private.CoreLib.dll:token 0x60009a0+0x0 at QuickInfo.Engine.<>c.<.ctor>b__2_1(Type t) in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Engine.cs:line 28 at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() in System.Linq.dll:token 0x6000244+0x3a at 
System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection) in System.Private.CoreLib.dll:token 
0x6006883+0xea at QuickInfo.Engine..ctor(Assembly[] assemblies) in C:\Users\appcompat\QuickInfo\src\QuickInfo\Engine.cs:line 28 at QuickInfo.Controllers.AnswersController..cctor() in C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 40 --- End of inner exception stack trace --- at QuickInfo.Controllers.AnswersController.get_Instance() in C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 40 at QuickInfo.Controllers.AnswersController.Get(String query) in 
C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 20

Findings :

We think error is related to System.IO.Compression.dll, when we replace it with working version, app works fine.

For minimal app repro :

Error is coming from nuget this package :
Library source code : GitHub - GoldenCrystal/NetUnicodeInfo: Unicode Character Inspector & Library providing a subset of the Unicode data for .NET clients.

This code will work on .net5 but throws exception on .net6

1. create console app (net5 or net6), but we need to run it on net6
2. Add this package : <PackageReference Include="UnicodeInformation" Version="2.5.0" />
3. Add this code in program.cs :

                var test = UnicodeInfo.GetBlocks();
                foreach (var item in test)
                {
                        Console.WriteLine(item.Name);
                 }

Exception :

Unhandled exception. System.TypeInitializationException: The type initializer for 'System.Unicode.UnicodeInfo' threw an exception.
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at System.Unicode.UnicodeData.ReadUnicodeCharacterDataEntry(BinaryReader reader, Byte[] nameBuffer, UnicodeCharacterData& value) in UnicodeInformation.dll:token 0x6000083+0x73
   at System.Unicode.UnicodeData.ReadFromStream(Stream stream) in UnicodeInformation.dll:token 0x6000082+0xbd
   at System.Unicode.UnicodeData.ReadFromResources() in UnicodeInformation.dll:token 0x6000081+0x26
   at System.Unicode.UnicodeInfo..cctor() in UnicodeInformation.dll:token 0x600009e+0x38
   --- End of inner exception stack trace ---
   at System.Unicode.UnicodeInfo.GetBlocks() in UnicodeInformation.dll:token 0x600009d+0x0
   at TestApp6.Program.Main(String[] args) in C:\Users\appcompat\source\repos\TestApp6\TestApp6\Program.cs:line 10

@dotnet-actwx-bot @dotnet/compat

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Application Name: QuickInfo
OS: Windows 10 RS5
CPU: X64
.NET Build Number: 6.0.100-preview.6.21321.10

Verify Scenarios:

1. Windows 10 RS5 X64 + dotnet-sdk-3.1.411 PASS
2. Windows 10 RS5 X64 + 6.0.100-preview.6.21315.23 PASS
3. Windows 10 RS5 X64 + 6.0.100-preview.6.21321.10: Fail

Github Link : https://github.com/KirillOsenkov/QuickInfo
Source Code & App check at : https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1345781

Minimal Repro steps:

This is 3.1 app, but we are running it on 6.0, machine only have 6.0 installed, and DOTNET_ROLL_FORWARD_ON_NO_CANDIDATE_FX=2, so it runs on 6.0

1)Copy App to machine
2) dotnet QuickInfoWeb.dll
3) Navigate to https://localhost:5001/
4) In the textbox, type color

Expected Result:
It should display colors

Actual Result:

System.TypeInitializationException: The type initializer for 'QuickInfo.Controllers.AnswersController' threw an exception. ---> 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> 
System.TypeInitializationException: The type initializer for 'System.Unicode.UnicodeInfo' threw an exception. ---> 
System.IO.EndOfStreamException: Attempted to read past the end of the stream. at 
System.Unicode.UnicodeData.ReadUnicodeCharacterDataEntry(BinaryReader reader, Byte[] nameBuffer, UnicodeCharacterData& 
value) in UnicodeInformation.dll:token 0x6000083+0x73 at System.Unicode.UnicodeData.ReadFromStream(Stream stream) in 
UnicodeInformation.dll:token 0x6000082+0xbd at System.Unicode.UnicodeData.ReadFromResources() in 
UnicodeInformation.dll:token 0x6000081+0x26 at System.Unicode.UnicodeInfo..cctor() in UnicodeInformation.dll:token 
0x600009e+0x38 --- End of inner exception stack trace --- at System.Unicode.UnicodeInfo.GetBlocks() in  
UnicodeInformation.dll:token 0x600009d+0x0 at QuickInfo.Unicode.BuildUnicodeList() in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Processors\Unicode.cs:line 183 at QuickInfo.Unicode..ctor() in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Processors\Unicode.cs:line 21 at 
System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions) in 
System.Private.CoreLib.dll:token 0x6000687+0x43 --- End of inner exception stack trace --- at 
System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions) in 
System.Private.CoreLib.dll:token 0x6000687+0x66 at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean 
wrapExceptions) in System.Private.CoreLib.dll:token 0x60009a9+0x2d at System.Activator.CreateInstance(Type type) in 
System.Private.CoreLib.dll:token 0x60009a0+0x0 at QuickInfo.Engine.<>c.<.ctor>b__2_1(Type t) in 
C:\Users\appcompat\QuickInfo\src\QuickInfo\Engine.cs:line 28 at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() in System.Linq.dll:token 0x6000244+0x3a at 
System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection) in System.Private.CoreLib.dll:token 
0x6006883+0xea at QuickInfo.Engine..ctor(Assembly[] assemblies) in C:\Users\appcompat\QuickInfo\src\QuickInfo\Engine.cs:line 28 at QuickInfo.Controllers.AnswersController..cctor() in C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 40 --- End of inner exception stack trace --- at QuickInfo.Controllers.AnswersController.get_Instance() in C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 40 at QuickInfo.Controllers.AnswersController.Get(String query) in 
C:\Users\appcompat\QuickInfo\src\QuickInfoWeb\Controllers\AnswersController.cs:line 20

Findings :

We think error is related to System.IO.Compression.dll, when we replace it with working version, app works fine.

For minimal app repro :

Error is coming from nuget this package :
Library source code : GitHub - GoldenCrystal/NetUnicodeInfo: Unicode Character Inspector & Library providing a subset of the Unicode data for .NET clients.

This code will work on .net5 but throws exception on .net6

1. create console app (net5 or net6), but we need to run it on net6
2. Add this package :
3. Add this code in program.cs :

                var test = UnicodeInfo.GetBlocks();
                foreach (var item in test)
                {
                        Console.WriteLine(item.Name);
                 }

Exception :

Unhandled exception. System.TypeInitializationException: The type initializer for 'System.Unicode.UnicodeInfo' threw an exception.
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at System.Unicode.UnicodeData.ReadUnicodeCharacterDataEntry(BinaryReader reader, Byte[] nameBuffer, UnicodeCharacterData& value) in UnicodeInformation.dll:token 0x6000083+0x73
   at System.Unicode.UnicodeData.ReadFromStream(Stream stream) in UnicodeInformation.dll:token 0x6000082+0xbd
   at System.Unicode.UnicodeData.ReadFromResources() in UnicodeInformation.dll:token 0x6000081+0x26
   at System.Unicode.UnicodeInfo..cctor() in UnicodeInformation.dll:token 0x600009e+0x38
   --- End of inner exception stack trace ---
   at System.Unicode.UnicodeInfo.GetBlocks() in UnicodeInformation.dll:token 0x600009d+0x0
   at TestApp6.Program.Main(String[] args) in C:\Users\appcompat\source\repos\TestApp6\TestApp6\Program.cs:line 10

@dotnet-actwx-bot @dotnet/compat

Author:	Junjun-zhao
Assignees:	jeffhandley
Labels:	area-System.IO.Compression
Milestone:	-

[Junjun-zhao]

@jeffhandley @adamsitnik @carlossanlop @jozkee @stephentoub We have 2 apps failed with this issue. Could you please help look into this issue and help assign if it is incorrect ? Thanks a lot~

[adamsitnik]

That was dotnet/docs#24649 introduced by #53644

I've sent a fix to the library: hexawyz/NetUnicodeInfo#7

Since this was a breaking change introduced on purpose and for good reasons, can we close the issue?

cc @stephentoub

[Junjun-zhao]

@adamsitnik Thanks a lot for checking this issue and quick fix for the library.

We had another app failed and is also related to System.IO.Compression.dll. Could you please help check whether they are same root cause? Thanks.

App: Countdown (.NET 5.0 App)
GitHub Links: https://github.com/DHancock/Countdown

Test scenarios:
Windows 10 RS5 X64 + 5.0.300 PASS
Windows 10 RS5 X64 + 6.0.100-preview.6.21321.10: Fail

Findings:

It returns more data when read resource file with .net 5, we think there is incomplete read.

https://github.com/DHancock/Countdown/blob/11b6baa23a3cd0cf9e193c656fef9a8baf703be7/Countdown/Models/WordDictionary.cs#L145

private void LoadResourceFile()
        {
            Stream resourceStream = typeof(App).Assembly.GetManifestResourceStream(cResourceName);

            if (resourceStream != null)
            {
                using DeflateStream stream = new DeflateStream(resourceStream, CompressionMode.Decompress);

                StreamManager sm = new StreamManager(stream);
                byte[] line;

                while ((line = sm.ReadLine()) != null)
                {
                    int keyLength = line.Length;

                    // check for a word break within the line
                    if (keyLength > (cMinLetters * 2))
                    {
                        for (keyLength = cMinLetters; keyLength < line.Length; ++keyLength)
                        {
                            if (line[keyLength] == cWord_seperator)
                                break;
                        }
                    }

                    // make key
                    char[] c = GetChars(line, keyLength);
                    Array.Sort(c);
                    string key = new string(c);

                    // add to dictionary
                    if ((keyLength == cMaxLetters) && (keyLength == line.Length))
                        conundrumWords[key] = line;
                    else
                        otherWords[key] = line;
                }
            }
        }

[stephentoub]

whether they are same root cause?

It is:
https://github.com/DHancock/Countdown/blob/3e86bcc8897e3a5f921cb0cc66421cbf5814a65a/Countdown/Models/WordDictionary.cs#L274-L275

[adamsitnik]

@stephentoub @Junjun-zhao how do we deal with this issue from here? Are we going to send a PR with a patch to https://github.com/DHancock/Countdown?

[adamsitnik]

A new version of the NuGet package has been shipped to nuget.org: https://www.nuget.org/packages/UnicodeInformation/