Skip to content

Latest commit

 

History

History
283 lines (202 loc) · 14.8 KB

KnownIssues.md

File metadata and controls

283 lines (202 loc) · 14.8 KB
Raw

Known Issues

VSTest stops process execution early

Affected drivers: msbuild (dotnet test) , dotnet tool(if you're using --targetargs "test ... --no-build")

Symptoms:

  • warning or error like:

    Unable to read beyond end of stream

    warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp'

  • zero coverage result (often only on CI but not on local)

    Calculating coverage result...
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp.Tests_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp.Tests' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj]
  Generating report 'C:\Users\REDACTED\Documents\repo\testapp\lcov.info'

+---------------+------+--------+--------+
| Module        | Line | Branch | Method |
+---------------+------+--------+--------+
| testApp       | 0%   | 0%     | 0%     |
+---------------+------+--------+--------+
| testApp.Tests | 0%   | 100%   | 0%     |
+---------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

The issue is related to VSTest platform microsoft/vstest#1900 (comment)

However if testhost doesn't shut down within 100ms (as the execution is completed, we expect it to shutdown fast). vstest.console forcefully kills the process.

Coverlet collects and writes hits data on process exit, if for some reason the process is too slow to close, it will be killed and we cannot collect coverage result. This happen also if there are other "piece of code" during testing that slow down process exit. We found problem for instance with tests that use RabbitMQ.

Solution:

The only way to get around this issue is to use collectors integration https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application. With the collector, we're injected in test host through a in-proc collector that communicates with the VSTest platform so we can signal when we end our work.

Upgrade coverlet.collector to version > 1.0.0

Affected drivers: VSTest integration dotnet test --collect:"XPlat Code Coverage"

Symptoms: The same as "known issue 1".

There is a bug inside the VSTest platform: microsoft/vstest#2205.

If you upgrade the collector package with a version greater than 1.0.0, in-proc collector won't be loaded so you could incur "known issue number 1" and get zero coverage result

Solutions:

  1. Reference Microsoft.NET.Test.Sdk with version greater than 17.7.0

    <ItemGroup>
...
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
...
</ItemGroup>

  2. You can pass custom coverage.runsettings file like this

    <?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
    <DataCollectors>
        <DataCollector friendlyName="XPlat code coverage">
            <Configuration>
                <Format>cobertura</Format>
            </Configuration>
        </DataCollector>
    </DataCollectors>
</DataCollectionRunSettings>
<InProcDataCollectionRunSettings>
    <InProcDataCollectors>
        <InProcDataCollector assemblyQualifiedName="Coverlet.Collector.DataCollection.CoverletInProcDataCollector, coverlet.collector, Version=1.1.0.0, Culture=neutral, PublicKeyToken=null"
                        friendlyName="XPlat Code Coverage"
                        enabled="True"
                        codebase="coverlet.collector.dll" />
    </InProcDataCollectors>
</InProcDataCollectionRunSettings>
</RunSettings>

And pass it to command line

dotnet test --settings coverage.runsettings

Nerdbank.GitVersioning and /p:UseSourceLink=true option

Affected drivers: all drivers that support /p:UseSourceLink=true

Symptoms: some tool like SonarSource doesn't work well see coverlet-coverage#482

Nerdbank.GitVersioning generates a version file on the fly but this file is not part of user solution and it's not committed to repo so the generated remote source file reference does not exit, i.e.

...
<File uid="1" fullPath="https://raw.githubusercontent.com/iron9light/HOCON.Json/654d4ea8ec524f72027e2b2d324aad9acf80b710/src/Hocon.Json/obj/Release/netstandard2.0/Hocon.Json.Version.cs" />
...

Solution: we can exclude Nerdbank.GitVersioning autogenerated file from instrumentation using filters

# example for coverlet.msbuild
/p:ExcludeByFile=\"**/*Json.Version.cs\"
# example for coverlet.collector
dotnet test test-assembly.dll /collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.ExcludeByFile="**/*.Version.cs"

Failed to resolve assembly during instrumentation

Affected drivers: all drivers

Symptoms: during build/instrumentation you may get an exception like:

[coverlet] Unable to instrument module: ..\UnitTests\bin\Debug\net6.0\Core.Messaging.dll because : Failed to resolve assembly: 'Microsoft.Azure.ServiceBus, Version=7.16.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c' [..\UnitTests.csproj]

In the instrumentation phase, Coverlet needs to load all references used by your instrumented module. Sometimes the build phase (out of Coverlet's control) does not copy those dlls to the output folder because they are not resolved till runtime or at publish phase from the NuGet packages folders.

Solution: we need to tell to MSBuild to copy all NuGet dll reference to output using MSBuild switch CopyLocalLockFileAssemblies

dotnet test /p:CollectCoverage=true /p:CopyLocalLockFileAssemblies=true

or by adding the property <CopyLocalLockFileAssemblies> to the project file

<PropertyGroup>
...
  <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
...
</PropertyGroup>

NB. This DOESN'T ALWAYS WORK, for example in case of the shared framework https://github.com/dotnet/cli/issues/12705#issuecomment-536686785

We can do nothing at the moment as this is a build behaviour out of our control.

For .NET runtime version >= 3.0 the new default behavior is to copy all assets to the build output (CopyLocalLockFileAssemblies=true) https://github.com/dotnet/cli/issues/12705#issuecomment-535150372, unfortunately the issue could still arise.

In this case the only workaround at the moment is to manually copy missing dlls to the output folder: coverlet-coverage#560 (comment)

The only reliable way to work around this problem is to drop the DLL in the unit tests project's bin\Release\netcoreapp2.2 directory.

Tests fail if assembly is strong named

Affected drivers: all drivers

Symptoms: Running coverage on .NET Framework runtime(i.e. .NET 4.6.1) and get error like:

Failed   Tests.MinMax.Min_AsyncSelector_Int32_4
Error Message:
 System.TypeInitializationException : The type initializer for 'Tests.AsyncEnumerableTests' threw an exception.
---- System.IO.FileLoadException : Could not load file or assembly 'System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' or one of its dependencies. Strong name signature could not be verified.  The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045)
Stack Trace:
  at Tests.AsyncEnumerableTests..ctor()
  at Tests.MinMax..ctor()
----- Inner Stack Trace -----
  at Tests.AsyncEnumerableTests..cctor()

Solution: Looks like this is caused by xUnit's app domains. For dotnet test, it can be disabled with the following argument: -- RunConfiguration.DisableAppDomain=true

Code coverage returns NaN

Symptoms: You are getting following result when running Coverlet within CI/CD pipeline:

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

SUT (System Under Test) assembly is also not listed in MSBuild logs - "Instrumented module" is missing for your dll.

Solution: Check whether deterministic build is turned on for your solution, if so, follow the instructions on how to handle deterministic builds.

Failure to produce Code Coverage Report

Symptoms:

C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error : Stream was too long. [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Xml.XmlStreamNodeWriter.FlushBuffer() [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Xml.XmlStreamNodeWriter.GetBuffer(Int32 count, Int32& offset) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Xml.XmlStreamNodeWriter.UnsafeWriteUTF8Chars(Char* chars, Int32 charCount) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Xml.XmlUTF8NodeWriter.WriteEscapedText(String s) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Xml.XmlBaseWriter.WriteString(String value) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteString(XmlWriterDelegator xmlWriter, String value, XmlDictionaryString name, XmlDictionaryString ns) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(39,5): error :    at WriteLineToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract ) [REDACTED.csproj]

....

Calculating coverage result...
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(71,5): error : Unexpected end of file. [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(71,5): error :    at System.Xml.EncodingStreamWrapper.ReadBOMEncoding(Boolean notOutOfBand) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(71,5): error :    at System.Xml.EncodingStreamWrapper..ctor(Stream stream, Encoding encoding) [REDACTED.csproj]
C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\6.0.0\build\coverlet.msbuild.targets(71,5): error :    at System.Xml.XmlUTF8TextReader.SetInput(Stream stream, Encoding encoding, XmlDictionaryReaderQuotas quotas, OnXmlDictionaryReaderClose onClose) [REDACTED.csproj]

The XML code coverage report is too large for the coverlet to parse.

Potential Solutions:

  • Reduce noise from auto generated code, for example excluding your EntityFrameworkCore Migrations namespace or automatically generated typed Http Clients. See Excluding From Coverage for more information on ignoring namespaces from code coverage.

BadImageFormatException .NET Framework 4.7.x, 4.8.x

Symptoms:

BadImageFormatException during MetadataReaderProvider.FromPortablePdbStream in InstrumentationHelper.PortablePdbHasLocalSource, unable to check if the module has got local source.

Solutions:

Change DebugType from full to portable.

Important

NET Core introduces a new symbol file (PDB) format - portable PDBs. Unlike traditional PDBs which are Windows-only, portable PDBs can be created and read on all platforms.

CoverletCoverageDataCollector: Failed to instrument modules

Affected drivers: VSTest integration dotnet test --collect:"XPlat Code Coverage"

Symptoms:

Data collector 'XPlat code coverage' message: [coverlet]Coverlet.Collector.Utilities.CoverletDataCollectorException: CoverletCoverageDataCollector: Failed to instrument modules
 ---> System.AggregateException: One or more errors occurred. (The process cannot access the file XXX\ABC.pdb
 ---> System.IO.IOException: The process cannot access the file 'XXX\ABC.pdb' because it is being used by another process.
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at Coverlet.Core.Helpers.FileSystem.Copy(String sourceFileName, String destFileName, Boolean overwrite) in /_/src/coverlet.core/Helpers/FileSystem.cs:line 35
   at Coverlet.Core.Helpers.InstrumentationHelper.<>c__DisplayClass16_0.<RestoreOriginalModule>b__1() in /_/src/coverlet.core/Helpers/InstrumentationHelper.cs:line 277
   at Coverlet.Core.Helpers.RetryHelper.<>c__DisplayClass0_0.<Retry>b__0() in /_/src/coverlet.core/Helpers/RetryHelper.cs:line 28
   at Coverlet.Core.Helpers.RetryHelper.Do[T](Func`1 action, Func`1 backoffStrategy, Int32 maxAttemptCount) in /_/src/coverlet.core/Helpers/RetryHelper.cs:line 55
   --- End of inner exception stack trace ---
   ...

Note

This is not an coverlet issue but running tests in parallel without proper separation of test case resources

dotnet vstest cli option

--Parallel

Run tests in parallel. By default, all available cores on the machine are available for use. Specify an explicit number of cores by setting the MaxCpuCount property under the RunConfiguration node in the runsettings file.

Solutions:

use VSTest setting -maxcpucount:1 which limits "worker processes".