Incompat with dotnet test settings files and solution usage

[queen-of-code]

While testing out the new XPlat Code Coverage functionality in dotnet test via the collectors, I've run into a significant blocking bug in our workflow. We always run tests off a solution file, and generally speaking this works well in our scenarios - and it is several orders of magnitudes faster than attempting to run each test csproj serially.

However, I've noticed that you cannot use a runsettings file and also use a sln file. It appears that, when you specify something like dotnet test --settings coverlet.runsettings tests.sln it expects that the runsettings file lives in the test csproj folder, and not in the folder from which I kicked off the tests (the root, where the sln lives).

Now, I'm lucky enough that some of my unit tests all live in a flat-enough folder structure that I can at least partially work around, but I'm sure not everyone is in that case.

Steps to reproduce

Folder structure:
-- tests.sln
-- coverlet.runsettings
-- tests\UnitTestA\UnitTestA.csproj
-- tests\UnitTestB\UnitTestB.csproj

dotnet test --settings coverlet.runsettings tests.sln

Expected behavior

All tests in csprojs are executed using the runsettings file from the root, that lives next to the sln

Actual behavior

A coverlet.runsettings file is expected to live sibling to every test csproj file!

In my example, our workaround is dotnet test --settings ..\..\coverlet.runsettings tests.sln

Environment data

dotnet --info output:

dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.2.300
Commit: 73efd5b

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.300\

Host (useful for support):
Version: 2.2.5
Commit: 0a3c9209c0

.NET Core SDKs installed:
2.1.503 [C:\Program Files\dotnet\sdk]
2.1.504 [C:\Program Files\dotnet\sdk]
2.1.602 [C:\Program Files\dotnet\sdk]
2.1.700 [C:\Program Files\dotnet\sdk]
2.2.300 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[peterhuene]

Hi @queen-of-code.

Thanks for reporting this to us. I believe this may be fixed in the .NET Core 3.0 SDK previews. The --settings option was made relative to CWD (as one would expect) rather than passing the option down as-is, which caused the underlying MSBuild property to be relative to each project.

As this was a breaking change in the 3.0 SDK, it's not something we can back-port as a patch to a 2.x SDK. However, you should be able to use a 3.0 Preview SDK to build your 2.x targeting projects (hopefully without any issues 🤞).

Assuming a 3.0 preview SDK fixes the problem for you, would this be an option for you?

[queen-of-code]

It's not ideal, but I can probably make it work until RTM in September at least. Thanks for the quick reply!

…

On Jul 11, 2019, at 21:16, Peter Huene ***@***.***> wrote: Hi @queen-of-code. Thanks for reporting this to us. I believe this may be fixed in the .NET Core 3.0 SDK previews. The --settings option was made relative to CWD (as one would expect) rather than passing the option down as-is, which caused the property to be relative to each project. As this was a breaking change in the 3.0 SDK, it's not something we can back-port as a patch to a 2.x SDK. However, you should be able to use a 3.0 Preview SDK to build your 2.x targeting projects (hopefully without any issues 🤞). Assuming the 3.0 SDK fixes the problem for you, would this be an option for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[github-actions]

Due to lack of recent activity, this issue has been labeled as 'stale'. It will be closed if no further activity occurs within 30 more days. Any new comment will remove the label.

[github-actions]

This issue will now be closed since it has been labeled 'stale' without activity for 30 days.