Feature Request: Run Tests in Parallel with Shared Context

[zijianhuang]

Background
Currently by default tests in the same assembly but in different classes run in parallel.

To run tests in different classes not in parallel, you put them into the same test collection.

To have multiple classes sharing the same contexts, you use a Collection Fixture, however, then the tests run one by one.

I have some integration tests that need to launch IIS or DotNet Host, or drop/create databases, via collection fixtures.

Feature Request

Allow to run tests in different classes in parallel sharing the same context like dropping/creating databases.

Programming interface expected:

public sealed class CollectionAttribute : Attribute
	{
		public CollectionAttribute(string name, boolean parallel=false);
	} 



[CollectionDefinition("Database collection")]
public class DatabaseCollection : ICollectionFixture<DatabaseFixture>
{
    // This class has no code, and is never created. Its purpose is simply
    // to be the place to apply [CollectionDefinition] and all the
    // ICollectionFixture<> interfaces.
}

[Collection("Database collection", true)]
public class DatabaseTestClass1
{
 //...
}

[Collection("Database collection", true)]
public class DatabaseTestClass2
{
    // ...
}

[redoz]

I'll chime in and say it would be very nice if v3 offered more control over how tests in collections were executed (e.g. in parallel) or offered up a different means of having shared contexts and controlling execution. I understand this isn't really a concern when unit-testing, but we're using xbehave as our integration testing framework (mostly because it lets us re-use existing infrastructure that supports xunit, runners, reporting, etc.) and I've ended up hacking on my own "AssemblyFixture" support so we can handle shared context across our test-suite. But even so I'm forced to run some tests in a non-parallel collection simply to make sure it doesn't interfere with the rest of the test-suite which is a bit inefficient as those tests run serialized now.

[fjmorel]

edit: I moved this comment to the IAssemblyFixture request, which fits what I need better, I think.

[markm77]

I made a comment on the other parallelisation issue: #1986 (comment).

Yes, it would be really great to have the ability to run tests in parallel when using a collection fixture. We use Xunit for both unit and integration tests, and for the integration tests often instance a webhost via a collection fixture. In addition, we also often use a single [Theory] test with multiple [MemberData] cases (i.e. dynamic test case generation). I'm really hoping the implementation of this allows [MemberData] cases for a single [Theory] test to run in parallel!

[pdevito3]

This will be a huge win for my projects either way, but in the proposed API, why default these to not be parallelized? I'd rather have it be opt out than opt in and would think that would be a better default?

[bradwilson]

@pdevito3 Because that's how it works in v2, and by changing the default, it could potentially cause a bunch of tests to fail without knowing why when a user upgrades from v2 to v3. Additionally, I consider this to be an advanced feature, so enabling it makes sense to me.

[pdevito3]

Makes sense. As long as I can turn it on I'll be a happy camper 🙂

[pdevito3]

👋 @bradwilson just saw that v3's out but this didn't make it in :-( any thoughts on it's potential future?

[bradwilson]

This is still something that I would like to get into v3, it just didn't make the bar for 1.0.

I don't have a particular design or time frame in mind at this moment.