Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ingredients, affecting order of tests #392

Open
Bodigrim opened this issue Sep 29, 2023 · 0 comments
Open

Support ingredients, affecting order of tests #392

Bodigrim opened this issue Sep 29, 2023 · 0 comments

Comments

@Bodigrim
Copy link
Collaborator

Currently there is no simple way to write a tasty ingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:

  • Randomize order of tests to detect implicit dependencies between them.
  • Execute the longest running tests first, maximizing multithreaded CPU utilization.

I do not propose to implement such mechanisms in tasty itself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of [TestAction] in launchTestTree:

launchTestTree opts tree k0 = do
(testActions, fins) <- createTestActions opts tree
let NumThreads numTheads = lookupOption opts
(t,k1) <- timed $ do
abortTests <- runInParallel numTheads (testAction <$> testActions)
(do let smap = IntMap.fromDistinctAscList $ zip [0..] (testStatus <$> testActions)

One could extend launchTestTree with an argument (OptionSet -> [TestAction] -> [TestAction]), but that would require us to expose data TestAction, which I deem undesirable:

data TestAction act = TestAction
{ testAction :: act
-- ^ Some action, typically 'UnresolvedAction', 'ResolvedAction', or 'Action'.
, testPath :: Path
-- ^ Path pointing to this action (a series of group names + a test name)
, testDeps :: Seq Dependency
-- ^ Dependencies introduced by AWK-like patterns
, testStatus :: TVar Status
-- ^ Status var that can be used to monitor test progress
}

Instead I suggest adding an argument of type (OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extract Path from a, I'll be able to reorder [a]".

How does it sound?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant