Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set NumThreads to 1 in cardano-testnet-test #5758

Merged
merged 1 commit into from
Apr 11, 2024
Merged

Set NumThreads to 1 in cardano-testnet-test #5758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions cardano-testnet/cardano-testnet.cabal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ library
, scientific
, si-timers
, stm
, tasty
, tasty ^>= 1.5
, tasty-expected-failure
, tasty-hedgehog
, text
Expand Down Expand Up @@ -152,7 +152,7 @@ test-suite cardano-testnet-golden
, hedgehog-extras
, process
, regex-compat
, tasty
, tasty ^>= 1.5
, tasty-hedgehog
, text

Expand Down Expand Up @@ -220,7 +220,7 @@ test-suite cardano-testnet-test
, mtl
, process
, regex-compat
, tasty
, tasty ^>= 1.5
, text
, time
, transformers
Expand Down
55 changes: 30 additions & 25 deletions cardano-testnet/test/cardano-testnet-test/cardano-testnet-test.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,47 +24,50 @@ import qualified Cardano.Testnet.Test.SubmitApi.Babbage.Transaction
import Prelude

import qualified System.Environment as E
import qualified System.Exit as IO
import qualified System.IO as IO
import System.IO (BufferMode (LineBuffering), hSetBuffering, hSetEncoding, stdout, utf8)

import qualified Testnet.Property.Run as H

import qualified Test.Tasty as T
import Test.Tasty (TestTree)
import qualified Test.Tasty.Ingredients as T
import qualified Test.Tasty.Options as T
import qualified Test.Tasty.Runners as T

tests :: IO TestTree
tests = do
testGroup <- runTestGroup <$> shouldRunInParallel
pure $ testGroup "test/Spec.hs"
[ testGroup "Spec"
[ testGroup "Ledger Events"
pure $ T.testGroup "test/Spec.hs"
[ T.testGroup "Spec"
[ T.testGroup "Ledger Events"
[ H.ignoreOnWindows "Sanity Check" LedgerEvents.hprop_ledger_events_sanity_check
, H.ignoreOnWindows "Treasury Growth" LedgerEvents.prop_check_if_treasury_is_growing
-- TODO: Replace foldBlocks with checkLedgerStateCondition
, testGroup "Governance"
, T.testGroup "Governance"
[ H.ignoreOnMacAndWindows "ProposeAndRatifyNewConstitution" Cardano.Testnet.Test.LedgerEvents.Gov.ProposeNewConstitution.hprop_ledger_events_propose_new_constitution
-- FIXME Those tests are flaky
-- , H.ignoreOnWindows "InfoAction" LedgerEvents.hprop_ledger_events_info_action
, H.ignoreOnWindows "ProposeNewConstitutionSPO" LedgerEvents.hprop_ledger_events_propose_new_constitution_spo
, H.ignoreOnWindows "DRepRetirement" DRepRetirement.hprop_drep_retirement
]
, testGroup "Plutus"
, T.testGroup "Plutus"
[ H.ignoreOnWindows "PlutusV3" Cardano.Testnet.Test.Cli.Conway.Plutus.hprop_plutus_v3]
]
, testGroup "CLI"
, T.testGroup "CLI"
[ H.ignoreOnWindows "Shutdown" Cardano.Testnet.Test.Node.Shutdown.hprop_shutdown
-- ShutdownOnSigint fails on Mac with
-- "Log file: /private/tmp/tmp.JqcjW7sLKS/kes-period-info-2-test-30c2d0d8eb042a37/logs/test-spo.stdout.log had no logs indicating the relevant node has minted blocks."
, H.ignoreOnMacAndWindows "ShutdownOnSigint" Cardano.Testnet.Test.Node.Shutdown.hprop_shutdownOnSigint
-- ShutdownOnSlotSynced FAILS Still. The node times out and it seems the "shutdown-on-slot-synced" flag does nothing
-- , H.ignoreOnWindows "ShutdownOnSlotSynced" Cardano.Testnet.Test.Node.Shutdown.hprop_shutdownOnSlotSynced
, testGroup "Babbage"
, T.testGroup "Babbage"
[ H.ignoreOnMacAndWindows "leadership-schedule" Cardano.Testnet.Test.Cli.Babbage.LeadershipSchedule.hprop_leadershipSchedule -- FAILS
, H.ignoreOnWindows "stake-snapshot" Cardano.Testnet.Test.Cli.Babbage.StakeSnapshot.hprop_stakeSnapshot
, H.ignoreOnWindows "transaction" Cardano.Testnet.Test.Cli.Babbage.Transaction.hprop_transaction
]
-- TODO: Conway - Re-enable when create-staked is working in conway again
--, testGroup "Conway"
--, T.testGroup "Conway"
-- [ H.ignoreOnWindows "stake-snapshot" Cardano.Testnet.Test.Cli.Conway.StakeSnapshot.hprop_stakeSnapshot
-- ]
-- Ignored on Windows due to <stdout>: commitBuffer: invalid argument (invalid character)
Expand All @@ -75,27 +78,27 @@ tests = do
, H.ignoreOnWindows "CliQueries" Cardano.Testnet.Test.Cli.Queries.hprop_cli_queries
]
]
, testGroup "SubmitApi"
[ testGroup "Babbage"
, T.testGroup "SubmitApi"
[ T.testGroup "Babbage"
[ H.ignoreOnWindows "transaction" Cardano.Testnet.Test.SubmitApi.Babbage.Transaction.hprop_transaction
]
]
]

shouldRunInParallel :: IO Bool
shouldRunInParallel = (== Just "1") <$> E.lookupEnv "PARALLEL_TESTNETS"
defaultMainWithIngredientsAndOptions :: [T.Ingredient] -> T.OptionSet -> T.TestTree -> IO ()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just copied defaultMainWithIngredients with ability to change OptionSet or is there something else added here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is correct.

defaultMainWithIngredientsAndOptions ins opts testTree = do
T.installSignalHandlers
parsedOpts <- T.parseOptions ins testTree
let opts' = opts <> parsedOpts

-- FIXME Right now when running tests concurrently it makes them flaky
runTestGroup
:: Bool -- ^ True to run in parallel
-> T.TestName
-> [TestTree]
-> TestTree
runTestGroup True name = T.testGroup name
runTestGroup False name = T.sequentialTestGroup name T.AllFinish

ingredients :: [T.Ingredient]
ingredients = T.defaultIngredients
case T.tryIngredients ins opts' testTree of
Nothing -> do
IO.hPutStrLn IO.stderr
"No ingredients agreed to run. Something is wrong either with your ingredient set or the options."
IO.exitFailure
Just act -> do
ok <- act
if ok then IO.exitSuccess else IO.exitFailure

main :: IO ()
main = do
Expand All @@ -105,4 +108,6 @@ main = do
hSetEncoding stdout utf8
args <- E.getArgs

E.withArgs args $ tests >>= T.defaultMainWithIngredients ingredients
let opts = T.singleOption $ T.NumThreads 1
Copy link
Contributor

@carbolymer carbolymer Apr 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have an environment variable to override this? It's still useful to me to be able to run tests in parallel locally.

Copy link
Contributor Author

@newhoggy newhoggy Apr 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to do this:

--test-options '--num-threads 8'

Because all I've done is change the default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default is:

  defaultValue = NumThreads numCapabilities

Whatever that works out to be.


E.withArgs args $ tests >>= defaultMainWithIngredientsAndOptions T.defaultIngredients opts
Loading