fork-aware-tx-pool: add heavy load tests based on zombienet #7257
Conversation
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
iulianbarbu
added
the
R0-silent
iulianbarbu
changed the title
substrate/client/transaction-pool/tests/zombienet/small_network_yap.rs
Outdated
Show resolved
Hide resolved
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
There was a problem hiding this comment.
So far looks good.
I was thinking about CLI. Maybe we don't need it, after all? Instead, we could use following:
$ cargo test --test stand_alone -- --exact run_single_collator_networkwhich would just:
- spawn the network,
- print out the location of the executed binaries (this one is important to me, to be absolutely sure that I don't run test on old binaries),
- print out the location of logs file,
- ... or maybe just print zombienet summary - with ports, params, etc...
- wait forever
Then one could use any tooling to just send transcations to this network.
We could start with this and see how it goes. We could do next iteration from here, and not over-complicate it from the beginning. In that way we would use the same config for manual testing, and for pre-defined test suits.
One more idea to control parameters would be using environment variables (not needed in the first step). Provides flexibility, less convenient to use comparing to CLI args, but much easier to implement.
export TXPOOLTESTS_POOL_LIMIT=1000
$ cargo test --test stand_alone -- --exact run_single_collator_networkStill we can have all the integration tests in different test module, reusing the same network configurations as those spawned in stand_alone mod, for example:
$ cargo test --test integration -- --exact single_collator_network__single_account_1M_txsstand_alone tests would be excluded from cargo test command (as they never terminated on their own).
Any thoughts on this?
|
|
||
| #[async_trait::async_trait] | ||
| impl Network for Limits30Network { | ||
| fn ensure_bins_on_path(&self) -> bool { |
There was a problem hiding this comment.
Hi, sorry to chime in. This is already checked by zombienet-sdk internally (for each cmd to execute and the workers).
There was a problem hiding this comment.
Is zombienet-sdk capable of printing full executable paths? (I know, I am a bit paranoid on this 😅)
There was a problem hiding this comment.
@pepoviola can you point me to where we're doing these checks in zombienet-sdk?
There was a problem hiding this comment.
So far the logs zombienet emits aren't showing the full binary paths for the node binaries it executes. @michalkucharczyk does it help if the logs contain the dump of the $PATH variable? Thinking we can make a feature request for zn-sdk logs to show that when configuring the network.
At the same time, @pepoviola , is there a way to tell a log file path to zn-sdk so that the log records emitted by zn-sdk could be tailed and easily analyzed later on, instead of showing it directly to stdout (which depending on the terminal settings, can be limited and polluted with other outputs?
|
Hi @iulianbarbu / @michalkucharczyk, I'm working on an small cli to spawn from |
My 3 cents: We actually want to spawn network programatically. So I am not sure that cli will be helpful here. But having some API in zombienet that would accept toml and spawn the network could be potentially helpful. Especially when it comes to customization - instead of playing with CLI args or enviroment variables as I proposed in my previous comment we could just edit toml file. On the other hand, it seems that using zn-sdk is not that difficult. @iulianbarbu what is your opinion? |
|
Responding to the last messages where @pepoviola chimed in:
+1 to this idea @michalkucharczyk . I personally prefer Rust and zn-sdk for the testsuite, while for manual runs, if we'd be able to import the tomls directly with zn-sdk, and have the option to also use them with a CLI, then we can have the best of both worlds. It would be just a preference for how we'd like to do the manual testing, because we can still run the testsuite locally, by changing things within the rust tests, but if we want just to run the network and then do other stuff against it, we'd have the CLI as well.
yup, thanks @pepoviola for confirming this offline. For reference: https://docs.rs/zombienet-sdk/latest/zombienet_sdk/struct.NetworkConfig.html#method.load_from_toml.
@pepoviola how different would be from the existing zombienet CLI and why do we need another one? |
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
|
We don't want to commit the chainspecs to the repo, right? The better way is to generate them using build scripts, like we do here for example: https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/zombienet-sdk-tests/build.rs |
Good point. Chain-spec can be avoided with this commit: Shall be enough to define dev-accounts in the genesis-patch (which in turn can be given in zobienet toml file). However, not sure if we need extra aka- |
|
I think we need to mention the path to the runtime as well if mentioning the patch. That's a variable path, but we can assume it is
Can't see either how a build.rs can help when using zombienet-sdk with tomls loading. |
Signed-off-by: Iulian Barbu <[email protected]>
Maybe we don't need runtime path? We could skip |
|
Oh yeah, I guess asset-hub-* is covered. Leaving out experimental runtimes sounds good. |
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
Signed-off-by: Iulian Barbu <[email protected]>
| let future_scenario_executor = ScenarioBuilder::new() | ||
| .with_rpc_uri(ws.to_string()) | ||
| .with_chain_type(ChainType::Sub) | ||
| .with_block_monitoring(block_monitor) |
There was a problem hiding this comment.
I suspect we will duplicate this across many tests, so this could be common function:
fn zn_scenario_builder(ws:&str) -> ScenarioBuilder {
let send_threshold = 20_000;
let block_monitor = false;
let watched_txs = true;
ScenarioBuilder::new()
.with_rpc_uri(ws.to_string())
.with_chain_type(ChainType::Sub)
.with_block_monitoring(block_monitor)
.with_watched_txs(watched_txs)
.with_send_threshold(send_threshold)
}
There was a problem hiding this comment.
Good idea, implemented here: 9a665b6
| @@ -58,28 +51,40 @@ async fn send_future_and_then_ready_from_many_accounts() { | |||
| // Shared params. | |||
| let send_threshold = 20_000; | |||
| let ws = "ws://127.0.0.1:9933"; | |||
There was a problem hiding this comment.
Can we get this URL from zn API? (if we update toml this can diverge).
There was a problem hiding this comment.
Yes, done here: 9a665b6
Signed-off-by: Iulian Barbu <[email protected]>
| let _ = net.wait_collator_client("charlie").await.unwrap(); | ||
| let ws = net.node_rpc_uri("charlie").unwrap(); | ||
| let future_scenario_executor = default_zn_scenario_builder() | ||
| .with_rpc_uri(ws.clone()) | ||
| .with_chain_type(ChainType::Sub) |
There was a problem hiding this comment.
should go to default too
There was a problem hiding this comment.
right 🙈 will do
| # single-state | ||
| # fork-aware |
There was a problem hiding this comment.
| # single-state | |
| # fork-aware |
| # single-state | ||
| # fork-aware |
There was a problem hiding this comment.
| # single-state | |
| # fork-aware |
| validator = true | ||
|
|
||
| [[relaychain.nodes]] | ||
| name = "bob" |
There was a problem hiding this comment.
Maybe we should expose rpc port for bob too?
I think we can also enabled fatxpool on the relaychain. So we would need to copy some args to the relaychain nodes.
Some scenarios sending to the relay-chain would nice. As discussed in element they can also be quicker for development (no need to wait for 10 blocks).
| # single-state | ||
| # fork-aware |
There was a problem hiding this comment.
| # single-state | |
| # fork-aware |
| array-bytes = { workspace = true, default-features = true } | ||
| assert_matches = { workspace = true } | ||
| async-trait = { workspace = true } | ||
| chrono = { workspace = true } |
There was a problem hiding this comment.
From what I can see, some of other crates seems to be not used, too.
| pub const ASSET_HUB_LOW_POOL_LIMIT_FATP_SPEC_PATH: &'static str = | ||
| "tests/zombienet/network-specs/asset-hub-low-pool-limit-fatp.toml"; | ||
| pub const ASSET_HUB_HIGH_POOL_LIMIT_FATP_SPEC_PATH: &'static str = | ||
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-fatp.toml"; | ||
| pub const ASSET_HUB_HIGH_POOL_LIMIT_OLDP_3_COLLATORS_SPEC_PATH: &'static str = | ||
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-3-collators.toml"; | ||
| pub const ASSET_HUB_HIGH_POOL_LIMIT_OLDP_4_COLLATORS_SPEC_PATH: &'static str = | ||
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-4-collators.toml"; |
There was a problem hiding this comment.
nit: I would try to shorten the toml paths names using mod, e.g.:
| pub const ASSET_HUB_LOW_POOL_LIMIT_FATP_SPEC_PATH: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-low-pool-limit-fatp.toml"; | |
| pub const ASSET_HUB_HIGH_POOL_LIMIT_FATP_SPEC_PATH: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-fatp.toml"; | |
| pub const ASSET_HUB_HIGH_POOL_LIMIT_OLDP_3_COLLATORS_SPEC_PATH: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-3-collators.toml"; | |
| pub const ASSET_HUB_HIGH_POOL_LIMIT_OLDP_4_COLLATORS_SPEC_PATH: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-4-collators.toml"; | |
| pub mod asset_hub_based_spec_paths { | |
| pub const LOW_POOL_LIMIT_FATP: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-low-pool-limit-fatp.toml"; | |
| pub const HIGH_POOL_LIMIT_FATP: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-fatp.toml"; | |
| pub const HIGH_POOL_LIMIT_OLDP_3_COLLATORS: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-3-collators.toml"; | |
| pub const HIGH_POOL_LIMIT_OLDP_4_COLLATORS: &'static str = | |
| "tests/zombienet/network-specs/asset-hub-high-pool-limit-oldp-4-collators.toml"; | |
| } |
Shall be nicer to read when used in tests.
There was a problem hiding this comment.
I would also use sstp (single-state-transaction-pool) instead of oldp.
| } | ||
|
|
||
| /// Returns a node client and waits for blocks productio to kick-off. | ||
| pub async fn wait_collator_client( |
There was a problem hiding this comment.
collator is redundant here, we can wait for any node.
I would change the name to indicate that we are waiting for node to start producing blocks, maybe:
wait_for_block_production ?
| .subscribe_best() | ||
| .await | ||
| .map_err(|_| Error::FailedToGetBlocksStream)?; | ||
| loop { |
There was a problem hiding this comment.
we shall wait for block with number 1, otherwise we'll exit the loop on genesis hash.
| let (future_logs, ready_logs) = futures::future::join( | ||
| future_scenario_executor.execute(), | ||
| ready_scenario_executor.execute(), |
There was a problem hiding this comment.
One more thing which would very useful is adding a prefix for debug logs allowing to distinguish instances of ttxt.
I think that tracing::span could be used for this, but I don't have enough experience with them to judge if it is doable.
Description
Builds up towards addressing #5497 by creating some zombienet-sdk code infra that can be used to spin regular networks, as described in the fork aware transaction pool testing setup added here #7100. It will be used for developing tests against such networks, and to also spawn them on demand locally through tooling that will be developed in follow ups.
Integration
Node/runtime developers can run tests based on the zombienet-sdk infra that spins frequently used networks which can be used for analyzing behavior of various node related components, like fork aware transaction pool.
Review Notes