Bridges - revert-back and improve congestion #6231

bkontur · 2024-10-25T08:51:14Z

Closes: #5551
Closes: #5550

Context

Before permissionless lanes, bridges only supported hard-coded, static lanes. The congestion mechanism was based on sending Transact(report_bridge_status(is_congested)) from pallet-xcm-bridge-hub to pallet-xcm-bridge-hub-router. Depending on is_congested, we adjusted the fee factor to increase or decrease fees. This congestion mechanism relied on monitoring XCMP queues, which could cause issues like suspending the entire XCMP queue rather than just the affected bridge.

Additionally, we are progressing with deploying bridge message pallets/routing directly on AssetHub, where we don’t interact with XCMP to perform ExportXcm locally.

Description

This PR re-introduces and improves congestion for bridges:

Enhanced Bridge Congestion Mechanism: The bridge queue mechanism has been restructured to operate independently of XCMP, with a refined protocol for congestion detection and suspension management.
Bridge-Specific Channel Suspension: pallet-xcm-bridge-hub and pallet-xcm-bridge-hub-router now use BridgeId to identify specific bridges, enabling selective suspension and resumption of individual bridge channels.
Dynamic Congestion Detection: pallet-xcm-bridge-hub now includes callbacks for fn suspend_bridge and fn resume_bridge based on congestion status:
- For sibling chains, the router sends xcm::Transact(report_bridge_status(bridge_id, is_congested)) using the stored callback information.
- For local chain deployments, the router manages state directly.
New Stop Threshold: A stop_threshold limit in pallet-xcm-bridge-hub enables or disables ExportXcm::validate, providing a fallback mechanism when the router does not adhere to the suspend signal.
Flexible Message Routing: pallet-xcm-bridge-hub-router has been refactored to support message routing for both sibling chains (ExportMessage) and local deployment (ExportXcm).

These updates improve modularity, allow more granular bridge congestion handling, and support diverse deployment scenarios.

bkontur · 2024-10-25T11:25:35Z

bot fmt
/cmd prdoc --audience runtime_dev --bump patch

prdoc/pr_6231.prdoc

bkontur · 2024-10-26T20:30:05Z

bot fmt

bkontur · 2024-10-28T15:04:20Z

bot fmt

bkontur · 2024-11-05T12:40:39Z

bot fmt

bkontur · 2024-11-07T15:57:44Z

/cmd bench --runtime asset-hub-westend asset-hub-rococo --pallet pallet_xcm_bridge_hub_router

bkontur · 2024-11-07T16:18:32Z

bot bench cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

bkontur · 2024-11-07T16:51:47Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages

bkontur · 2024-11-07T21:51:26Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-rococo --pallet=pallet_xcm_benchmarks::generic
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-westend --pallet=pallet_xcm_benchmarks::generic

bkontur · 2024-11-08T23:55:04Z

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_bridge_messages
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-rococo --pallet=pallet_xcm_bridge_hub
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --runtime=bridge-hub-westend --pallet=pallet_xcm_bridge_hub

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-rococo --pallet=pallet_xcm_benchmarks::generic
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-bridge-hubs --subcommand=xcm --runtime=bridge-hub-westend --pallet=pallet_xcm_benchmarks::generic

bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-westend --pallet=pallet_xcm_bridge_hub_router
bot bench -v PIPELINE_SCRIPTS_REF=bko-fix cumulus-assets --runtime=asset-hub-rococo --pallet=pallet_xcm_bridge_hub_router

serban300

Did just a first pass

serban300 · 2024-11-18T07:28:58Z

bridges/modules/xcm-bridge-hub-router/src/lib.rs

+						{
+							break;
+						}
+						bridges_to_update.push((bridge_id, previous_factor, bridge_state));


Why not process it on the spot ?

Why not process it on the spot ?

Well, at least for bridges_to_remove I can't/shouldn't do that according to the documentation:

/// Enumerate all elements in the map in no particular order.
///
/// If you alter the map while doing this, you'll get undefined results.

I don't know, maybe inserting the same key with different value while iter would work (I didn't try), but I assume that it is also "alter the map", so I better used the same pattern for bridges_to_update.

Oh, yes, you're right. How about translate() ?

hmm, it says that translate() iterates all elements (removes for None), so then we would not need this weight metering, which @franciscoaguirre reported here: #6231 (comment).

I think I've seen translate used only for migrations, I don't know :)
@serban300 so what do you suggest? if I want to also trigger events, should I do it inside translate function?

I think I've seen translate used only for migrations, I don't know :)

I don't know either. I never used translate. It just seemed to permit editing items on the spot.

@serban300 so what do you suggest? if I want to also trigger events, should I do it inside translate function?

I guess so. I don't know. Is there a reason not to do it ?

The current code is good, AFAICT with translate() you can't control amount of weight used, which is needed on_idle()

bridges/modules/xcm-bridge-hub-router/src/lib.rs

bridges/modules/xcm-bridge-hub-router/src/impls.rs

bridges/modules/xcm-bridge-hub-router/src/lib.rs

serban300 · 2024-11-18T14:04:59Z

bridges/modules/xcm-bridge-hub/src/lib.rs

 		pub fn open_bridge(
 			origin: OriginFor<T>,
 			bridge_destination_universal_location: Box<VersionedInteriorLocation>,
+			maybe_notify: Option<Receiver>,


maybe_notify doesn't seem very suggestive. Maybe something like congestion_notif_receiver would be better.

Well, actually, I re-used maybe_notify name/pattern from the pallet_xcm's QueryStatus maybe_notify: Option<(u8, u8)>, :), which does exactly the same, Receiver is basically the same as (u8, u8).

serban300

As far as I understand, the mechanism definitely works. It solves the congestion problem. But I don't like that it adds complexity and in some aspects we have to duplicate the XCMP congestion ideas.

Personally, I liked more the idea of adding XCMP logical channels and rely on the XCMP congestion logic. Not sure if it's still applicable or what happened to it.

bkontur · 2024-11-20T11:37:41Z

As far as I understand, the mechanism definitely works. It solves the congestion problem. But I don't like that it adds complexity and in some aspects we have to duplicate the XCMP congestion ideas.

Well, before the permissionless lanes PR, we used this exact mechanism with report/update_bridge_status. However, it was hard-coded and specifically adjusted to support the AH<>BH lane. Additionally, we expected using HrmpXcmpSignal::Suspend/Resume here. There are concerns from SA that when the bridge queue is congested and we suspend HRMP, we inadvertently disable all other non-bridging scenarios between the sibling parachain and the parachain where the bridge messages pallets are deployed. Yes, the solution would indeed be HRMP logical channels, as you mentioned:

Personally, I liked more the idea of adding XCMP logical channels and rely on the XCMP congestion logic. Not sure if it's still applicable or what happened to it.

Yes, we discussed HRMP/XCMP logical channels, but I think this is not part of the near, short or mid-term plan. Similarly, we discussed HRMP/XCMP protocol credits, but implementing that would require reworking the HRMP/XCMP queue system, which I would also say is not on the near, short or mid-term plan.

Handling bridge congestion over XCMP is only half the story. The other important aspect is that we also want (and need) to manage bridge congestion beyond XCMP. We are moving towards deploying permissionless lanes directly on AssetHub (with just the messaging pallets). This approach would mean the following:

Other sibling parachains can use AssetHub as an XCM message exporter. In this case, we need to handle congestion over HRMP/XCMP using the update_bridge_status extrinsic with maybe_notify.
Additionally, we will have an AHP<>AHK lane deployed directly on AssetHub. When functionality like moving assets over the bridge is triggered, we won't touch any HRMP/XCMP since the messaging pallet will be directly on AssetHub. In this case, we also need to address bridge congestion.

This PR essentially:

Reverts report/update_bridge_status and extends it for use with permissionless lanes.
Adds support for handling congestion in both scenarios mentioned above: as a message exporter for sibling/relay chains and as a message exporter for the local chain.

bkontur · 2024-11-21T11:52:12Z

/cmd fmt

github-actions · 2024-11-21T11:54:13Z

Command "fmt" has started 🚀 See logs here

github-actions · 2024-11-21T11:54:37Z

Command "fmt" has finished ✅ See logs here

…(check Xcmp, check UMP, ..)

paritytech-workflow-stopper · 2024-11-27T12:15:20Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12049990133
Failed job name: fmt

bkontur · 2024-11-27T12:28:35Z

/cmd fmt

github-actions · 2024-11-27T12:30:20Z

Command "fmt" has started 🚀 See logs here

github-actions · 2024-11-27T12:30:45Z

Command "fmt" has finished ✅ See logs here

acatangiu

All of the XCM transport (pallet-messages layer) will be deployed to Asset Hub instead of Bridge Hub, so would it be easier to not go through all of the effort of upgrading Bridge Hub with all this only to deprecate it on the next iteration?

What if instead of changing and renaming xcm-bridge-hub to support permissionless lanes, we just create a new pallet xcm-bridge which is what you basically have in this PR, and we revert xcm-bridge-hub to the same code actually deployed today on Bridge Hub?

That way we keep legacy code with legacy lane on BH without any migrations or risk, and all of this new code goes straight to AH, then at some point we switch the AH exporter from legacy to new?

I know this is late and maybe we should've had this idea earlier, so I'm leaving it up to you to decide which way is easiest to do.

acatangiu · 2024-11-29T13:46:30Z

bridges/modules/relayers/src/extension/mod.rs

@@ -1127,7 +1126,6 @@ mod tests {
 		Option<PreDispatchData<ThisChainAccountId, BridgedChainBlockNumber, TestLaneIdType>>,
 		TransactionValidityError,
 	> {
-		sp_tracing::try_init_simple();


nit: why remove this? it's useful in debugging, no?

acatangiu · 2024-11-29T13:56:09Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+		// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
+		// the congestion and dynamic_fees features (if possible).


Suggested change

// Here, we have the actual result fees covering bridge fees, so now we need to check/apply

// the congestion and dynamic_fees features (if possible).

// `fees` is populated with base bridge fees, now let's apply congestion/dynamic fees if required.

acatangiu · 2024-11-29T14:01:25Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+/// Adapter implementation for [`SendXcm`] that allows adding a message size fee and/or dynamic fees
+/// based on the `BridgeId` resolved by the `T::BridgeIdResolver` resolver, if and only if `E`
+/// supports routing. This adapter can be used, for example, as a wrapper over
+/// [`xcm_builder::UnpaidLocalExporter`], enabling it to compute message and/or dynamic fees using a
+/// fee factor.


Naming of these is confusing. When/how is UnpaidLocalExporter used when there are bridge fees to be paid?

Maybe rename UnpaidLocalExporter to just LocalExporter or smth?.

Or maybe this one can completely integrate UnpaidLocalExporter instead of being a wrapper over it?

acatangiu · 2024-11-29T14:14:37Z

bridges/modules/xcm-bridge-hub-router/src/impls.rs

+				bridged_dest.clone()
+			}
+		} else {
+			// if `bridged_dest` does not contain `GlobalConsensus`, let's prepend one


is there a case when ensure_is_remote(UniversalLocation::get(), dest.clone()) returns Ok, but dest does not contain GlobalConsensus?
should we even support such a case?

acatangiu · 2024-11-29T14:31:46Z

bridges/modules/xcm-bridge-hub-router/src/lib.rs

+//!
+//! ### On the Same Chain as the Message Exporter
+//! In this setup, the router directly calls an `ExportXcm` implementation. In this case,
+//! `ViaLocalBridgeHubExporter` can be used as a wrapper with `T::ToBridgeHubSender`.


We need to rename T::ToBridgeHubSender since it's not fit for the local case, where Bridge Hub is not involved.

acatangiu · 2024-11-29T15:41:42Z

bridges/primitives/xcm-bridge-hub/src/lib.rs

+	/// `bool` - `true` means that we keep accepting messages to the bridge.
+	Suspended(bool),


I would add another enum variant instead of overcharging Suspended:

Suggested change

/// `bool` - `true` means that we keep accepting messages to the bridge.

Suspended(bool),

/// We keep accepting messages to the bridge to allow any inflight messages to be processed.

SoftSuspended,

/// Bridge is suspended and new messages are now being actively dropped.

HardSuspended,

acatangiu · 2024-11-29T15:42:49Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

@@ -151,6 +144,20 @@ where
 			message,
 		)?;

+		// Here, we know that the message is relevant to this pallet instance, so let's check for
+		// congestion defense.
+		if bridge.state == BridgeState::Suspended(false) {


This would be much more readable

Suggested change

if bridge.state == BridgeState::Suspended(false) {

if bridge.state == BridgeState::HardSuspended {

acatangiu · 2024-11-29T15:44:32Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

+			BridgeState::Suspended(true) => {
+				if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {
+					// If its suspended and reached `outbound_lane_stop_threshold`, we stop
+					// accepting new messages (a.k.a. start dropping).
+					Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {
+						bridge.state = BridgeState::Suspended(false);
+					});
+					return


Suggested change

BridgeState::Suspended(true) => {

if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {

// If its suspended and reached `outbound_lane_stop_threshold`, we stop

// accepting new messages (a.k.a. start dropping).

Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {

bridge.state = BridgeState::Suspended(false);

});

return

BridgeState::SoftSuspended => {

if enqueued_messages > T::CongestionLimits::get().outbound_lane_stop_threshold {

// If its suspended and reached `outbound_lane_stop_threshold`, we stop

// accepting new messages (a.k.a. start dropping).

Bridges::<T, I>::mutate_extant(bridge_id, |bridge| {

bridge.state = BridgeState::HardSuspended;

});

return

acatangiu · 2024-11-29T15:46:25Z

bridges/modules/xcm-bridge-hub/src/exporter.rs

+				// We cannot accept new messages to the suspended bridge, hoping that it'll be
+				// actually resumed soon


Suggested change

// We cannot accept new messages to the suspended bridge, hoping that it'll be

// actually resumed soon

// We cannot accept new messages and start dropping messages, until the outbound lane goes below the drop limit.

acatangiu · 2024-11-29T16:02:34Z

prdoc/pr_6231.prdoc

@@ -0,0 +1,45 @@
+title: Bridges - revert-back and improve congestion


Suggested change

title: Bridges - revert-back and improve congestion

title: Bridges - Add improved congestion control mechanism

bkontur added the T15-bridges This PR/Issue is related to bridges. label Oct 25, 2024

bkontur self-assigned this Oct 25, 2024

bkontur force-pushed the bko-bridges-congestion branch 2 times, most recently from 659be89 to b48b8a5 Compare October 25, 2024 21:14

bkontur commented Oct 26, 2024

View reviewed changes

prdoc/pr_6231.prdoc Outdated Show resolved Hide resolved

bkontur force-pushed the bko-bridges-congestion branch from a663bc2 to cbc6ae7 Compare October 26, 2024 20:29

bkontur force-pushed the bko-bridges-congestion branch from c78f9bc to 501a5c0 Compare October 28, 2024 15:01

bkontur force-pushed the bko-bridges-congestion branch 7 times, most recently from c78e707 to 152389a Compare November 5, 2024 12:33

bkontur force-pushed the bko-bridges-congestion branch 3 times, most recently from edd9c5c to 38f1bb3 Compare November 7, 2024 13:33

bkontur force-pushed the bko-bridges-congestion branch from f06433a to d329dec Compare November 7, 2024 16:44

bkontur force-pushed the bko-bridges-congestion branch from 21baa7f to c00ff6b Compare November 8, 2024 17:44

bkontur added the A4-needs-backport Pull request must be backported to all maintained releases. label Nov 11, 2024

Nits

ad0931c

serban300 reviewed Nov 18, 2024

View reviewed changes

bkontur added 4 commits November 19, 2024 22:50

PR review

416e2e0

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

9f66e49

PR review - removed brackets

15ef0a4

PR review

296476e

paritytech-review-bot bot requested a review from a team November 19, 2024 22:05

bkontur requested a review from serban300 November 19, 2024 22:13

bkontur added 2 commits November 20, 2024 09:21

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

d69c22b

PR review - renamed report_bridge_status to update_bridge_status

02285ac

bkontur mentioned this pull request Nov 20, 2024

Update to SDK stable2409-1 polkadot-fellows/runtimes#490

Open

12 tasks

serban300 reviewed Nov 20, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

f18e230

paritytech-review-bot bot requested a review from a team November 20, 2024 15:34

Update from bkontur running command 'fmt'

a5eb1c3

bkontur added 5 commits November 21, 2024 13:14

Merge branch 'master' into bko-bridges-congestion

4c1c315

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

85dbd00

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

9ad289b

Tuple implementation for ChannelStatusProvider to cover more cases …

c93d722

…(check Xcmp, check UMP, ..)

Merge remote-tracking branch 'origin/master' into bko-bridges-congestion

8ccb773

Update from bkontur running command 'fmt'

981e3df

Merge branch 'master' into bko-bridges-congestion

91567e0

acatangiu reviewed Nov 29, 2024

View reviewed changes

		// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
		// the congestion and dynamic_fees features (if possible).

	// Here, we have the actual result fees covering bridge fees, so now we need to check/apply
	// the congestion and dynamic_fees features (if possible).
	// `fees` is populated with base bridge fees, now let's apply congestion/dynamic fees if required.

		/// `bool` - `true` means that we keep accepting messages to the bridge.
		Suspended(bool),

-	/// `bool` - `true` means that we keep accepting messages to the bridge.
-	Suspended(bool),
+	/// We keep accepting messages to the bridge to allow any inflight messages to be processed.
+	SoftSuspended,
+	/// Bridge is suspended and new messages are now being actively dropped.
+	HardSuspended,

	if bridge.state == BridgeState::Suspended(false) {
	if bridge.state == BridgeState::HardSuspended {

		// We cannot accept new messages to the suspended bridge, hoping that it'll be
		// actually resumed soon

	// We cannot accept new messages to the suspended bridge, hoping that it'll be
	// actually resumed soon
	// We cannot accept new messages and start dropping messages, until the outbound lane goes below the drop limit.

		@@ -0,0 +1,45 @@
		title: Bridges - revert-back and improve congestion

	title: Bridges - revert-back and improve congestion
	title: Bridges - Add improved congestion control mechanism

Bridges - revert-back and improve congestion #6231

Are you sure you want to change the base?

Bridges - revert-back and improve congestion #6231

Conversation

bkontur commented Oct 25, 2024 • edited Loading

Context

Description

bkontur commented Oct 25, 2024

bkontur commented Oct 26, 2024

bkontur commented Oct 28, 2024

bkontur commented Nov 5, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 7, 2024

bkontur commented Nov 8, 2024

serban300 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serban300 left a comment • edited Loading

Choose a reason for hiding this comment

bkontur commented Nov 20, 2024

bkontur commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

paritytech-workflow-stopper bot commented Nov 27, 2024

bkontur commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

acatangiu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkontur commented Oct 25, 2024 •

edited

Loading

serban300 left a comment •

edited

Loading