Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage corruption in parachain_system::HostConfiguration due to async backing params #4603

Closed
2 tasks done
wischli opened this issue May 27, 2024 · 10 comments
Closed
2 tasks done

Storage corruption in parachain_system::HostConfiguration due to async backing params #4603

wischli opened this issue May 27, 2024 · 10 comments
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known. T16-async_backing Issues related to asynchronous backing.

Comments

@wischli
Copy link

wischli commented May 27, 2024 •
edited
Loading

Is there an existing issue?

  • I have searched the existing issues

This issue was already reported in #1978. However, the solution was to upgrade to the latest SDK version.

Just learned very low priority, upgrading to the latest polkadot sdk should fix this. Will close once fix is confirmed (source)

We experienced this issue as part of our upgrade from SDK v1.1.0 to v1.7.2 which was released in February 2024. The quote is from October 2023. So I don't think that solution is truly feasible.

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

The AbridgedHostConfiguration value of the HostConfiguration storage of parachain-system was appended by an async_backing_params: AsyncBackingParams field in a5f421e without bumping the storage version and adding a storage migration. Since AsyncBackingParams does not derive Default, this leads to storage corruption to the best of my knowledge.

I discovered this thanks to the execute-block CLI of try-runtime during our v1.1.0 -> v1.7.2 SDK update which logs an error for that storage.

[2024-05-27T15:50:45Z ERROR runtime::storage] Corrupted state at `0x45323df7cc47150b3930e2666b0aa313c522231880238a0c56021b8744a00743`: Error

Steps to reproduce

git clone -b wf/polkadot-v1.7.2-client https://github.com/centrifuge/centrifuge-chain.git
cd centrifuge-chain
cargo b -r -p centrifuge-chain -F try-runtime
try-runtime --runtime target/release/wbuild/centrifuge-runtime/centrifuge_runtime.wasm execute-block --try-state=all live --uri=wss://fullnode.centrifuge.io:443
@wischli wischli added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels May 27, 2024
@bkchr
Copy link
Member

bkchr commented May 27, 2024

I discovered this thanks to the execute-block CLI of try-runtime during our v1.1.0 -> v1.7.2 SDK update which logs an error for that storage.

https://github.com/centrifuge/centrifuge-chain/blob/v0.10.39/Cargo.toml looking at this, your runtime is still at 0.9.43. When you run try-runtime it will go from 0.9.43 to 1.7.2. So, triggering the issue that you linked above. So, there should be no issue.

Since AsyncBackingParams does not derive Default

This not being the issue. The value is loaded from storage and fails to be decoded and this prints the error that you posted above.

@alexggh alexggh added the T16-async_backing Issues related to asynchronous backing. label May 28, 2024
@wischli
Copy link
Author

wischli commented May 28, 2024

https://github.com/centrifuge/centrifuge-chain/blob/v0.10.39/Cargo.toml looking at this, your runtime is still at 0.9.43. When you run try-runtime it will go from 0.9.43 to 1.7.2. So, triggering the issue that you linked above. So, there should be no issue.

I see your point but the error occurs with the WASM compiled on our v1.7.2 branch as stated in the steps to reproduce. So the onchain WASM shouldn't matter as, should it? If it does, then this indicates we will still run into that issue when upgrading from v0.9.43 to v1.7.2 in one runtime upgrade. I would assume the fix to still exist in v1.7.2 without requiring an intermediate update, especially because it is not tied to any manual migration.

[2024-05-27T15:50:43Z INFO  try-runtime::cli] Original runtime [Name: RuntimeString::Owned("centrifuge")] [Version: 1028] [Code hash: 0x944d...baa6]
[2024-05-27T15:50:44Z INFO  try-runtime::cli] New runtime      [Name: RuntimeString::Owned("centrifuge")] [Version: 1029] [Code hash: 0x5c15...3ebf]

Moreover, what is the point of versioning the storage of a pallet if the version is not bumped for any change to the storage? Is this my assumption absolutely infeasible or is this issue and exception?

Since AsyncBackingParams does not derive Default

This not being the issue. The value is loaded from storage and fails to be decoded and this prints the error that you posted above.

Thanks. I thought that any storage (sub) item which cannot be decoded, falls back to the default implementation. Has this been true up to some point or always been false?

@bkchr
Copy link
Member

bkchr commented May 28, 2024 •
edited
Loading

I see your point but the error occurs with the WASM compiled on our v1.7.2 branch as stated in the steps to reproduce. So the onchain WASM shouldn't matter as, should it?

I mean I assume that you try the try-runtime from your on chain state to your new WASM? The on chain runtime doesn't know the async parameter and thus, the error will appear when running try-runtime because your new runtime requires the async parameter.

Moreover, what is the point of versioning the storage of a pallet if the version is not bumped for any change to the storage? Is this my assumption absolutely infeasible or is this issue and exception?

This should not have happened! We now have the try runtime checks for all the chains in CI to ensure that this does not happen again. In handsight, we should have created a fix and backported it. This should never happen again!

Has this been true up to some point or always been false?

I mean it falls back to the default value. However, the error message would still be printed because it failed to decode the value in the first place.

@wischli
Copy link
Author

wischli commented May 28, 2024

Thanks a lot for the quick and direct responses. IIUC, this does not require to write a migration and is just an issue as part of try-runtime for chains such as Centrifuge which are currently running on an ancient SDK version. Upgrading to a newer version will fix it automatically.

@bkchr
Copy link
Member

bkchr commented May 28, 2024

Upgrading to a newer version will fix it automatically.

Yes this will fix it. When the first block is build with the new runtime, the correct config is set and the issue will not appear anymore. It also has (should) not no influence on your chain.

@wischli wischli closed this as completed May 29, 2024
@wischli wischli mentioned this issue May 31, 2024
Merged
4 tasks
@Polkadot-Forum
Copy link

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/paritytech-update-for-april-2024/7646/5

@lemunozm lemunozm mentioned this issue Jun 24, 2024
Open
@Ad96el
Copy link

Ad96el commented Sep 2, 2024

Hi @bkchr,

Sorry to jump into this closed discussion, but we at the KILT protocol are currently facing the same issue. We're preparing a runtime release where we update the Polkadot dependencies from 1.0.0 to 1.7.0. Here is our current Cargo.toml. We are using the release-crates-io branches for the update. As mentioned in the issue here, try-runtime checks are failing with the same error:

[2024-05-27T15:50:45Z ERROR runtime::storage] Corrupted state at 0x45323df7cc47150b3930e2666b0aa313c522231880238a0c56021b8744a00743: Error

When we try to simulate the runtime upgrade in a local parachain setup (Paseo or Rococo as relay chain), the same error occurs, causing the chain to stall.

Here are the logs of the parachain:

Logs 

2024-08-30 14:27:48.006 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 484    
2024-08-30 14:27:48.006 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [100] 0 extrinsics, length: 484 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 8422064563, proof_size: 60079) (0%)    
2024-08-30 14:27:48.006  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #100 (0x69c3…205b)    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker cumulus-collator: [Parachain] Starting collation. relay_parent=0x69c36a088f33146cc3c8bb729d8b3ff72692b1c90e5507eb7c6dc4d65d78205b at=0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98
2024-08-30 14:27:48.008  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "AssetSwitchPool1" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(0)    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "Fungibles" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "MessageQueue" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] Initiating migration.    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] pallet_assets already on v1. No migration will run.    
2024-08-30 14:27:48.008  INFO tokio-runtime-worker frame_support::migrations: [Parachain] 🚚 Pallet "XcmpQueue" VersionedMigration migrating storage version from 3 to 4.    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for System    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Timestamp    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Indices    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Balances    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TransactionPayment    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Sudo    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Aura    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Session    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainStaking    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Authorship    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AuraExt    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Democracy    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Council    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalCommittee    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalMembership    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Treasury    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Utility    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Vesting    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Scheduler    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Proxy    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Preimage    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TipsMembership    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Tips    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Multisig    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AssetSwitchPool1    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Fungibles    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Ctype    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Attestation    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Delegation    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Did    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Inflation    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DidLookup    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Web3Names    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for PublicCredentials    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Migration    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DipProvider    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DepositStorage    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainSystem    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainInfo    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for XcmpQueue    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for PolkadotXcm    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for CumulusXcm    
2024-08-30 14:27:48.008 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DmpQueue    
2024-08-30 14:27:48.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for MessageQueue    
2024-08-30 14:27:48.009 ERROR tokio-runtime-worker runtime::storage: [Parachain] Corrupted state at `0x45323df7cc47150b3930e2666b0aa313c522231880238a0c56021b8744a00743`: Error    
2024-08-30 14:27:48.009 TRACE tokio-runtime-worker runtime::system: [Parachain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 6381120900, proof_size: 55726 } }    
2024-08-30 14:27:48.009 TRACE tokio-runtime-worker runtime::system: [Parachain] Used block length: 11    
2024-08-30 14:27:48.009 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/e046d3b/cumulus/pallets/parachain-system/src/lib.rs:1297:30:
included head not present in relay storage proof    
2024-08-30 14:27:48.009  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-08-30 14:27:48.009  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-08-30 14:27:48.009  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x592f9a - <unknown>!rust_begin_unwind
    1: 0x116c9 - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1
    2: 0x3fab73 - <unknown>!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors::h04cd1a17b3363e4a
    3: 0x2b6e8a - <unknown>!frame_support::storage::transactional::with_transaction::hcba920fbac0c8326
    4: 0x4740dd - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h12a172eaf43af8bc
    5: 0x481c79 - <unknown>!frame_support::dispatch_context::run_in_context::h6b7c0c332af222d8
    6: 0x86dfe - <unknown>!<peregrine_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h71849d6240075e86
    7: 0x86295 - <unknown>!<peregrine_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h5ba1509891806112
    8: 0x22a125 - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::ha8bd4268da23e05c
    9: 0x3d5b03 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h799a4f7ed1eb6d96
   10: 0x51cb4c - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-08-30 14:27:48.009 DEBUG tokio-runtime-worker runtime::xcmp-queue-migration: [Parachain] Lazy migration finished: item empty    
2024-08-30 14:27:48.009 DEBUG tokio-runtime-worker runtime::dmp-queue-export-xcms: [Parachain] on_idle: block=26, state=NotStarted, index=<wasm:stripped>    
2024-08-30 14:27:48.009 DEBUG tokio-runtime-worker runtime::dmp-queue-export-xcms: [Parachain] Init export at page 0    
2024-08-30 14:27:48.009 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/e046d3b/cumulus/pallets/parachain-system/src/lib.rs:265:18:
set_validation_data inherent needs to be present in every block!    
2024-08-30 14:27:48.009  WARN tokio-runtime-worker aura: [Parachain] Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x592f9a - <unknown>!rust_begin_unwind
    1: 0x116c9 - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1
    2: 0x1160f - <unknown>!core::option::expect_failed::hb9bb62f6de69a8ca
    3: 0x3f5017 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h6950f00409446b1d
    4: 0x575b6a - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hf094ae62d359ba82
    5: 0x3d5fcb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::hd4dfda8773968454
    6: 0x51cc5b - <unknown>!BlockBuilder_finalize_block    
2024-08-30 14:27:49.562  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #100 (0x69c3…205b), finalized #97 (0xd85e…8ea3), ⬇ 1.5kiB/s ⬆ 1.2kiB/s    
2024-08-30 14:27:49.572  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #25 (0xc808…ba98), finalized #24 (0x8362…4991), ⬇ 0 ⬆ 0    
2024-08-30 14:27:51.789  INFO tokio-runtime-worker grandpa: [Relaychain] 👴 Applying authority set change scheduled at block #98    
2024-08-30 14:27:51.789  INFO tokio-runtime-worker grandpa: [Relaychain] 👴 Applying GRANDPA set change to new set [(Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...)), 1), (Public(d17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69 (5GoNkf6W...)), 1)]    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] elems: [0x2df9bfdc0f63917efcea32e2141b1d2d72df632602587290bcbd31dd14741bf4]    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_before: [126, 189, 196]    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_after: [126, 189, 196, 197]    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 197 parent_hash 0x69c36a088f33146cc3c8bb729d8b3ff72692b1c90e5507eb7c6dc4d65d78205b key [12, 109, 109, 114, 197, 0, 0, 0, 0, 0, 0, 0, 105, 195, 106, 8, 143, 51, 20, 108, 195, 200, 187, 114, 157, 139, 63, 247, 38, 146, 177, 201, 14, 85, 7, 235, 124, 109, 196, 214, 93, 120, 32, 91]    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 2772454563, proof_size: 48930 } }    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 11    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] [process_inherent_data] bitfields.len(): 2, backed_candidates.len(): 0, disputes.len() 0    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Size before filter: 142, candidates + bitfields: 141, disputes: 1    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Time weight before filter: 5071950000, candidates + bitfields: 5071950000, disputes: 0    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Max block weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block time weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block size: 5242880    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block weight: Weight(ref_time: 2000000000000, proof_size: 5242880)    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::parachains::scheduler: [Relaychain] [occupied] now_occupied {}    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 7942640563, proof_size: 49072 } }    
2024-08-30 14:27:54.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 484    
2024-08-30 14:27:54.007 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [101] 0 extrinsics, length: 484 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 8100081563, proof_size: 52592) (0%)    
2024-08-30 14:27:54.008  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #101 (0xf92a…8b51)    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker cumulus-collator: [Parachain] Starting collation. relay_parent=0xf92a612c0a22ed19c6d5ac05c05e7b9f5d57cc8d86ec9abc5c5dea2da3838b51 at=0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98
2024-08-30 14:27:54.009  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "AssetSwitchPool1" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(0)    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "Fungibles" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "MessageQueue" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] Initiating migration.    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] pallet_assets already on v1. No migration will run.    
2024-08-30 14:27:54.009  INFO tokio-runtime-worker frame_support::migrations: [Parachain] 🚚 Pallet "XcmpQueue" VersionedMigration migrating storage version from 3 to 4.    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for System    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Timestamp    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Indices    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Balances    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TransactionPayment    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Sudo    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Aura    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Session    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainStaking    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Authorship    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AuraExt    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Democracy    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Council    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalCommittee    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalMembership    
2024-08-30 14:27:54.009 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Treasury    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Utility    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Vesting    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Scheduler    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Proxy    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Preimage    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TipsMembership    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Tips    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Multisig    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AssetSwitchPool1    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Fungibles    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Ctype    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Attestation    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Delegation    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Did    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Inflation    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DidLookup    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Web3Names    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for PublicCredentials    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Migration    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DipProvider    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DepositStorage    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainSystem    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainInfo    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for XcmpQueue    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for PolkadotXcm    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for CumulusXcm    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DmpQueue    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for MessageQueue    
2024-08-30 14:27:54.010 ERROR tokio-runtime-worker runtime::storage: [Parachain] Corrupted state at `0x45323df7cc47150b3930e2666b0aa313c522231880238a0c56021b8744a00743`: Error    
2024-08-30 14:27:54.010 TRACE tokio-runtime-worker runtime::system: [Parachain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 6381120900, proof_size: 55726 } }    
2024-08-30 14:27:54.010 TRACE tokio-runtime-worker runtime::system: [Parachain] Used block length: 11    
2024-08-30 14:27:54.010 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/e046d3b/cumulus/pallets/parachain-system/src/lib.rs:1297:30:
included head not present in relay storage proof    
2024-08-30 14:27:54.010  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-08-30 14:27:54.010  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-08-30 14:27:54.010  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x592f9a - <unknown>!rust_begin_unwind
    1: 0x116c9 - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1
    2: 0x3fab73 - <unknown>!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors::h04cd1a17b3363e4a
    3: 0x2b6e8a - <unknown>!frame_support::storage::transactional::with_transaction::hcba920fbac0c8326
    4: 0x4740dd - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h12a172eaf43af8bc
    5: 0x481c79 - <unknown>!frame_support::dispatch_context::run_in_context::h6b7c0c332af222d8
    6: 0x86dfe - <unknown>!<peregrine_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h71849d6240075e86
    7: 0x86295 - <unknown>!<peregrine_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h5ba1509891806112
    8: 0x22a125 - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::ha8bd4268da23e05c
    9: 0x3d5b03 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h799a4f7ed1eb6d96
   10: 0x51cb4c - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::xcmp-queue-migration: [Parachain] Lazy migration finished: item empty    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::dmp-queue-export-xcms: [Parachain] on_idle: block=26, state=NotStarted, index=<wasm:stripped>    
2024-08-30 14:27:54.010 DEBUG tokio-runtime-worker runtime::dmp-queue-export-xcms: [Parachain] Init export at page 0    
2024-08-30 14:27:54.010 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/e046d3b/cumulus/pallets/parachain-system/src/lib.rs:265:18:
set_validation_data inherent needs to be present in every block!    
2024-08-30 14:27:54.010  WARN tokio-runtime-worker aura: [Parachain] Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x592f9a - <unknown>!rust_begin_unwind
    1: 0x116c9 - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1
    2: 0x1160f - <unknown>!core::option::expect_failed::hb9bb62f6de69a8ca
    3: 0x3f5017 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h6950f00409446b1d
    4: 0x575b6a - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hf094ae62d359ba82
    5: 0x3d5fcb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::hd4dfda8773968454
    6: 0x51cc5b - <unknown>!BlockBuilder_finalize_block    
2024-08-30 14:27:54.562  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #101 (0xf92a…8b51), finalized #98 (0x0948…244e), ⬇ 1.2kiB/s ⬆ 1.0kiB/s    
2024-08-30 14:27:54.572  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #25 (0xc808…ba98), finalized #25 (0xc808…ba98), ⬇ 12 B/s ⬆ 12 B/s    
2024-08-30 14:27:59.563  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #101 (0xf92a…8b51), finalized #98 (0x0948…244e), ⬇ 1.0kiB/s ⬆ 0.9kiB/s    
2024-08-30 14:27:59.573  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #25 (0xc808…ba98), finalized #25 (0xc808…ba98), ⬇ 92 B/s ⬆ 92 B/s    
2024-08-30 14:28:00.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] elems: [0xbfbd6567b79e9026549d2384f6c98c8b0a3e8890aa31fe9c7b1f9b8590cd6ec6, 0x7b2b3e9feb15a2b67bc11759a461cff4bbdcc34f925a806dea7fa79a9fddb2c4]    
2024-08-30 14:28:00.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_before: [126, 189, 196, 197]    
2024-08-30 14:28:00.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_after: [126, 189, 196, 199]    
2024-08-30 14:28:00.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 198 parent_hash 0xf92a612c0a22ed19c6d5ac05c05e7b9f5d57cc8d86ec9abc5c5dea2da3838b51 key [12, 109, 109, 114, 198, 0, 0, 0, 0, 0, 0, 0, 249, 42, 97, 44, 10, 34, 237, 25, 198, 213, 172, 5, 192, 94, 123, 159, 93, 87, 204, 141, 134, 236, 154, 188, 92, 93, 234, 45, 163, 131, 139, 81]    
2024-08-30 14:28:00.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 199 parent_hash 0xf92a612c0a22ed19c6d5ac05c05e7b9f5d57cc8d86ec9abc5c5dea2da3838b51 key [12, 109, 109, 114, 199, 0, 0, 0, 0, 0, 0, 0, 249, 42, 97, 44, 10, 34, 237, 25, 198, 213, 172, 5, 192, 94, 123, 159, 93, 87, 204, 141, 134, 236, 154, 188, 92, 93, 234, 45, 163, 131, 139, 81]    
2024-08-30 14:28:00.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 2772454563, proof_size: 48930 } }    
2024-08-30 14:28:00.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 11    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] [process_inherent_data] bitfields.len(): 2, backed_candidates.len(): 0, disputes.len() 0    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Size before filter: 142, candidates + bitfields: 141, disputes: 1    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Time weight before filter: 5071950000, candidates + bitfields: 5071950000, disputes: 0    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Max block weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block time weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block size: 5242880    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block weight: Weight(ref_time: 2000000000000, proof_size: 5242880)    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::parachains::scheduler: [Relaychain] [occupied] now_occupied {}    
2024-08-30 14:28:00.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 7942640563, proof_size: 49072 } }    
2024-08-30 14:28:00.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 484    
2024-08-30 14:28:00.008 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [102] 0 extrinsics, length: 484 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 8100081563, proof_size: 52592) (0%)    
2024-08-30 14:28:00.009  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #102 (0x7cb2…709a)    
2024-08-30 14:28:00.010  INFO tokio-runtime-worker cumulus-collator: [Parachain] Starting collation. relay_parent=0x7cb2b26ebbcae76fbff08b12c775b575a307d81d61112967f31bba99c560709a at=0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98
2024-08-30 14:28:04.563  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #102 (0x7cb2…709a), finalized #100 (0x69c3…205b), ⬇ 2.5kiB/s ⬆ 1.4kiB/s    
2024-08-30 14:28:04.573  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #25 (0xc808…ba98), finalized #25 (0xc808…ba98), ⬇ 41 B/s ⬆ 41 B/s    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] elems: [0xeb4f475fbc742b50ec3dd730d0a41b96c31c87d3f78d4341af1ffeec09197204]    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_before: [126, 189, 196, 199]    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_after: [126, 189, 196, 199, 200]    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 200 parent_hash 0x7cb2b26ebbcae76fbff08b12c775b575a307d81d61112967f31bba99c560709a key [12, 109, 109, 114, 200, 0, 0, 0, 0, 0, 0, 0, 124, 178, 178, 110, 187, 202, 231, 111, 191, 240, 139, 18, 199, 117, 181, 117, 163, 7, 216, 29, 97, 17, 41, 103, 243, 27, 186, 153, 197, 96, 112, 154]    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 2897454563, proof_size: 48930 } }    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 11    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] [process_inherent_data] bitfields.len(): 2, backed_candidates.len(): 0, disputes.len() 0    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Size before filter: 142, candidates + bitfields: 141, disputes: 1    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Time weight before filter: 5071950000, candidates + bitfields: 5071950000, disputes: 0    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Max block weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block time weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block size: 5242880    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block weight: Weight(ref_time: 2000000000000, proof_size: 5242880)    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::parachains::scheduler: [Relaychain] [occupied] now_occupied {}    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 8067640563, proof_size: 49072 } }    
2024-08-30 14:28:06.008 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 484    
2024-08-30 14:28:06.008 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [103] 0 extrinsics, length: 484 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 8225081563, proof_size: 52592) (0%)    
2024-08-30 14:28:06.009  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #103 (0x2f9c…d0bc)    
2024-08-30 14:28:06.010  INFO tokio-runtime-worker cumulus-collator: [Parachain] Starting collation. relay_parent=0x2f9c5304a9d01feee929f356645b613c46aac7ce91f638748cde733a6e1fd0bc at=0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98
2024-08-30 14:28:09.563  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #103 (0x2f9c…d0bc), finalized #100 (0x69c3…205b), ⬇ 1.3kiB/s ⬆ 1.1kiB/s    
2024-08-30 14:28:09.573  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #25 (0xc808…ba98), finalized #25 (0xc808…ba98), ⬇ 12 B/s ⬆ 12 B/s    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] elems: [0x8c22e23369ba1810c550567fc07efb9d3766caf69b110b25bcfcaeb55548a3ac, 0xddc9a44ebc54c83a97439fe39626fc2d55c11cc470f6532da5e0de6912e5878c, 0xa8db2b91f365a7a1126084bfe38d8e3bc5429890a6cf5e4a893bef83797f1afb, 0x054aa4781073d2d95759c66e2d77aed54d044a144528a8236478b4342baee57c]    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_before: [126, 189, 196, 199, 200]    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::mmr: [Relaychain] peaks_after: [126, 189, 204]    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 201 parent_hash 0x2f9c5304a9d01feee929f356645b613c46aac7ce91f638748cde733a6e1fd0bc key [12, 109, 109, 114, 201, 0, 0, 0, 0, 0, 0, 0, 47, 156, 83, 4, 169, 208, 31, 238, 233, 41, 243, 86, 100, 91, 97, 60, 70, 170, 199, 206, 145, 246, 56, 116, 140, 222, 115, 58, 110, 31, 208, 188]    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 202 parent_hash 0x2f9c5304a9d01feee929f356645b613c46aac7ce91f638748cde733a6e1fd0bc key [12, 109, 109, 114, 202, 0, 0, 0, 0, 0, 0, 0, 47, 156, 83, 4, 169, 208, 31, 238, 233, 41, 243, 86, 100, 91, 97, 60, 70, 170, 199, 206, 145, 246, 56, 116, 140, 222, 115, 58, 110, 31, 208, 188]    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 203 parent_hash 0x2f9c5304a9d01feee929f356645b613c46aac7ce91f638748cde733a6e1fd0bc key [12, 109, 109, 114, 203, 0, 0, 0, 0, 0, 0, 0, 47, 156, 83, 4, 169, 208, 31, 238, 233, 41, 243, 86, 100, 91, 97, 60, 70, 170, 199, 206, 145, 246, 56, 116, 140, 222, 115, 58, 110, 31, 208, 188]    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::mmr::offchain: [Relaychain] offchain db set: pos 204 parent_hash 0x2f9c5304a9d01feee929f356645b613c46aac7ce91f638748cde733a6e1fd0bc key [12, 109, 109, 114, 204, 0, 0, 0, 0, 0, 0, 0, 47, 156, 83, 4, 169, 208, 31, 238, 233, 41, 243, 86, 100, 91, 97, 60, 70, 170, 199, 206, 145, 246, 56, 116, 140, 222, 115, 58, 110, 31, 208, 188]    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 2897454563, proof_size: 48930 } }    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 11    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] [process_inherent_data] bitfields.len(): 2, backed_candidates.len(): 0, disputes.len() 0    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Size before filter: 142, candidates + bitfields: 141, disputes: 1    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Time weight before filter: 5071950000, candidates + bitfields: 5071950000, disputes: 0    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Max block weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block time weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block size: 5242880    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block weight: Weight(ref_time: 2000000000000, proof_size: 5242880)    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::parachains::scheduler: [Relaychain] [occupied] now_occupied {}    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 8067640563, proof_size: 49072 } }    
2024-08-30 14:28:12.007 TRACE tokio-runtime-worker runtime::system: [Relaychain] Used block length: 484    
2024-08-30 14:28:12.007 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [104] 0 extrinsics, length: 484 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 8225081563, proof_size: 52592) (0%)    
2024-08-30 14:28:12.008  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #104 (0x8365…3949)    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker cumulus-collator: [Parachain] Starting collation. relay_parent=0x8365de398b3be16c453934cf116123e56ecfa74ef88eb69f9285223f22b83949 at=0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98
2024-08-30 14:28:12.009  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xc8083cb85a7fc82ca3cd6f361a541938b7df1761509b16f22a4855ec47a3ba98    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "AssetSwitchPool1" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(0)    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "Fungibles" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker runtime::frame-support: [Parachain] 🐥 New pallet "MessageQueue" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] Initiating migration.    
2024-08-30 14:28:12.009  INFO tokio-runtime-worker migration::BumpStorageVersion: [Parachain] pallet_assets already on v1. No migration will run.    
2024-08-30 14:28:12.010  INFO tokio-runtime-worker frame_support::migrations: [Parachain] 🚚 Pallet "XcmpQueue" VersionedMigration migrating storage version from 3 to 4.

We are hesitant to continue with the release until we can successfully update the runtime in a local environment.

I would appreciate any suggestions and support to help overcome this problem. 🙌

@Ad96el
Copy link

Ad96el commented Sep 2, 2024 •
edited
Loading

Okay, I reviewed the implementations of other parachain teams, and noticed that none of them were implementing the parameterized-consensus-hook. After removing the hook and sticking with the old implementation, the runtime upgrade was successful.

@bkchr
Copy link
Member

bkchr commented Sep 2, 2024

Sorry for the late reply.

included head not present in relay storage proof

This should be the problem. Updating your collators before doing the runtime upgrade should fix the issue.

@Ad96el
Copy link

Ad96el commented Sep 3, 2024

Thanks for the hint. I introduced a silly bug while updating the client to be compatible with async backing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known. T16-async_backing Issues related to asynchronous backing.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bkchr @wischli @Ad96el @alexggh @Polkadot-Forum