Update Scheduler to Support Relay Chain Block Number Provider #6362

gupnik · 2024-11-05T09:02:14Z

Step in #6297

This PR adds the ability for the Scheduler pallet to specify its source of the block number. This is needed for the scheduler pallet to work on a parachain which does not produce blocks on a regular schedule, thus can use the relay chain as a block provider. Because blocks are not produced regularly, we cannot make the assumption that the block number increases monotonically, and thus have a new logic via a Queue to handle multiple blocks with valid agenda passed between them.

This change only needs a migration for the Queue:

If the BlockNumberProvider continues to use the system pallet's block number
When a pallet deployed on the relay chain is moved to a parachain, but still uses the
relay chain's block number

However, we would need separate migrations if the deployed pallets are upgraded on an existing parachain, and the BlockNumberProvider uses the relay chain block number.

Todo

Update Benchmarks
Migration

substrate/frame/scheduler/src/lib.rs

muharem · 2024-11-07T12:43:17Z

substrate/frame/scheduler/src/lib.rs

@@ -1157,24 +1181,30 @@ impl<T: Config> Pallet<T> {
 			return
 		}

-		let mut incomplete_since = now + One::one();
-		let mut when = IncompleteSince::<T>::take().unwrap_or(now);


Can you explain why would not it work with IncompleteSince, without the block Queue?
How we determine the MaxScheduledBlocks bound?
With the IncompleteSince we iterate over blocks that might have no task to execute and this might make a situation with many incomplete blocks even worth. But probably not too much? One more read?
Both solutions need a strategy for a situation when there are two many tasks that can not be completed and the task queue only grow. If such strategy not yet in place.

With the IncompleteSince we iterate over blocks that might have no task to execute and this might make a situation with many incomplete blocks even worth. But probably not too much? One more read?

Yes, but then this becomes unbounded in case too many blocks are skipped. The idea behind using the Queue is to bound this to a sufficient number.

How we determine the MaxScheduledBlocks bound?

This should be determined similar to the existing MaxScheduledPerBlock?

Both solutions need a strategy for a situation when there are two many tasks that can not be completed and the task queue only grow. If such strategy not yet in place.

There is already a retry mechanism and the task is purged if the retry count is exceeded (even if failed).

The Queue not only bounds how many blocks gonna be processed from the past. It bounds for how many blocks we can schedule. If the number is 50, we can schedule only 50 jobs with distinct schedule time.

The MaxScheduledPerBlock for me seems simpler to define. Because the block size its exiting constrain the system have. But how many distinct schedule time points you can have is something new.

Retries work in case if a certain task fails while it's function call is being executed (not the scheduler fail). I meant a case when there are many (or few but too heavy) overdue tasks (task_block < now), so that the scheduler never (or needs too many time) to complete them and exist such overdue state to start processing tasks in time. Do we handle such case?

The Queue not only bounds how many blocks gonna be processed from the past. It bounds for how many blocks we can schedule. If the number is 50, we can schedule only 50 jobs with distinct schedule time

Indeed, I do not find it quite comfortable to run a for loop with IncompleteSince when there could be an unknown number of blocks passed between the successive runs. You could always keep the MaxScheduledBlocks on the higher side that would give you a similar experience?

I meant a case when there are many (or few but too heavy) overdue tasks (task_block < now), so that the scheduler never (or needs too many time) to complete them and exist such overdue state to start processing tasks in time. Do we handle such case?

But this stays as an issue even in the current implementation? The change here just makes it bounded, so that the scheduling itself is blocked in such a case.

Maybe we can put a quite big bound on the MaxScheduledBlocks, it is just a vec of block numbers.

I see, indeed it is bad for PoV, as it is read every block.

The situation we want to fix is when the scheduler is using the relay chain block, and the parachain doesn't execute often.

(1) Maybe in this case the scheduler should use a different block provider with less granularity like relay chain block / 100 so that when doing IncompleteSince it increments with a step of 100 relay chain block until it arrives to now.

(2) Or otherwise we can have a more complex structure for the queue. We cut the vector in chunck of 100 blocks.
So we have a double map with first key is block number / 100 and second key is block number % 100, the value is a vector of length at most 100.

But still if the parachains wake up every month it can be not good. But at this point they should use (1).

EDIT: I agree we can also just ignore this situation with a MaxStaleTaskAge parameter. IMO it is fine. And people can do (1) if their parachain executes too much rarely

Added MaxStaleTaskAge as suggested. Thanks both.

I see, indeed it is bad for PoV, as it is read every block.

Also the task scheduling is affected.

The situation we want to fix is when the scheduler is using the relay chain block, and the parachain doesn't execute often.

I think we have next cases today/planned soon:

Relay Chain with scheduler working with local block provider. No concerns. The new Queue is even redundant;

Parachain with scheduler working with local block provider. Same as (1);

Parachain with scheduler working with Relay Chain block provider;
3.1 runs scheduler on every second RC block, same as (1);
3.2 RC or Parachain for some reason is not producing blocks for 2 hours, we have 1200 blocks to iterate through.

We have a problem with (3.2) case only. On the current version (without Queue) it will eventually handle the overdue blocks (we can even calculate how many blocks it will take, lets say if there is no tasks scheduled in that period). With the Queue such situation as (3.2) gonna be handled well, but with a cost.

I would look into numbers, if with the current version we can handle 2 hours of overdue in some reasonable time (lets say 10 blocks), then I think we are fine even with current solution, we just need tests for it. If not, may be we can introduce the Queue in a way that it can be disabled for (1) and (2) cases.

I just checked that we currently use Scheduler only for the Governance related pallets. I think the related tasks should be better eventually processed than dropped if too old. So MaxStaleTaskAge should be at least optional.

Yes the referenda pallet creates an alarm for every ref to check the voting turnout.

We have a problem with (3.2) case only. On the current version (without Queue) it will eventually handle the overdue blocks (we can even calculate how many blocks it will take, lets say if there is no tasks scheduled in that period).

Depends on how many blocks are produced. I guess when we assume that the parachain will produce blocks at least as fast as it can advance the scheduler then yes.
Playing devils advocate here since there could be parachains that only produce one block every two hours, which would get stuck without ever catching up the IncompleteSince.

Conceptually, I believe that a priority Queue is the right data structure. We try to evaluate an ordered list of tasks by their order. It is exactly what a priority queue is good at. The issue with implementing this as a Vector is obviously the PoV.

Maybe we can implement the Queue as a B Tree? Then we can get the next task in log reads and insert in log writes. And it allows us to do exactly what we want: get the next pending task. It could be PoV optimized by using chunks as well.
To me it just seems that most of the pain here is that we are using the wrong data structure for the job.

…ik/scheduler-bnp

ggwpez · 2024-11-27T14:53:01Z

substrate/frame/scheduler/src/lib.rs


-	#[pallet::storage]
-	pub type IncompleteSince<T: Config> = StorageValue<_, BlockNumberFor<T>>;
+		/// Provider for the block number. Normally this is the `frame_system` pallet.


Normally in what case? Parachain or relay/solo?

ggwpez · 2024-11-27T14:53:35Z

substrate/frame/scheduler/src/lib.rs

+		/// Provider for the block number. Normally this is the `frame_system` pallet.
+		type BlockNumberProvider: BlockNumberProvider;
+
+		/// The maximum number of blocks that can be scheduled.


Any hints on how to configure this? Parachain teams will read this and not know what number to put.

ggwpez · 2024-11-27T14:53:46Z

substrate/frame/scheduler/src/lib.rs

+		#[pallet::constant]
+		type MaxScheduledBlocks: Get<u32>;
+
+		/// The maximum number of blocks that a task can be stale for.


Also maybe a hint for a sane default value.

ggwpez · 2024-11-27T15:00:54Z

substrate/frame/scheduler/src/lib.rs

+	/// The queue of block numbers that have scheduled agendas.
+	#[pallet::storage]
+	pub(crate) type Queue<T: Config> =
+		StorageValue<_, BoundedVec<BlockNumberFor<T>, T::MaxScheduledBlocks>, ValueQuery>;


Do we know if one vector is enough? I think the referenda pallet creates an alarm for each ref...

ggwpez · 2024-11-27T15:21:32Z

substrate/frame/scheduler/src/lib.rs

@@ -1157,24 +1181,30 @@ impl<T: Config> Pallet<T> {
 			return
 		}

-		let mut incomplete_since = now + One::one();
-		let mut when = IncompleteSince::<T>::take().unwrap_or(now);


Yes the referenda pallet creates an alarm for every ref to check the voting turnout.

We have a problem with (3.2) case only. On the current version (without Queue) it will eventually handle the overdue blocks (we can even calculate how many blocks it will take, lets say if there is no tasks scheduled in that period).

Depends on how many blocks are produced. I guess when we assume that the parachain will produce blocks at least as fast as it can advance the scheduler then yes.
Playing devils advocate here since there could be parachains that only produce one block every two hours, which would get stuck without ever catching up the IncompleteSince.

Conceptually, I believe that a priority Queue is the right data structure. We try to evaluate an ordered list of tasks by their order. It is exactly what a priority queue is good at. The issue with implementing this as a Vector is obviously the PoV.

Maybe we can implement the Queue as a B Tree? Then we can get the next task in log reads and insert in log writes. And it allows us to do exactly what we want: get the next pending task. It could be PoV optimized by using chunks as well.
To me it just seems that most of the pain here is that we are using the wrong data structure for the job.

gupnik added 2 commits November 5, 2024 14:24

Adds BlockNumberProvider

f8e5c63

Adds a Queue to handle non-sequential block numbers

455d765

gupnik added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Nov 5, 2024

gupnik requested a review from a team as a code owner November 5, 2024 09:02

muharem self-requested a review November 5, 2024 09:49

gui1117 reviewed Nov 6, 2024

View reviewed changes

substrate/frame/scheduler/src/lib.rs Outdated Show resolved Hide resolved

muharem reviewed Nov 7, 2024

View reviewed changes

gupnik mentioned this pull request Nov 9, 2024

[tracking] adopt BlockNumberProvider for the pallets migrating to AH #6297

Open

Removes constraint

762c816

paritytech-review-bot bot requested a review from a team November 11, 2024 10:51

gupnik added 4 commits November 13, 2024 09:52

Updates benchmarks

86701cf

Adds migration

27d1fa1

FMT

c79900c

Adds PrDoc

622805d

gupnik changed the title ~~[WIP]: Update Scheduler to Support Relay Chain Block Number Provider #3970~~ Update Scheduler to Support Relay Chain Block Number Provider Nov 14, 2024

gupnik and others added 12 commits November 14, 2024 10:20

Updates PrDoc

abbd346

Merge branch 'master' into gupnik/scheduler-bnp

844662e

Fixes

813e596

Adds MaxStaleTaskAge

8c40f24

Fixes

b905628

Minor fix

2ea271a

Minor fix

9a73c23

Removes unused import

ece2027

Minor

da116a7

Minor

b1b414a

Merge branch 'master' of github.com:paritytech/polkadot-sdk into gupn…

584f98f

…ik/scheduler-bnp

Merge branch 'master' into gupnik/scheduler-bnp

fe918e5

ggwpez reviewed Nov 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Scheduler to Support Relay Chain Block Number Provider #6362

Update Scheduler to Support Relay Chain Block Number Provider #6362

gupnik commented Nov 5, 2024 •

edited

Loading

muharem Nov 7, 2024

gupnik Nov 8, 2024

muharem Nov 13, 2024

gupnik Nov 14, 2024

gui1117 Nov 14, 2024

gui1117 Nov 15, 2024 •

edited

Loading

gupnik Nov 15, 2024

muharem Nov 25, 2024

muharem Nov 25, 2024

ggwpez Nov 27, 2024 •

edited

Loading

ggwpez Nov 27, 2024

ggwpez Nov 27, 2024

ggwpez Nov 27, 2024

ggwpez Nov 27, 2024

ggwpez Nov 27, 2024 •

edited

Loading

Update Scheduler to Support Relay Chain Block Number Provider #6362

Are you sure you want to change the base?

Update Scheduler to Support Relay Chain Block Number Provider #6362

Conversation

gupnik commented Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gui1117 Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggwpez Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggwpez Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

gupnik commented Nov 5, 2024 •

edited

Loading

gui1117 Nov 15, 2024 •

edited

Loading

ggwpez Nov 27, 2024 •

edited

Loading

ggwpez Nov 27, 2024 •

edited

Loading