CeleryBeat unable to load large number of tasks

[nitishxp]

Summary:

Include a brief descrioption of the problem here, and fill out the version info below.

• Celery Version: 5.1.2
• Celery-Beat Version: 2.2.1

Exact steps to reproduce the issue:

1. Load 1 million task in Periodic task
2. Start celery beat it will be stucked forever

Detailed information

We can have multiple(in millions) periodic active tasks. Below screenshot shows that celery beat is stucked in loading tasks from db

[nitishxp]

I overwritten the DatabaseScheduler to have implemented threading in schedule parsing does it make sense?

[auvipy]

can you come with a PR with that? to have more inputs and eyes?

[ktowen]

I'm having the same problem with around 30k ClockedSchedule tasks. The celery beat process seems to be stuck for some time at a stable medium cpu usage, then it drops and sends some tasks and starts again.

Any clue about where the problem could be or possible solutions? I would be happy to collaborate

[auvipy]

can you please try this patch and see if that helps? #478