Enhancement request: ability to postpone jobs

[tomaszklim]

Use case is simple:

• add the new job to the queue
• but postpone execution by eg. 7 minutes

How is that different from using sleep() within jobs?

If a particular job sleeps, then it affects also all waiting jobs. Job postpone time should be counted by task spooler, separately for each job.

How it should be implemented, to avoid breaking task spooler architecture?

1. ts should accept new command line parameter, eg. number of seconds to postpone the currently added job.
2. Each ts instance should internally count the minimal Unix epoch time, at which the job should start.
3. Job server should be contacted, when this time comes - earlier ts instances should just sleep and wait.

[justanhduc]

Hi @tomaszklim. This should be easy. May I ask what the practical application of such delay is in your workflow?

[tomaszklim]

Sure. I use it on https://polityka.co.pl/

Currently:

• import new articles from RSS sources and run AI analysis
• send notifications to various services, eg. to Bing IndexNow

What I want to do is implement multi-language sites, and use only task spooler without bigger queue mechanisms:

• import articles, run AI analysis and all tasks that need to be synchronous
• send notifications about articles in original language
• enqueue notifications to other services, about translated versions (mainly to english) - but postpone them by 5-7 minutes (until translations are ready)
• translate articles using external translation services
• do some other english version-related stuff

My goals are:

1. to avoid using complex workflows, where one asynchronous tasks creates another task, or even multiple tasks. I already have such workflows for importing images and it causes queue deadlocks at bigger scale.

2. to avoid crashing the importer/AI classifier engine - that's why, after importing the article, I want to trigger series of asynchronous tasks. And the problem is: tasks are run typically from 8 to 40 instances, depending on machine load etc. details. And since some tasks are much faster than others, there is no guarantee of executing them in order.

3. to avoid loading the AI engine and querying the database every X minutes just to trigger the notifications (I know it would be the most obvious approach).