When we scale Nest in K8s or node cluster, @Cron()
decorator will run many times since each Nest instance is isolated.
A Nest instance can not know did cron run in others instance or not?
If "every day cronjob" fails, we must wait a day for next cron. We better config maxRetries
and turnOffWhenMaxRetries
Save cronjob history to show metric, how long does it take? how many times it retries and view history result to debug
Some important cronjob need to alert when fails but we can't check console log every times
- Horizontal Scale
- Retry Mechanism
- Logging and Metric
- Notification Alert
We use a Redis key to mark a cronjob is running by a Nest instance. Redis atomic action behavior will make sure only 1 (random) instance can run a cronjob
@Cron(CronExpression.EVERY_HOUR, {
name: 'redis_key',
})
handleCronEveryHour() {
}