Events sources watching Redis Broker events to run Celery tasks #2882
bumper-jorge-pino
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have an idea that consists on triggering an Argo workflow that spins up a pod to run a Celery task every time Celery distributes a queue through a Redis broker. That way I could capture the task we want to run and the parameters required and run a pod that runs a task until it is completed rather than having Celery workers running permanently.
So far, the only way I can configure Redis event sources is through streams or channels which I don't think is the way Celery distributes the queues.
I don't know whether there is a simple way to watch Redis broker events or if the right way to do this would be to replace all Celery functionality with Argo Events (Websockets & Cronjob) and Workflows (which it would require more effort and collaboration).
Beta Was this translation helpful? Give feedback.
All reactions