Dashboard for /busy, /retries & actively running jobs

[nitsujri]

This may be possible already in CloudWatch/SQS API, but it'd be amazing to have deeper insight into the system status.

• What is currently being worked on and how long it's been running (sidekiq/busy).
• What is awaiting retried (sidekiq/retries) (possible looking at message queues, but it's sort of wonky interface).
• Poll for specific job_id's current status.

The last one allows me to tell endusers what the status of their job is. Users can start long running processes and wait for it to finish (like requesting data outputs).

Is that last one possible without crazy heavy lifting?

[nitsujri]

Actually I just found https://github.com/inkstak/activejob-status for the last one. I'll give it a shot.

[metaskills]

Interesting. I'll assume you saw this section in the docs on observability https://github.com/customink/lambdakiq#observability-with-cloudwatch which gives you access to a TON of data directly within CloudWatch?

But to your point, feedback on how one job is doing via an ad-hoc SDK query too CloudWatch is never something I considered. In fact, not possible form what I understand either? Like you can not treat it as a Database for item level detail. So that project you linked too looks PERFECT for just that. Very cool share.

[nitsujri]

Yeah the CloudWatch dashboards are fantastic for rolled up view, similar to sidekiq/ but even more so since you can do crazy slicing & dicing.

For my 2nd request of peeking into retry queue (sidekiq/retries), I've learned that's not how SQS works. You can't see messages that are 'inflight' which is how the visibility timeout works for Lambdakiq's retry system.

Also, looking into a specific queue's pending messages like sidekiq/queues isn't a thing. Viewing the queue means to consume the message, a Schrödinger's message if you will, and it would trigger the redrive policy, meaning it increases the retry count by 1, moving it one step closer to a dead job.

This means getting visibility into the background job's health is done mostly via Rollbar or another error capturing system and a bit of CloudWatch as opposed to looking directly at it.

This message is mostly for other people who are still learning like me.

[metaskills]

Thanks for that reply!