Introduce a delay feature to help stop rate-limiting?

[shhlife]

Some of our ComfyUI users who are using Groq are getting hit by rate limits. They had a suggestion to use a delay of a second to help stop that from happening. There's a ControlFlowUtility that has a wait node which we could use, but those wouldn't work with subtasks - only between agent runs.

How might we look at putting in a delay in the framework to stop ratelimits? is that something that would go on a driver? or should it be part of a task?

[collindutter]

All Prompt Drivers implement the ExponentialBackoffMixin which means they should automatically do this. How is the Groq Driver implemented?

[shhlife]

It's just using OpenAiChatPromptDriver with a different URL. Is the min_retry_delay in seconds? I wonder if I can try setting that to be a bit longer?

[collindutter]

@shhlife did you get a chance to try this out?

[shhlife]

thank you for the reminder! I haven't had a chance yet.. I'll give it a try!

[shhlife]

Hmm.. I tried setting it to 10, but I don't know if this is actually the right thing? the problem is hitting it too often when agents go back and forth, or are asking multiple agents in parallel.. so it's not retrying a call because of an error, but it's an agent that's using another agent (or multiple agents) as tools.. so it's just hitting the api too frequently.

[collindutter]

If i'm understanding the problem correctly, I think this is exactly what we want. Though I would expect the default value of 2 to be fine since it will exponentially increase on each attempt, giving the API plenty of breathing room.

[shhlife]

It seems like the user was hitting the rate limiting errors. sadly I havent been able to reproduce :(

[collindutter]

@shhlife I'm going to close this issue since we can't reproduce at this time. Feel free to re-open if we can get a repo.