Code halts if OpenAI rate limit is reached. openai.RateLimitError: Error code: 429

[JustUser1410]

After spending quite a bit of time and using a chunk of my resources code suddenly halted just to tell me that I need to "wait for 2.75s", but there is no option to continue research. The exception is raised by openai module, but could be handled more gracefully than just dropping all the progress. Agent could just take a break.

So far I have only encountered this while using multi-agent system.

File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai\_base_client.py", line 1020, in _request raise self._make_status_error_from_response(err.response) from None openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o in organization org-xxxxxxxxxxxxxxxxxxxxxxxxx on tokens per min (TPM): Limit 30000, Used 28966, Requested 2409. Please try again in 2.75s. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

Edit:
After looking into the code, I realized it's not that simple.

[JustUser1410]

Traceback (most recent call last):
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\main.py", line 32, in
asyncio.run(main())
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\asyncio\runners.py", line 118, in run
return self.loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\asyncio\base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\main.py", line 27, in main
research_report = await chief_editor.run_research_task()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\agents\master.py", line 57, in run_research_task
result = await chain.ainvoke({"task": self.task})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init.py", line 1504, in ainvoke
async for chunk in self.astream(
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init_.py", line 1333, in astream
panic_or_proceed(done, inflight, step)
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init.py", line 1537, in panic_or_proceed
raise exc
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel\retry.py", line 120, in arun_with_retry
await task.proc.ainvoke(task.input, task.config)
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\runnables\base.py", line 2540, in ainvoke
input = await step.ainvoke(input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\utils.py", line 117, in ainvoke
ret = await asyncio.create_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\agents\editor.py", line 85, in run_parallel_research
research_results = [result['draft'] for result in await asyncio.gather(*final_drafts)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init.py", line 1504, in ainvoke
async for chunk in self.astream(
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init_.py", line 1333, in astream
panic_or_proceed(done, inflight, step)
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel_init.py", line 1537, in _panic_or_proceed
raise exc
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\pregel\retry.py", line 120, in arun_with_retry
await task.proc.ainvoke(task.input, task.config)
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\runnables\base.py", line 2540, in ainvoke
input = await step.ainvoke(input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langgraph\utils.py", line 117, in ainvoke
ret = await asyncio.create_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\runnables\config.py", line 557, in run_in_executor
return await asyncio.get_running_loop().run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\runnables\config.py", line 548, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\agents\reviser.py", line 46, in run
revision = self.revise_draft(draft_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\agents\reviser.py", line 41, in revise_draft
response = call_model(prompt, model=task.get("model"), response_format='json')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\Documents\Python Workspace\gpt-researcher\multi_agents\agents\utils\llms.py", line 14, in call_model
response = ChatOpenAI(model=model, max_retries=max_retries, model_kwargs=optional_params).invoke(lc_messages).content
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\language_models\chat_models.py", line 170, in invoke
self.generate_prompt(
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\language_models\chat_models.py", line 599, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\language_models\chat_models.py", line 456, in generate
raise e
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\language_models\chat_models.py", line 446, in generate
self._generate_with_cache(
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_core\language_models\chat_models.py", line 671, in _generate_with_cache
result = self._generate(
^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\langchain_openai\chat_models\base.py", line 537, in _generate
response = self.client.create(messages=message_dicts, **params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_utils_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai\resources\chat\completions.py", line 606, in create
return self._post(
^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1005, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1053, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1005, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1053, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "C:\Users\Tomas\anaconda3\envs\gpt2\Lib\site-packages\openai_base_client.py", line 1020, in _request
raise self._make_status_error_from_response(err.response) from None
openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o in organization org-xxxxxxxxxxxxxxxxxxxxxxxxx on tokens per min (TPM): Limit 30000, Used 28966, Requested 2409. Please try again in 2.75s. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

[assafelovic]

Hey @JustUser1410, I'll have a look at this and add graceful timeouts

[Mitchell-xiyunfeng]

Hey @JustUser1410, I'll have a look at this and add graceful timeouts

Two important questions please:

1. First important question: Will GPT Researcher deployed on the RepoCloud also be able to synchronize updates once these improvements are completed?
2. The second important question: How does GPT Researcher deployed on the RepoCloud use local documents for knowledge QA? After making sure to add the DOC_PATH environment variable pointing to the documents folder.

[Daniel-K-Ivanov]

@assafelovic Do you have any update on the graceful timeouts?

[assafelovic]

@Daniel-K-Ivanov in our backlog to be shipped soon. Does this occur often?

[OX304]

@assafelovic Yes, especially when run on local documents that are of considerable size.

[craig-matadeen]

Similar to @OX304, I'm having the same issue when working with local documents. When this occurs, the user interface gives no indication of an issue that it stopped working and it just seems that it's taking a long time to complete.

Only in the docker compose logs do you see something like,

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for text-embedding-3-small in organization org-************** on tokens per min (TPM): Limit 1000000, Used 958784, Requested 189929. Please try again in 8.922s. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

The solution for this is to either use less documents and a shorter query or gracefully wait until the delayed time passes before continuing. It would be nice if the system doesn't just give up but rather backs off until the try again time, at least in the case of OpenAI. With the premature stop, it just ends up wasting resources from the OpenAI api.

As an aside, thanks for all the work that you guys do with this.

[coreylansdell]

@Daniel-K-Ivanov in our backlog to be shipped soon. Does this occur often?

I'm experience rate limits on 80% of my research attempts. Also curious if there is a limit on volume of text in local file in my-docs? When I enter larger form books it also seems to lock up. Unsure why.

[coreylansdell]

In general there really needs to be some more transparency as to the processes going on. Some kind of addition to the interface that indicates process and allows you to make decisions if an error occurs. such as restart, quit, wait, try again or anything. Also, having to look at terminal for the error info is annoying. I have no idea how all this works, I'm not a coder, I'm a designer. This interface is hard to work with and doesn't offer enough user prompts, feedback etc. Right now I am trying to run a research job and it just stopped. ... no feedback or error or anything. It's been quite a challenge to use this.

I'm also curious if there is a document limit for my-docs. Not just a single document length limitation that is causing problems.

[coreylansdell]

I just also want to say that it's amazing that you've built this. I have no concept of how hard this must be to do!