Cancelled error with langgraph runs

[kedarsp-informa]

I have an inconsistent issue as CancelledError().
I have a langgraph and the error seems to be within langgraph library.
I am also using asyncio in my code as:

workflow.add_node(RAG, lambda state: asyncio.run(generate_with_rag(state=state, generation_rag_chain=generation_rag_chain)))
workflow.add_node(STORE_CHAT, lambda state: asyncio.run(store_chat(state=state, memory=memory)))

and

if chat_history:
        task1 = asyncio.create_task(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        task2 = asyncio.create_task(rephrase_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        source, follow_up_question = await asyncio.gather(task1, task2)
    else:
        source = await asyncio.shield(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        follow_up_question = question

The below is the error trace:
_CancelledError()Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1189, in astream
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 418, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 525, in wait
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traeback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1132, in astream
async with AsyncPregelLoop(
asyncio.exceptions.CancelledError
If anyone has encountered this error before or any idea on this would be helpful. Thanks

[delorenzotommaso]

I get the same error when execution time reaches about 60s

[tommasodelorenzo]

Hi, nope. It is a graph called from a fastAPI endpoint deployed as an azure container app. Logging everything to LangSmith

[vikyw89]

same, i'm experiencing the same thing on latest langgraph, cancelled error when ainvoke last about 60s. And i'm not using asyncio run, I do use asyncer asyncify inside the node, not to invoke the langgraph

[vikyw89]

I think it's internal langgraph timeout

[vikyw89]

Cancelled by cancel scope 7fea082e1520

[vikyw89]

Tried with await sleep, couldnt replicate this one...

[nfcampos]

Hi @kedarsp-informa you shouldn't use asyncio.run in add_node we have native support for async functions in nodes. Using asyncio.run would create a new event loop for each node, which is very inefficient and can cause issues such as cancellation not propagating from graph to node etc

[NicholasMicali]

I get the same error when attempting to run a subprocess to install a linter on startup. I am using langgraph cloud. Unsure about what "shouldn't use asyncio.run in add_node" means. this is a very simple graph, which also successfully runs my subprocess to install a linter, but fails afterwards. Deploys successfully without this line:

subprocess.run(["bash", "build_linter.sh"], check=True)

Registering graph with id 'agent'
12/17/2024, 1:46:06 PM
Starting metadata loop
12/17/2024, 1:46:07 PM
Starting cron scheduler
12/17/2024, 1:46:07 PM
Starting 10 background workers
12/17/2024, 1:46:07 PM
Application startup complete.
12/17/2024, 1:46:07 PM
Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
12/17/2024, 1:46:07 PM
Shutting down background workers
12/17/2024, 1:46:07 PM
Cancelling remote graphs
12/17/2024, 1:46:07 PM
Traceback (most recent call last):
File "/api/langgraph_api/lifespan.py", line 37, in lifespan
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 700, in lifespan
await receive()
File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive
return await self.receive_queue.get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 693, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/contextlib.py", line 231, in aexit
await self.gen.athrow(typ, value, traceback)
File "/api/langgraph_api/lifespan.py", line 41, in lifespan
File "/usr/local/lib/python3.11/site-packages/langgraph_storage/database.py", line 175, in stop_pool
await _pg_pool.close()
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 474, in close
await agather(*workers, timeout=timeout)
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/_acompat.py", line 142, in agather
await asyncio.wait_for(asyncio.shield(wait), timeout=timeout)
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 627, in worker
task = await q.get()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

[n-sviridenko]

@nfcampos I have this issue too - wrote to support and shared traces. In my case there are no asyncio.run calls - only asyncio.gather (but that one has to be for parallelisation). The rest is async/await.

[n-sviridenko]

Could this issue be related to having no listeners on the stream (as the run it purely a background one)? It seems to occur only under those circumstances on LangGraph Cloud.

[codebreach]

We are also facing this and are not doing any asyncio.run in nodes. Its getting a timeout from redis which is caused by cancelled error

  File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 584, in fetch_push_message
    await self._read_flag.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
DEFAULT 2025-02-12T08:32:30.866416Z asyncio.exceptions.CancelledError
DEFAULT 2025-02-12T08:32:30.866422Z The above exception was the direct cause of the following exception:
DEFAULT 2025-02-12T08:32:30.866440Z TimeoutError
DEFAULT 2025-02-12T08:32:30.866445Z During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langgraph_storage/ops.py", line 2029, in listen_for_cancellation
    event = await pubsub.listen()
DEFAULT 2025-02-12T08:32:30.866457Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866462Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 322, in listen
DEFAULT 2025-02-12T08:32:30.866466Z response = await self.parse_response(block=True)
DEFAULT 2025-02-12T08:32:30.866472Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866476Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 242, in parse_response
DEFAULT 2025-02-12T08:32:30.866480Z return await asyncio.wait_for(
DEFAULT 2025-02-12T08:32:30.866485Z ^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866488Z File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
DEFAULT 2025-02-12T08:32:30.866492Z return await fut
DEFAULT 2025-02-12T08:32:30.866495Z ^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866500Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 216, in _execute
DEFAULT 2025-02-12T08:32:30.866504Z return await command(*args)
DEFAULT 2025-02-12T08:32:30.866509Z ^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866514Z File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 586, in fetch_push_message
DEFAULT 2025-02-12T08:32:30.866519Z raise TimeoutError
DEFAULT 2025-02-12T08:32:30.866526Z coredis.exceptions.TimeoutError