No response from .ainvoke sometimes

[NicolasHoferer]

We are experiencing issues when using langchain-aws to connect to the bedrock-runtime service in AWS.
Sometimes we don't get any response.
We could not find any helpful information in our logs, not even with DEBUG log level.
The issue is critical because our application requires minimal latency, and even a 3-second timeout can impact performance.

It does not happen everytime only sometimes and until now only when we run the application in the Azure cloud.

Anyone has an idea or similar issues?

Here is the code we use to setup the ChatBedrock using boto3.

def _get_boto3_client(region=None):
    region_name = config().aws_region
    session = boto3.Session()
    boto3_config = Config(connect_timeout=3, read_timeout=30)
    return session.client(
        service_name='bedrock-runtime',
        region_name=region_name,
        aws_access_key_id=config().aws_access_key_id,
        aws_secret_access_key=config().aws_secret_access_key,
        config=boto3_config
    )

bedrock_client = _get_boto3_client(region)
return ChatBedrock(
    client=bedrock_client,
    model_id=model_id,
    model_kwargs={"temperature": temperature, "max_tokens": 4096},
    callbacks=callbacks,
)

[3coins]

@NicolasHoferer
Thanks for reporting this issue. Acknowledge that this is a concern, but would need more data to reproduce this and verify if this is an issue with the LangChain implementation or an underlying issue with the Bedrock service.

It does not happen everytime only sometimes and until now only when we run the application in the Azure cloud.

One thing that could help here is turning on the CloudWatch logs in AWS for Bedrock services, so that you can see if the request completed on the Bedrock side. Here is the documentation to turn on monitoring for Bedrock.
https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html

[guilt]

@NicolasHoferer

Could we request you to find out the network endpoint boto3 is resolving to, and doing a TCP connect check with nc or other commands to see how long it takes to connect? We want to ensure we are not experiencing network connectivity issues between your environment and Bedrock. What is your observed time for TCP connect?

Could we trace where the boto request is failing for you with:

import boto3
import logging

boto3.set_stream_logger('', logging.DEBUG)

# rest of the code as above

[NicolasHoferer]

Sorry for the late reply.
It only happens when I'm using ainvoke() not with invoke()
Here is a screenshot from httpsession
Strangely in CloudWatch it shows it was successful, i.e. stop_reason is end_turn

Any idea how I can debug further?

[guilt]

Okay, may we ask you for a minimally reproducible test case where you see this Remote end closed connection without response at your end?

@NicolasHoferer could you provide us a test case?