Support Sagemaker async endpoint deployment.

[yinsong1986]

Description

Some one reported error when deploying models on LMI Sagemaker containers using Sagemaker async endpoint. Please refer to vllm-project/vllm#2912

Expected Behavior

Error Message

2024-11-16T23:22:06.943:[sagemaker logs] [xxxxxxxxxxxxxxxxxxxxxx] The response from container primary did not specify the required Content-Length header

How to Reproduce?

(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)

Steps to reproduce

[(Paste the commands you ran that produced the error.)

(vllm-project/vllm#2912)

What have you tried to solve it?

[prashantsolanki975]

Hi Team!
I'm also getting the same error when I'm passing the X-Asynchronus: false in CustomAttributes in Async ? Any idea why ?

[prashantsolanki975]

Hi @yinsong1986
were you able to get this issues fixed ? is this issue due to sagemaker being incompatible with current endpoints ?

[siddvenk]

Hi All, sorry for the delayed response here.

We have enabled async inference in 0.31.0 container version. The root cause is due to how SageMaker handles HTTP responses for async - we are working with them to fix this issue. A workaround is introduced in 0.31.0 to make this functional.

Note that streaming is not supported on async (you cannot specify "stream": true) in the parameters. Please see this notebook for an example https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/lmi-async-inference-demo.ipynb