Make healthcheck dependent on models loaded #1027
Comments
|
I was just looking into this to understand what how ping and Sagemaker The problem is that BOTH the server thread (the main thread/process running Since the model is thought to be loaded when the first I think the quickest fix on your end without a code change on MMS is to preload the model, in which case the model will attempt to be loaded when the Python backend, Note: if you're using this in Sagemaker, Sagemaker docs says that |
Good morning,
I'm using the MMS server to host a single model in AWS Sagemaker. The model is loaded on MMS startup with
sagemaker_inference.model_server.start_model_serverwith a custom handler_service.The problem is that loading the model can take significant amount of time (up to 20 minutes). In the time that model is being loaded MMS server
/pingendpoint returns200 OKresponses, causing sagemaker endpoint to assume the model is already loaded.Furthermore, I tested MMS locally to see how the server behaves in the model loading stage and it looks like the model details endpoint returns model worker status as
READYbefore the worker is finished loading the model.In the logs below you can see that workers start and immediately there's:
after that there's
and only finally:
In the meantime
/pingreturns200 OKandlocalhost:8081/models/testreturns:{ "modelName": "test", "modelUrl": "test.mar", "runtime": "python3", "minWorkers": 2, "maxWorkers": 2, "batchSize": 1, "maxBatchDelay": 100, "loadedAtStartup": true, "workers": [ { "id": "9000-57e7cf49", "startTime": "2024-04-23T11:01:33.316Z", "status": "READY", "gpu": false, "memoryUsage": 13099008 }, { "id": "9000-be19b2af", "startTime": "2024-04-23T11:01:33.317Z", "status": "READY", "gpu": false, "memoryUsage": 13434880 } ] }All logs:
Questions:
/pingendpoint to return something else than200 OKwhen workers are still loading the models?thank you! :)
The text was updated successfully, but these errors were encountered: