Error during llm node initialization for models_path #2912
Comments
|
The commands look correct. I'm just not sure if the difference between the model name in the export and deployment is accidental. I tested manually that this model work fine in ovms. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A clear and concise description of what the bug is.
I have prepared a
text-generationmodel using the filedemos/common/export_models/export_model.py. The config file is:{ "mediapipe_config_list": [ { "name": "HuggingFaceTB/SmolLM2-135M-Instruct", "base_path": "HuggingFaceTB/SmolLM2-135M-Instruct" } ], "model_config_list": [] }When I run the inference server using the docker container:
sudo docker run \ --rm -d \ -p 8085:8085 \ -v $MODEL_DIR:/workspace:ro \ openvino/model_server:2024.5 \ --rest_port 8085 \ --rest_bind_address 0.0.0.0 \ --config_path /workspace/config.jsonThe server starts but i also get an error:
To Reproduce
Steps to reproduce the behavior:
Run the command:
python export_model.py \ text_generation \ --source_model meta-llama/Llama-3.2-3B-Instruct \ --weight-format fp32 \ --config_file_path $CONFIG_FILE_PATH \ --model_repository_path $MODEL_DIR \ --kv_cache_precision u8 \ --overwrite_modelsRun the docker image:
sudo docker run \ --rm -d \ -p 8085:8085 \ -v $MODEL_DIR:/workspace:ro \ openvino/model_server:2024.5 \ --rest_port 8085 \ --rest_bind_address 0.0.0.0 \ --config_path /workspace/config.json --log_level DEBUGExpected behavior
Expected behaviour is for the server to start and to be able to respond to the requests.
Configuration
I need help with identifying any mistakes that I am doing during preparation and running the docker container.
The text was updated successfully, but these errors were encountered: