部署huggingface/text-embeddings-inference启动时卡住 #774
Labels
kind/bug
Something isn't working
Comments
|
I encountered the same issue. There were no problems when using Docker or the GPU operator in Kubernetes, but the issue appeared after migrating to HAMi. |
|
could you add my wechat id:'xuanzong4493', so we can dig further into this issue? |
|
更新到v2.5.0后已解决 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
使用text-embeddings-inference提供的镜像
ghcr.io/huggingface/text-embeddings-inference:89-1.6部署 bge-m3，启动时卡住。我不太确定这个问题是否和hami相关，但脱离hami仅使用docker run是没有问题的：
docker run ：
docker run --name bge-m3-tei -v /data/bge-m3:/data/bge-m3 -e "NVIDIA_VISIBLE_DEVICES=0" ghcr.io/huggingface/text-embeddings-inference:89-1.6 --model-id /data/bge-m3 --port 56246 --hostname 0.0.0.0 --tokenization-workers 1Container logs:
如果使用Deployment部署，则会卡在
Starting FlashBert model on Cuda(CudaDevice(DeviceId(1)))。但是从nvidia-smi来看，模型应该已经加载到显存中，占用的显存也与docker run相当（约1516MiB）。Deployment info:
{ "metadata": { "annotations": { "hami.io/vgpu-devices-allocated": "GPU-404c3ace-26a7-8536-05c7-97b3d38744c3,NVIDIA,24564,0:;", "hami.io/vgpu-devices-to-allocate": "GPU-404c3ace-26a7-8536-05c7-97b3d38744c3,NVIDIA,24564,0:;", "hami.io/vgpu-node": "master01", "hami.io/vgpu-time": "1735832772", "nvidia.com/use-gpuuuid": "GPU-404c3ace-26a7-8536-05c7-97b3d38744c3" }, "creationTimestamp": 1735832772.000000000, "finalizers": [], "generateName": "9187a5c0f8b54c80b3f56fafa18c0fb1-666865984c-", "labels": { "appId": "9187a5c0f8b54c80b3f56fafa18c0fb1", "kubernetes.io/hostname": "master01", "modelId": "bge-m3-gpu", "pod-template-hash": "666865984c" }, "managedFields": [], "name": "9187a5c0f8b54c80b3f56fafa18c0fb1-666865984c-k7qjv", "namespace": "ai-studio", "ownerReferences": [ { "apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "ReplicaSet", "name": "9187a5c0f8b54c80b3f56fafa18c0fb1-666865984c", "uid": "4682f042-917b-4ac9-8fdb-60a667ce3f21" } ], "resourceVersion": "14527711", "uid": "d9a98a0d-ac02-463d-be86-6afbe412550e" }, "spec": { "containers": [ { "args": [ "--model-id", "/data/bge-m3", "--hostname", "0.0.0.0", "--port", "38045" ], "command": [], "env": [], "envFrom": [], "image": "ghcr.io/huggingface/text-embeddings-inference:89-1.6", "imagePullPolicy": "IfNotPresent", "name": "fdb963f4b78e4acda0c70e553e4821d6", "ports": [ { "containerPort": 38045, "hostPort": 38045, "protocol": "TCP" } ], "resizePolicy": [], "resources": { "claims": [], "limits": { "cpu": { "number": 1, "format": "DECIMAL_SI" }, "nvidia.com/gpu": { "number": 1, "format": "DECIMAL_SI" } }, "requests": { "cpu": { "number": 1, "format": "DECIMAL_SI" }, "nvidia.com/gpu": { "number": 1, "format": "DECIMAL_SI" } } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeDevices": [], "volumeMounts": [ { "mountPath": "/etc/localtime", "name": "localtime" }, { "mountPath": "/etc/timezone", "name": "timezone" }, { "mountPath": "/data/bge-m3", "name": "bge-m3-model" }, { "mountPath": "/dev/shm", "name": "cache-volume" }, { "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "kube-api-access-z6hjb", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "enableServiceLinks": true, "ephemeralContainers": [], "hostAliases": [], "imagePullSecrets": [], "initContainers": [], "nodeSelector": { "kubernetes.io/hostname": "master01" }, "overhead": {}, "preemptionPolicy": "PreemptLowerPriority", "priority": 0, "readinessGates": [], "resourceClaims": [], "restartPolicy": "Always", "schedulerName": "hami-scheduler", "schedulingGates": [], "securityContext": { "supplementalGroups": [], "sysctls": [] }, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [ { "effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists", "tolerationSeconds": 300 }, { "effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists", "tolerationSeconds": 300 } ], "topologySpreadConstraints": [], "volumes": [ { "hostPath": { "path": "/etc/localtime", "type": "" }, "name": "localtime" }, { "hostPath": { "path": "/etc/localtime", "type": "" }, "name": "timezone" }, { "hostPath": { "path": "/data/bge-m3", "type": "" }, "name": "bge-m3-model" }, { "emptyDir": { "medium": "Memory", "sizeLimit": { "number": 10737418240, "format": "BINARY_SI" } }, "name": "cache-volume" }, { "name": "kube-api-access-z6hjb", "projected": { "defaultMode": 420, "sources": [ { "serviceAccountToken": { "expirationSeconds": 3607, "path": "token" } }, { "configMap": { "items": [ { "key": "ca.crt", "path": "ca.crt" } ], "name": "kube-root-ca.crt" } }, { "downwardAPI": { "items": [ { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.namespace" }, "path": "namespace" } ] } } ] } } ] }, "status": { "conditions": [], "containerStatuses": [], "ephemeralContainerStatuses": [], "hostIPs": [], "initContainerStatuses": [], "phase": "Pending", "podIPs": [], "qosClass": "Burstable", "resourceClaimStatuses": [] } }pod log:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
/etc/docker/daemon.json)sudo journalctl -r -u kubelet)Environment:
The text was updated successfully, but these errors were encountered: