Skip to content

Latest commit

 

History

History
118 lines (102 loc) · 4.24 KB

csi-debug.md

File metadata and controls

118 lines (102 loc) · 4.24 KB
Raw

CSI driver debug tips

Case#1: volume create/delete failed

There could be multiple controller pods (only one pod is the leader), if there are no helpful logs, try to get logs from the leader controller pod.

  • find csi driver controller pod
$ kubectl get po -o wide -n kube-system | grep csi-smb-controller
NAME                                     READY   STATUS    RESTARTS   AGE     IP             NODE
csi-smb-controller-56bfddd689-dh5tk      5/5     Running   0          35s     10.240.0.19    k8s-agentpool-22533604-0
csi-smb-controller-56bfddd689-sl4ll      5/5     Running   0          35s     10.240.0.23    k8s-agentpool-22533604-1
  • get pod description and logs
$ kubectl describe po csi-smb-controller-56bfddd689-dh5tk -n kube-system > csi-smb-controller-description.log
$ kubectl logs csi-smb-controller-56bfddd689-dh5tk -c smb -n kube-system > csi-smb-controller.log

Case#2: volume mount/unmount failed

  • find csi driver pod that does the actual volume mount/unmount
$ kubectl get po -o wide -n kube-system | grep csi-smb-node
NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE
csi-smb-node-cvgbs                        3/3     Running   0          7m4s    10.240.0.35    k8s-agentpool-22533604-1
csi-smb-node-dr4s4                        3/3     Running   0          7m4s    10.240.0.4     k8s-agentpool-22533604-0
  • get pod description and logs
$ kubectl describe po csi-smb-node-cvgbs -n kube-system > csi-smb-node-description.log
$ kubectl logs csi-smb-node-cvgbs -c smb -n kube-system > csi-smb-node.log
  • check cifs mount inside driver
kubectl exec -it csi-smb-node-cvgbs -n kube-system -c smb -- mount | grep cifs
  • get Windows csi-proxy logs inside driver
kubectl exec -it csi-smb-node-win-xxxxx -n kube-system -c smb cmd
type c:\k\csi-proxy.err.log

Update driver version quickly by editing driver deployment directly

  • update controller deployment
kubectl edit deployment csi-smb-controller -n kube-system
  • update daemonset deployment
kubectl edit ds csi-smb-node -n kube-system

change below deployment config, e.g.

        image: registry.k8s.io/sig-storage/smbplugin:v1.8.0
        imagePullPolicy: Always

troubleshooting connection failure on agent node

  • On Linux node
mkdir /tmp/test
sudo mount -v -t cifs //smb-server/fileshare /tmp/test -o vers=3.0,username=accountname,password=accountkey,dir_mode=0777,file_mode=0777,cache=strict,actimeo=30
  • Check whether original smb mount directory works
sudo mount | grep cifs
Get client-side logs on Linux node if there is mount error 
kubectl debug node/node-name --image=nginx
kubectl cp node-debugger-node-name-xxxx:/host/var/log/messages /tmp/messages
kubectl cp node-debugger-node-name-xxxx:/host/var/log/syslog /tmp/syslog
kubectl cp node-debugger-node-name-xxxx:/host/var/log/kern.log /tmp/kern.log
#after log collected, delete the debug pod by:
kubectl delete po node-debugger-node-name-xxxx
  • On Windows node
$User = "AZURE\USERNAME"
$PWord = ConvertTo-SecureString -String "PASSWORD" -AsPlainText -Force
$Credential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $User, $Pword
New-SmbGlobalMapping -LocalPath x: -RemotePath \\smb-server\fileshare -Credential $Credential
Get-SmbGlobalMapping
cd x:
dir
Get client-side logs on Windows node if there is mount error 
Get SMBClient events from Event Viewer under following path:
Application and Services Logs -> Microsoft -> Windows -> SMBClient

Configure csi-proxy on Windows node

Start a Powershell window as admin

> cd c:\k
Invoke-WebRequest https://acs-mirror.azureedge.net/csi-proxy/v1.0.2/binaries/csi-proxy-v1.0.2.tar.gz -OutFile csi-proxy.tar.gz;
tar -xvf csi-proxy.tar.gz
copy .\bin\csi-proxy.exe .
sc.exe create csiproxy binPath= "c:\k\csi-proxy.exe -windows-service -log_file=c:\k\csi-proxy.log -logtostderr=false --v=5"
sc.exe failure csiproxy reset= 0 actions= restart/10000
sc.exe start csiproxy