Skip to content

Latest commit

 

History

History
executable file
·
60 lines (55 loc) · 3.02 KB

NOTES.md

File metadata and controls

executable file
·
60 lines (55 loc) · 3.02 KB
Raw

Kubernetes (>= 1.6)

  • kubeadm
  • Tectonic (coreos)
  • kops

PX deploy

  • DaemonSet on all nodes including or excluding master
  • DaemonSet allows to scale up with new nodes in k8s cluster
  • We use host network, no POD networks

Debug PX deploy

  • List PX pods: kubectl get pods -l name=portworx -n kube-system
  • Describe PX pods: kubectl describe pods -l name=portworx -n kube-system
  • Logs of all PX pods: kubectl logs -l name=portworx -n kube-system --tail=800
  • Follow logs of a particular px pod: kubectl logs -w <pod-name> -n kube-system
  • Check for custom taints on minions
  • Check for pod restarts
  • Check network on kube-controller-manager on master node
  • Standard px issues:
    • etcd
    • firewalls
    • namespaces
    • docker shared propagation mountflags

Volume lifecycle

  • Create, Delete generated by Kubernetes controller manager on master node.
    • These requests sent to localhost:9001. If fails, sent to our cluster service called "portworx-service"
    • Logs can be view by looking at logs of the kube-controller-manager container on the master.
  • Setup and TearDown sent to kubelet on each minion. So those logs are on kubelet on those nodes (journalctl -lu kubelet).
    • Setup: Attach, Mount
    • TearDown: Unmount, Detach
  • Dynamically provsioned volumes
  • Convergence
    • PX places labels where PVC data resides
    • Use node affinity in scheduler in schedule app pods on nodes with these labels

Apps

  • Pods
    • Basic unit of scheduling
    • Manages lifecycle of container within the pod
    • Portworx gets invoked only for Pod events (NOT container events)
  • Deployments
    • Controller to create and manage pods. You declare desired state (e.g replicas: 3) and it manages the actual state.
    • During node failures/decommissions, pod moves from one minion to another. TearDown and Setup invoked on the PVC.
  • StatefulSets
    • Stable, unique network identifiers.
    • Ordered, graceful deployment and scaling.
    • Ordered, graceful deletion and termination.
    • Ordered, automated rolling updates.
    • If a node is unreachable, pods don't migrate to other nodes. On node reboot, pod migrates.
    • [issue] Issues with Portworx convergence. No way to specify node affinity.
  • Other field Issues:
    • [issue] Startup and Shutdown order of the px container relative to other app containers.
    • [issue] If px is down, pods don't migrate. If our volume is read-only, only container restarts. So we don't get the TearDown and Setup events (attach, mount, unmount detach).
    • [issue] Pods stuck in terminating state because TearDown cannot complete (Unmount/Detach failures). (Fixed 1.2.9)
    • [issue] New pods stuck in creating state because older pod using same volume hasn't terminated. (Fixed 1.2.9)